AWS Fault Injection Service for AWS Lambda - Part 2 FIS Lambda Action invocation-add-delay

Introduction

In part 1 we gave and introduced AWS Fault Injection Service (FIS) and AWS Fault Injection Service Actions to Inject Chaos in Lambda functions. In this part of the series will set up an experiment with aws:lambda:invocation-add-delay AWS FIS Lambda Action. But first of all we'll introduce our sample application which we'll use for this and subsequent experiments.

Sample application for conducting experiments with AWS FIS Lambda Actions

For our experiments, we will use our sample application, the architecture of which is shown below.

In this application, we will create products and retrieve them by their ID and use Amazon DynamoDB as a NoSQL database for the persistence layer. We use Amazon API Gateway which makes it easy for developers to create, publish, maintain, monitor and secure APIs and AWS Lambda to execute code without the need to provision or manage servers. We also use AWS SAM, which provides a short syntax optimised for defining infrastructure as code (hereafter IaC) for serverless applications.

We basically have 2 Lambda functions: PutProductFunction and  GetProductByIdFunction defined in the AWS SAM template.yaml.

Now let's look into AWS FIS Lambda Actions specific parts in the SAM template.

First, we need to add AWS Lambda FIS extension like this:

  PutProductFunction:     Type: AWS::Serverless::Function     Properties:       FunctionName: PutProductWithJava21FISLambda       Layers:         - !Sub arn:aws:lambda:${AWS::Region}:339712832950:layer:aws-fis-extension-x86_64:164 .... 

I use the latest extension version number 164 at the time of writing this article. In order to find the latest version of the extension, go to the "AWS Systems Manager" in your region, for example for eu-central-1 to this page, then find  "Parameter Store" in the navigation bar. You need to store at least 1 parameter in order to search for Public parameters. Then select fis in the list of services:

Then select the Lambda architecture of your choice (default one is x86_64) and copy the extension ARN:

We also need to declare some global Lambda function environment parameters :

Globals:   Function:     CodeUri: target/java21-fis-lambda-1.0.0-SNAPSHOT.jar     Runtime: java21      Environment:       Variables:         AWS_FIS_CONFIGURATION_LOCATION: arn:aws:s3:::vadym-fis/configs/         AWS_LAMBDA_EXEC_WRAPPER: /opt/aws-fis/bootstrap         AWS_FIS_EXTENSION_METRICS: all     ......     

AWS_LAMBDA_EXEC_WRAPPER ‐ Required. Location of the AWS Lambda wrapper script used to configure the AWS FIS Lambda extension. This should be set to the /opt/aws-fis/bootstrap script that is included with the extension.
AWS_FIS_EXTENSION_METRICS ‐ Optional. Possible values are all and none. If set to all the extension will emit EMF metrics under the aws-fis-extension namespace.
Set the AWS_FIS_CONFIGURATION_LOCATION variable to the ARN of the Amazon S3 configuration folder, for example in my case it's arn:aws:s3:::vadym-fis/configs/. As the S3 Bucket is global, please create your own S3 bucket with a subfolder and set this value.

Also give all Lambda functions participating in the FIS experiment permission to access the created S3 Bucket like this:

  PutProductFunction:     Type: AWS::Serverless::Function     Properties:       FunctionName: PutProductWithJava21FISLambda       ....       Policies:         - S3FullAccessPolicy:             BucketName: vadym-fis         .... 

One again, replace the value of BucketName with your individual S3 bucket.

For the complete overview of all AWS FIS Lambda environment variables, please visit this page.

We use managed Java 21 Lambda runtime in this application, but the choice of the programming language doesn't matter for this article series.

In order to build and deploy the sample application, we need the following local installations: Java 21, Maven, AWS CLI and SAM CLI.

To build the application please run the following Maven command mvn clean package (function.zip is created and stored in the subdirectory named target) and to deploy it with the following SAM command sam deploy -g. We will see our customised Amazon API Gateway URL in the return. We can use it to create products and retrieve them by ID. The interface is secured with the API key. We have to send the following as HTTP header: "X-API-Key: a6ZbcDefQW12BN56WEAW7", see MyApiKey definition in template.yaml. To create the product with ID=1, we can use the following curl query:

curl -m PUT -d '{ "id": 1, "name": "Print 10x13", "price": 0.15 }' -H "X-API-Key: a6ZbcDefQW12BN56WEAW7" https://{$API_GATEWAY_URL}/prod/products

For example, to query the existing product with ID=1, we can use the following curl query:

curl -H "X-API-Key: a6ZbcDefQW12BN56WEAW7" https://{$API_GATEWAY_URL}/prod/products/1

In both cases, we need to replace the {$API_GATEWAY_URL} with the individual Amazon API Gateway URL that is returned by the sam deploy -g command. We can also search for this URL when navigating to our API in the Amazon API Gateway service in the AWS console.

Create AWS FIS Experiment Template for aws:lambda:invocation-add-delay

Let's create Experiment templates for our FIS Lambda Action invocation-add-delay. In eu-central one please navigate to Experiment Template, then "Create Experiment template", then specify description and name and what AWS account to target: single or multiple AWS accounts:

Then we need to "add action". Like in the image below

give it a name and description, for the "Action type" select Lambda and "aws:lambda:invocation-add-delay".  We set up "Target" later, fow now select the default value "Function-Target-1".  In the "Action parameters" specify 10 minutes "duration" of the experiment, "invocation percentage" of 100% (means all invocations) and "startup delay in milliseconds" of 10.000, which means we'll inject 10 seconds latency to the target Lambda function. Of course you can specify other values, mine are only for demonstration purposes. Then hit "Save".

Then click on "Function-Target-1" :

to specify the concrete Lambda target you can can you resource tags, filters or parameters or give the concrete ARN of the Lambda function like we're doing. Simpy search for the Lambda function by its name GetProductByIdJava21FISLambda and its ARN will show up.

Then hit "Save". You can create multiple actions and targets using the explanation above. Then press "Next" to configure "Configure service access". You need to create an IAM role for and give it enough permissions. Here are the permissions policies I used:

{     "Version": "2012-10-17",     "Statement": [         {             "Resource": [                 "arn:aws:s3:::vadym-fis/logs/*",                 "arn:aws:s3:::vadym-fis",                 "arn:aws:s3:::vadym-fis/reports/*",                 "arn:aws:s3:::vadym-fis/configs/*"             ],             "Effect": "Allow",             "Action": [                 "s3:PutObject",                 "s3:DeleteObject"             ]         },         {             "Effect": "Allow",             "Action": [                 "lambda:GetFunction"             ],             "Resource": [                 "*"             ]         },         {             "Effect": "Allow",             "Action": [                 "logs:*"             ],             "Resource": [                 "*"             ]         }     ] } 

Instead of my S3 bucket ARN "arn:aws:s3:::vadym-fis" please use yours (see explanation above). Here we allow the FIS experiment to create and delete objects in the specified S3 resources, to access all Lambda functions (you can shrink it only to the specific ones) and to write logs.

Then select this IAM role like in the image below :

and press "Next". In the "Configure optional settings" I don't specify "Stop conditions" and "Report configuration". The latter might make sense for you, but look into the pricing. In the "Logs" section (the destination that receives the experiment log data.) I specified vadym-fis S3 bucket (once again, please replace it with yours). It's the same S3 bucket specified in the IAM permissions policies and SAM template and I created "Log group ARN". This is how it looks like for me:

Then press "Next" and on the next page review everything and press "Create experiment template":

.

Now the experiment template is created :

and we can start it by hitting "Start experiment" in the upper right corner.

Start AWS FIS Experiment Template for aws:lambda:invocation-add-delay

When we start newly created experiment (inject 10 seconds latency for the all invocations of target Lambda function for 10 minutes), AWS FIS service will create experiment object in the subfolder configs of the vadym-fis S3 bucket for the as experiment target specified Lambda function GetProductByIdJava21FISLambda :

This object file contains the experiment details like:

{"faults":[{"actionId":"aws:lambda:invocation-add-delay","parameters": {"invocationPercentage":"100","startupDelayMilliseconds":"10000","qualifier":"$ANY", "minimumRequiredExtensionVersion":"1.0.0"}, "expiration":1751113980295,"startTime":1751113858706}]} 

The FIS Lambda extension that we deployed for our Lambda functions as a part our application will recognize that FIS experiment is going on when GetProductByIdJava21FISLambda Lambda function is invoked and apply experiment settings (inject 10 seconds latency for the all invocations), like explained in the article Introducing AWS Fault Injection Service Actions to Inject Chaos in Lambda functions :

Sometimes it takes up to several minutes or even a new Lambda execution environment to recognize that the experiment has been started and apply experiment settings to the Lambda invocation. For more details, please read the Use the AWS FIS aws:lambda:function actions article.

Let's invoke the GetProductByIdJava21FISLambda Lambda function via API Gateway console :

We see the response time approx 14 seconds : big Lambda cold start (it's Java without enabled SnapStart) and 10 additional seconds of the injected latency. Let's explore GetProductByIdJava21FISLambda Lambda CloudWatch log :

Experiments details have been logged as well:

And we see many metrics sent to the CloudWatch. Let's explore them:

There are FIS Fault Active/Injected metrics:

There are metrics about FIS experiments instructions, like LatencyInjected:

And there are FIS Lambda extension configuration metrics:

For each experiment you can also find its logs in the logs subfolder of the S3 bucket.

You can also use Amazon DevOps Guru - a machine learning service designed to detect abnormal operating patterns. I wrote the article Amazon DevOps Guru for the Serverless applications - Anomaly detection on Lambda invocations where I described how DevOps Guru can detect increased latency of Lambda function invocations.

Conclusion

In this article of the series, we learned how to create a sample application for conducting experiments with AWS FIS Lambda Actions, and how to create and start AWS FIS Experiment Template for the specific aws:lambda:invocation-add-delay Lambda Action.

In the next part of the series, we'll create and start the experiment with the aws:lambda:invocation-http-integration-response Lambda Action.