AWS Fault Injection Service for AWS Lambda - Part 3 FIS Lambda Action invocation-http-integration-response

Introduction

In the part 2 article of the series, we learned how to create a sample application for conducting experiments with AWS FIS Lambda Actions, and how to create and start AWS FIS Experiment Template for the specific aws:lambda:invocation-add-delay Lambda Action.

In this part of the series, we'll create and start the experiment with the aws:lambda:invocation-http-integration-response Lambda Action.

For this we'll re-use the sample application from the previous part. Also please read there how to create AWS FIS Experiment Template for Lambda Action, as we'll only focus on the differences required for aws:lambda:invocation-http-integration-response Action.

Create and start AWS FIS Experiment Template for aws:lambda:invocation-http-integration-response

We'll focus mainly on the Action itself :

give it a name and description, for the "Action type" select Lambda and "aws:lambda:invocation-http-integration-response". In the "Action parameters" specify 10 minutes "duration" of the experiment, "invocation percentage" of 100% (means all invocations) and "status code" of 500, and enable "percent execution". This means return HTTP status code 500 in response to HTTP integration like Amazon API Gateway for all invocations of the target Lambda function without executing its business logic. Of course you can specify other values (for example simulating HTTP 4XX errors like 403 and 404), mine are only for demonstration purposes. Then hit "Save".

For Targets, we'll select another Lambda function, please search for PutProductWithJava21FISLambda :

When we start newly created experiment, AWS FIS service will create experiment object in the subfolder configs of the vadym-fis S3 bucket for the as experiment target specified Lambda function PutProductWithJava21FISLambda :

This object file contains the experiment details like:

{"faults":[{"actionId":"aws:lambda:invocation-http-integration-response", "parameters"{"invocationPercentage":"100","preventExecution":"true", "replaceOutputWithString":"{\"body\":\"\",\"headers\":{\"Content-Type\":\"text/plain\"}, \"statusCode\"\"500\"}","injectAsError":"false","qualifier":"$ANY","minimumRequiredExtensionVersion":"1.0.0"}, "expiration":1751296234169,"startTime":1751296009540}]} 

The FIS Lambda extension that we deployed for our Lambda functions as a part our application will recognize that FIS experiment is going on when PutProductWithJava21FISLambda Lambda function is invoked and apply experiment settings (return HTTP 500 status code as an integration response to the API Gateway for the all Lambda invocations), like explained in the article Introducing AWS Fault Injection Service Actions to Inject Chaos in Lambda functions :

Sometimes it takes up to several minutes or even a new Lambda execution environment to recognize that the experiment has been started and apply experiment settings to the Lambda invocation. For more details, please read the Use the AWS FIS aws:lambda:function actions article.

Let's invoke the PutProductWithJava21FISLambda Lambda function using the load test tool hey
with:

hey -q 3 -z 1m -c 4 -m PUT -d '{"id": 29, "name": "Print 10x13", "price": 0.15}' -H "X-API-Key: a6ZbcDefQW12BN56WEAW7" -H "Content-Type: application/json;charset=utf-8" https://{$API_GATEWAY_URL}/prod/products/

But you can use whatever tool you want, like Serverless-artillery or Postman.

We see many HTTP 5XX errors for our AWSJava21FISLambdaAPI in the CloudWatch :

Let's explore PutProductWithJava21FISLambda Lambda CloudWatch log :

Experiment details have been logged as well:

And we see many metrics sent to CloudWatch. Let's explore them:

There are FIS Fault Active/Injected metrics:

And there are FIS Lambda extension configuration metrics:

For each experiment you can also find its logs in the logs subfolder of the S3 bucket.

You can also use Amazon DevOps Guru - a machine learning service designed to detect abnormal operating patterns. I wrote the article Amazon DevOps Guru for the Serverless applications - Anomaly detection on API Gateway where I described how DevOps Guru can detect API Gateway 4XX and 5XX errors.

Conclusion

In this article of the series, we learned how to create and start AWS FIS Experiment Template for the specific aws:lambda:invocation-http-integration-response Lambda Action.

In the next part of the series, we will create and start the experiment with the aws:lambda:invocation-error Lambda Action.