Introduction:
This post details an implementation that monitors who or what changed AWS resources created by CloudFormation. By leveraging AWS Config, CloudTrail, Athena, and Lambda, changes can be tracked, logs can be analysed, and compliance reporting can be automated. The collected data is stored in Amazon S3, making it accessible for audits and compliance verification.
About the Project:
This post builds upon my earlier article on monitoring CloudFormation stack drift using AWS Config Rules. In this enhanced version, the monitoring capabilities are extended by:
Tracking user activities affecting CloudFormation resources.
Logging change details to Amazon S3 via CloudTrail.
Processing and querying logs using AWS Athena.
Automating remediation through AWS Systems Manager and Lambda.
Core Components:
AWS Config Rule: Monitors stacks for drift using CLOUDFORMATION_STACK_DRIFT_DETECTION_CHECK.
Systems Manager Automation Runbook: Invokes a Lambda function for compliance checks.
Remediation Action: Executes the InvokeLambdaFromConfig automation document.
Amazon S3 Bucket: Stores logs from CloudTrail and Athena query results.
Athena Table: Organises and queries raw log data.
CloudTrail Trail: Captures AWS API activity logs.
Lambda Function: Extracts CloudFormation resource names and queries Athena for recent changes.
Infrastructure Scema:
Configuration in infrastructure/monitoring_stack_cloudtrail.yaml CloudFormation template:
AWSTemplateFormatVersion: '2010-09-09' Description: CloudTrail setup for monitoring CFN stack modifications Parameters: AthenaDatabaseName: Type: String Description: Athena database name for running queries Default: 'cloudtrail_logs' StackNameToMonitor: Type: String Description: CloudFormation stack name to monitor Default: 'base-infrastructure' MaximumExecutionFrequency: Type: String Description: The maximum frequency with which drift in CloudFormation stacks need to be evaluated Default: 'One_Hour' Resources: ################################# # CloudTrail and Athena ################################# CloudTrailLogsBucket: Type: AWS::S3::Bucket Properties: BucketName: !Sub "aws-cloudtrail-logs-${AWS::AccountId}" VersioningConfiguration: Status: Enabled LifecycleConfiguration: Rules: - Id: ExpireLogs Status: Enabled ExpirationInDays: 365 CloudTrailLogsBucketPolicy: Type: AWS::S3::BucketPolicy Properties: Bucket: !Ref CloudTrailLogsBucket PolicyDocument: Version: "2012-10-17" Statement: - Sid: "AWSCloudTrailAclCheck" Effect: Allow Principal: Service: cloudtrail.amazonaws.com Action: s3:GetBucketAcl Resource: !Sub "arn:${AWS::Partition}:s3:::aws-cloudtrail-logs-${AWS::AccountId}" Condition: StringEquals: AWS:SourceArn: !Sub "arn:${AWS::Partition}:cloudtrail:${AWS::Region}:${AWS::AccountId}:trail/monitoring-cfn-policy-compliance" - Sid: "AWSCloudTrailWrite" Effect: Allow Principal: Service: cloudtrail.amazonaws.com Action: s3:PutObject Resource: !Sub "arn:${AWS::Partition}:s3:::aws-cloudtrail-logs-${AWS::AccountId}/AWSLogs/${AWS::AccountId}/*" Condition: StringEquals: AWS:SourceArn: !Sub "arn:${AWS::Partition}:cloudtrail:${AWS::Region}:${AWS::AccountId}:trail/monitoring-cfn-policy-compliance" s3:x-amz-acl: "bucket-owner-full-control" - Sid: "AthenaQueryResultPutObject" Effect: Allow Principal: Service: athena.amazonaws.com Action: s3:PutObject Resource: !Sub "arn:${AWS::Partition}:s3:::aws-cloudtrail-logs-${AWS::AccountId}/athena-results/*" Condition: StringEquals: aws:SourceArn: !Sub "arn:${AWS::Partition}:athena:${AWS::Region}:${AWS::AccountId}:workgroup/primary" CloudTrail: Type: AWS::CloudTrail::Trail Properties: TrailName: monitoring-cfn-policy-compliance S3BucketName: !Ref CloudTrailLogsBucket IncludeGlobalServiceEvents: true IsMultiRegionTrail: true EnableLogFileValidation: false IsOrganizationTrail: false IsLogging: true AthenaDatabase: Type: AWS::Glue::Database Properties: CatalogId: !Ref AWS::AccountId DatabaseInput: Name: !Ref AthenaDatabaseName AthenaTable: Type: AWS::Glue::Table Properties: CatalogId: !Ref AWS::AccountId DatabaseName: !Ref AthenaDatabase TableInput: Name: !Sub "aws_cloudtrail_logs_${AWS::AccountId}" TableType: EXTERNAL_TABLE Parameters: classification: cloudtrail StorageDescriptor: Columns: - Name: eventVersion Type: string - Name: userIdentity Type: struct<type:string,principalId:string,arn:string,accountId:string,invokedBy:string,accessKeyId:string,userName:string,sessionContext:struct<attributes:struct<mfaAuthenticated:string,creationDate:string>,sessionIssuer:struct<type:string,principalId:string,arn:string,accountId:string,username:string>,ec2RoleDelivery:string,webIdFederationData:struct<federatedProvider:string,attributes:map<string,string>>>> - Name: eventTime Type: string - Name: eventSource Type: string - Name: eventName Type: string - Name: awsRegion Type: string - Name: sourceIpAddress Type: string - Name: userAgent Type: string - Name: errorCode Type: string - Name: errorMessage Type: string - Name: requestParameters Type: string - Name: responseElements Type: string - Name: additionalEventData Type: string - Name: requestId Type: string - Name: eventId Type: string - Name: resources Type: array<struct<arn:string,accountId:string,type:string>> - Name: eventType Type: string - Name: apiVersion Type: string - Name: readOnly Type: string - Name: recipientAccountId Type: string - Name: serviceEventDetails Type: string - Name: sharedEventID Type: string - Name: vpcEndpointId Type: string - Name: tlsDetails Type: struct<tlsVersion:string,cipherSuite:string,clientProvidedHostHeader:string> Location: !Sub "s3://aws-cloudtrail-logs-${AWS::AccountId}/AWSLogs/${AWS::AccountId}/CloudTrail/" InputFormat: com.amazon.emr.cloudtrail.CloudTrailInputFormat OutputFormat: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat SerdeInfo: SerializationLibrary: org.apache.hive.hcatalog.data.JsonSerDe ################################# # Lambda function ################################# LambdaExecutionRole: Type: AWS::IAM::Role Properties: RoleName: LambdaAthenaQueryExecutionRole AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: Service: - lambda.amazonaws.com - athena.amazonaws.com Action: - sts:AssumeRole Policies: - PolicyName: CloudFormationDescribe PolicyDocument: Version: "2012-10-17" Statement: - Effect: Allow Action: - cloudformation:DescribeStackResources Resource: "arn:aws:cloudformation:*" - PolicyName: AthenaQueryPolicy PolicyDocument: Version: "2012-10-17" Statement: - Effect: Allow Action: - athena:StartQueryExecution - athena:GetQueryExecution - athena:GetQueryResults - athena:GetWorkGroup - athena:GetDataCatalog - athena:GetTableMetadata - glue:GetDatabase - glue:GetTable - glue:GetPartitions Resource: "*" - Effect: Allow Action: - s3:PutObject - s3:GetObject - s3:ListBucket - s3:GetBucketLocation - s3:PutObjectAcl Resource: - !Sub "arn:${AWS::Partition}:s3:::${CloudTrailLogsBucket}" - !Sub "arn:${AWS::Partition}:s3:::${CloudTrailLogsBucket}/*" - Effect: Allow Action: - lambda:AddPermission Resource: "*" - Effect: Allow Action: - logs:CreateLogGroup - logs:CreateLogStream - logs:PutLogEvents Resource: "*" CheckCloudTrailLogsLambda: Type: AWS::Lambda::Function Properties: FunctionName: CheckCloudTrailLogsLambda Runtime: nodejs22.x Handler: index.handler Role: !GetAtt LambdaExecutionRole.Arn Timeout: 120 MemorySize: 256 Environment: Variables: STACKS_TO_MONITOR: !Ref StackNameToMonitor ATHENA_DATABASE: !Ref AthenaDatabase ATHENA_TABLE: !Ref AthenaTable S3_OUTPUT_BUCKET: !Ref CloudTrailLogsBucket Code: ZipFile: | const { AthenaClient, StartQueryExecutionCommand } = require("@aws-sdk/client-athena"); const { CloudFormationClient, DescribeStackResourcesCommand } = require("@aws-sdk/client-cloudformation"); const athena = new AthenaClient({}); const cloudformation = new CloudFormationClient({}); exports.handler = async (event) => { console.log("Event received:", JSON.stringify(event, null, 2)); const stacks = process.env.STACKS_TO_MONITOR.split(","); const tableName = process.env.ATHENA_TABLE; const databaseName = process.env.ATHENA_DATABASE; const s3Bucket = process.env.S3_OUTPUT_BUCKET; let resourceNames = []; // Extract resource names from the CloudFormation stacks for (const stack of stacks) { const stackResources = await cloudformation.send( new DescribeStackResourcesCommand({ StackName: stack }) ); stackResources.StackResources.forEach(resource => { if (resource.PhysicalResourceId) { resourceNames.push(resource.PhysicalResourceId); } }); } // Construct Athena query let whereClause = resourceNames.map(name => `resource.arn LIKE '%${name}%'`).join(" OR "); let queryString = ` SELECT userIdentity.userName AS username, eventName AS action, eventTime AS timestamp, resource.arn AS resource_arn, sourceIPAddress AS request_source, userAgent AS user_agent FROM ${tableName} CROSS JOIN UNNEST(resources) AS t(resource) WHERE (${whereClause}) AND eventName IS NOT NULL AND userIdentity.userName IS NOT NULL AND from_iso8601_timestamp(eventTime) >= current_timestamp - INTERVAL '1' HOUR ORDER BY from_iso8601_timestamp(eventTime) DESC; `; // Run the Athena query const params = { QueryString: queryString, QueryExecutionContext: { Database: databaseName }, ResultConfiguration: { OutputLocation: `s3://${s3Bucket}/athena-results/` } }; try { const command = new StartQueryExecutionCommand(params); const queryExecution = await athena.send(command); console.log("Query started:", queryExecution.QueryExecutionId); return { status: "Query started successfully", queryExecutionId: queryExecution.QueryExecutionId }; } catch (error) { console.error("Error running query:", error); throw error; } }; LambdaPermissionForConfig: Type: AWS::Lambda::Permission Properties: FunctionName: !Ref CheckCloudTrailLogsLambda Action: lambda:InvokeFunction Principal: config.amazonaws.com ################################# # Config Rule ################################# IamRoleForConfig2: Type: AWS::IAM::Role Properties: RoleName: CfnDriftDetectionForCloudTrail Description: IAM role for AWS Config to access CloudFormation drift detection AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: Service: config.amazonaws.com Action: - sts:AssumeRole ManagedPolicyArns: - arn:aws:iam::aws:policy/ReadOnlyAccess Policies: - PolicyName: CloudFormationDriftDetectionpolicy PolicyDocument: Version: "2012-10-17" Statement: - Effect: Allow Action: - cloudformation:DetectStackResourceDrift - cloudformation:DetectStackDrift - cloudformation:DescribeStacks - cloudformation:DescribeStackResources - cloudformation:BatchDescribeTypeConfigurations - cloudformation:DescribeStackResourceDrifts - cloudformation:DescribeStackDriftDetectionStatus Resource: !Sub "arn:${AWS::Partition}:cloudformation:${AWS::Region}:${AWS::AccountId}:*" ConfigRuleCheckCloudTralLogs: DependsOn: - LambdaPermissionForConfig Type: AWS::Config::ConfigRule Properties: ConfigRuleName: ConfigRuleCheckCloudTrailLogs Description: AWS Config rule to detect drift in CFN stacks and check CloudTrail logs Scope: TagKey: stack-name TagValue: !Ref StackNameToMonitor Source: Owner: AWS SourceIdentifier: CLOUDFORMATION_STACK_DRIFT_DETECTION_CHECK MaximumExecutionFrequency: !Ref MaximumExecutionFrequency InputParameters: cloudformationRoleArn: !GetAtt IamRoleForConfig2.Arn IamRoleForRemediation: Type: AWS::IAM::Role Properties: RoleName: AwsConfigRemediationActionInvokeLambda Description: IAM role for AWS Config remediation action to invoke Lambda function AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: Service: - config.amazonaws.com - ssm.amazonaws.com Action: - sts:AssumeRole Policies: - PolicyName: InvokeLambdaPolicy PolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Action: - lambda:InvokeFunction Resource: !GetAtt CheckCloudTrailLogsLambda.Arn SsmDocumentInvokeLambda: Type: AWS::SSM::Document Properties: DocumentType: Automation Name: InvokeLambdaFromConfig Content: schemaVersion: "0.3" description: "SSM Automation document to invoke a Lambda function" parameters: AutomationAssumeRole: type: String description: (Optional) The ARN of the role that allows Automation to perform the actions default: !GetAtt IamRoleForRemediation.Arn mainSteps: - name: InvokeLambda action: aws:invokeLambdaFunction inputs: FunctionName: !Ref CheckCloudTrailLogsLambda Payload: '{}' InvocationType: Event LogType: None maxAttempts: 2 timeoutSeconds: 30 onFailure: Abort isCritical: true assumeRole: !GetAtt IamRoleForRemediation.Arn RemediationActionInvokeLambda: Type: AWS::Config::RemediationConfiguration Properties: ConfigRuleName: !Ref ConfigRuleCheckCloudTralLogs TargetType: SSM_DOCUMENT TargetId: !Ref SsmDocumentInvokeLambda Automatic: true MaximumAutomaticAttempts: 2 RetryAttemptSeconds: 30 Parameters: AutomationAssumeRole: StaticValue: Values: - !GetAtt IamRoleForRemediation.Arn Prerequisites:
Ensure the following prerequisites are in place:
- An AWS account with sufficient permissions to create and manage resources.
- The AWS CLI installed on the local machine.
- CloudFormation infrastructure deployed from my previous post (if applicable).
Deployment:
- Deploy the CloudFormation Stack.
aws cloudformation create-stack \ --stack-name monitoring-policy-compliance \ --template-body file://infrastructure/monitoring_stack_cloudtrail.yaml \ --capabilities CAPABILITY_NAMED_IAM \ --disable-rollback 2.Test the resources deployed with the stack. Change value of the resource from the base-infrastructure stack and evaluate the drift detection rule to verify functionality.
aws ssm put-parameter --name "ConnectionToken" --value "secret_token_value_2" --type "String" --overwrite aws configservice start-config-rules-evaluation --config-rule-names ConfigRuleCheckCloudTrailLogs 3.After drift detection runs, review Athena query results stored in S3 under /athena-results in .csv file.
aws s3 ls s3://<your-bucket-name>/athena-results/ --recursive aws s3 cp s3://<your-bucket-name>/<report_name>.csv ./ Here is an example of logs from this file:
)
4.Cleanup Resources. After testing, stop the CloudTrail Trail logging, delete all data from the S3 bucket, and delete the CloudFormation stack.
aws cloudtrail stop-logging --name monitor-cfn-policy-compliance aws s3 rm s3://<your-bucket-name> --recursive aws cloudformation delete-stack --stack-name monitoring-policy-compliance Conclusion:
Implementing this solution provides visibility into changes affecting CloudFormation-managed resources. This enhances security, compliance tracking, and audit readiness. The ability to log and query user actions simplifies responses to compliance requests from clients, regulators, or security teams.
If you found this post helpful and interesting, please click the reaction button below to show your support. Feel free to use and share this post. 🙂
Top comments (0)