Using AWS Distro for OpenTelemetry Collector for cross-account metrics collection on Amazon ECS

In November 2020, we announced OpenTelemetry support on AWS with AWS Distro for OpenTelemetry (ADOT), a secure, production-ready, AWS-supported distribution of the Cloud Native Computing Foundation (CNCF) OpenTelemetry project. With ADOT, you can instrument applications to send correlated metrics and traces to multiple AWS solutions, such as our Amazon Managed Service for Prometheus (AMP) and Partner monitoring solutions.

Many customers have their applications running on separate AWS accounts—and even separate AWS Regions—and would like to have a central place for observability. In a previous article, we explained how to collect metrics across multiple accounts with Amazon Elastic Kubernetes Service (Amazon EKS). The scenario will be similar, except, in this one, we use the ADOT agent to collect application and platform metrics for workloads running on Amazon Elastic Container Service (Amazon ECS), our native container orchestration platform to an AMP workspace.

Setup overview

To resolve this challenge, we will use the following structure.

On the workload accounts:

• Create an IAM role to be used by Amazon ECS tasks.

On the central monitoring account:

• Create an AMP workspace.
• Create an IAM role that allows cross-account access to AMP.

On the workload accounts:

• Create Amazon ECS tasks permissions to assume a cross-account IAM role.
• Set up the application and the AWS Distro for OpenTelemetry agent.
• Create an Amazon ECS cluster and run the application.

On the central monitoring account:

• Visualize metrics with Amazon Managed Grafana.

The entire architecture looks like the following:

Workload account: ECS role setup

Logged into the workload account, we create an IAM role that will be used later by Amazon ECS tasks. This role then will be trusted on the central monitoring account and granted assume-role permissions.

cat > task-assume-role.json <<EOF { "Version": "2012-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Principal": { "Service": "ecs-tasks.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } EOF aws iam create-role --role-name ecs-xaccount-task-role \ --assume-role-policy-document file://task-assume-role.json \ --region eu-west-1 

Monitoring account setup

Logged into the workload account, we create an AMP workspace with the following command with awscli:

aws amp create-workspace --alias ecs-xaccount-metrics-demo --region eu-west-1

Alternatively, we can use the AWS console and navigate to the AMP service.

We now can create an IAM role with write permissions to the AMP workspace. To grant multiple accounts, populate the "AWS" array with appropriate IAM role ARNs:

WORKLOAD_ACCOUNT_ID= cat > policy.json <<EOF { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": [ "arn:aws:iam::$WORKLOAD_ACCOUNT_ID:role/ecs-xaccount-task-role" ] }, "Action": "sts:AssumeRole", "Condition": {} } ] } EOF # Note: You might encounter an error if the ecs-xaccount-task-role # does not exists in the workload account. aws iam create-role \ --role-name ECS-AMP-Central-Role \ --assume-role-policy-document file://policy.json \ --query 'Role.RoleName' \ --output text aws iam attach-role-policy --role-name ECS-AMP-Central-Role \ --policy-arn arn:aws:iam::aws:policy/AmazonPrometheusRemoteWriteAccess

Workload account

Note: You can repeat instructions in this section for as many workload accounts as needed.

Logged into the workload account, we grant assumeRole permissions to the role created previously:

# Set the central account id CENTRAL_ACCOUNT_ID= cat > policy.json <<EOF { "Version":"2012-10-17", "Statement":[ { "Effect":"Allow", "Action":[ "sts:AssumeRole" ], "Resource":"arn:aws:iam::${CENTRAL_ACCOUNT_ID}:role/ECS-AMP-Central-Role" } ] } EOF POLICY_ARN=$(aws iam create-policy --policy-name xaccount-amp-write \ --policy-document file://policy.json | jq -r '.Policy.Arn') aws iam attach-role-policy --role-name ecs-xaccount-task-role \ --policy-arn $POLICY_ARN 

Workload configuration

Next, we set up a sample application that exposes Prometheus metrics:

• Configure the aws-otel-collector to scrape the application and ECS metrics.
• Build Docker images and host them on Amazon Elastic Container Registry (Amazon ECR).
• Configure, create an Amazon ECS cluster, and run everything using ecs-cli.

The layout should be organized as follows:

├── aws-otel-collector │ ├── Dockerfile │ └── config.yaml ├── demo-app │ ├── Dockerfile │ └── main.go ├── docker-compose.yml └── ecs-params.yml

To set up Amazon ECS, we need Docker and ecs-cli as requirements. On Linux, ecs-cli can be installed like this:

sudo curl -Lo /usr/local/bin/ecs-cli https://amazon-ecs-cli.s3.amazonaws.com/ecs-cli-linux-amd64-latest

Now, let’s create the sample application that exposes a /metrics Prometheus endpoint:

mkdir demo-app cd demo-app/ cat > main.go <<EOF package main import ( "github.com/prometheus/client_golang/prometheus/promhttp" "net/http" ) func main() { http.Handle("/metrics", promhttp.Handler()) http.ListenAndServe(":8000", nil) } EOF 

This will create a Dockerfile for the application:

cat > Dockerfile <<EOF FROM golang:1.18 as builder WORKDIR /go/src/app COPY . . RUN go mod init demo RUN go get . RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o app . FROM alpine:latest WORKDIR /app RUN apk --no-cache add ca-certificates COPY --from=builder /go/src/app/app . EXPOSE 8000 CMD ["./app"] EOF 

And finally, the following script will create an ECR repository, build the application image, and push the image to Amazon ECR:

APP_REPOSITORY=$(aws ecr create-repository --repository demo-app --query repository.repositoryUri --output text) docker build . -t demo-app aws ecr get-login-password --region eu-west-1 | docker login --username AWS --password-stdin $APP_REPOSITORY docker tag demo-app:latest $APP_REPOSITORY docker push $APP_REPOSITORY cd -

Now, let’s configure the AWS Distro for OpenTelemetry Collector. We will create a custom configuration to collect data called a Pipeline. A Pipeline defines a path the data follows in the collector starting from reception, then further processing or modification, and finally exiting the collector via exporters.

We will collect from the application with the /metrics endpoint and make use of the ecs-metrics-receiver to scrape various ECS task metadata from the ECS task metadata endpoint. Visit the documentation to learn more about ecs-metrics-receiver and other configuration options.

We will export collected metrics to the AMP workspace created on the monitoring account using awsprometheusremotewrite exporters configuration. We will provide both the AMP remote_write endpoint and the IAM role to assume—in our case, ECS-AMP-Central-Role.

Edit the WORKSPACE_ID and CENTRAL_ACCOUNT_ID variables and run the following script to create the pipeline:

WORKSPACE_ID= CENTRAL_ACCOUNT_ID= mkdir aws-otel-collector cd aws-otel-collector cat > config.yaml <<EOF receivers: prometheus: config: global: scrape_interval: 15s scrape_timeout: 10s scrape_configs: - job_name: "prometheus-demo-app" static_configs: - targets: [ 0.0.0.0:8000 ] awsecscontainermetrics: collection_interval: 20s processors: filter: metrics: include: match_type: strict metric_names: - ecs.task.memory.utilized - ecs.task.memory.reserved - ecs.task.cpu.utilized - ecs.task.cpu.reserved - ecs.task.network.rate.rx - ecs.task.network.rate.tx - ecs.task.storage.read_bytes - ecs.task.storage.write_bytes exporters: prometheusremotewrite: endpoint: https://aps-workspaces.eu-west-1.amazonaws.com/workspaces/$WORKSPACE_ID/api/v1/remote_write auth: authenticator: sigv4auth logging: loglevel: debug extensions: sigv4auth: service: "aps" assume_role: arn: arn:aws:iam::$CENTRAL_ACCOUNT_ID:role/ECS-AMP-Central-Role sts_region: us-west-2 service: extensions: [sigv4auth] pipelines: metrics: receivers: [prometheus] exporters: [logging, prometheusremotewrite] metrics/ecs: receivers: [awsecscontainermetrics] processors: [filter] exporters: [logging, prometheusremotewrite] EOF

From the latest version of the aws-otel-collector, create a custom image on Amazon ECR with our custom configuration:

cat > Dockerfile <<EOF FROM public.ecr.aws/aws-observability/aws-otel-collector:latest COPY config.yaml /etc/ecs/otel-config.yaml CMD ["--config=/etc/ecs/otel-config.yaml"] EOF 

Finally, build and push the image:

COLLECTOR_REPOSITORY=$(aws ecr create-repository --repository aws-otel-collector --query repository.repositoryUri --output text) docker build . -t aws-otel-collector aws ecr get-login-password --region eu-west-1 | docker login --username AWS --password-stdin $COLLECTOR_REPOSITORY docker tag aws-otel-collector:latest $COLLECTOR_REPOSITORY docker push $COLLECTOR_REPOSITORY cd -

Run application: Set up Amazon ECS

Amazon ECS needs an execution role—a set of permissions to run our tasks. Run the following script to create it:

cat > task-execution-assume-role.json <<EOF { "Version": "2012-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Principal": { "Service": "ecs-tasks.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } EOF aws iam create-role --role-name ecs-xaccount-task-execution-role \ --assume-role-policy-document file://task-execution-assume-role.json \ --region eu-west-1 aws iam --region eu-west-1 attach-role-policy --role-name ecs-xaccount-task-execution-role \ --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy 

Set up the WORKLOAD_ACCOUNT_ID variable and run the following script to create a docker-compose file:

WORKLOAD_ACCOUNT_ID= cat > docker-compose.yml <<EOF version: "3" services: aws-otel-collector: image: $WORKLOAD_ACCOUNT_ID.dkr.ecr.eu-west-1.amazonaws.com/aws-otel-collector:latest environment: - AWS_REGION=eu-west-1 logging: driver: awslogs options: awslogs-group: ecs-xaccount-metrics-demo awslogs-region: eu-west-1 awslogs-stream-prefix: aws-otel-collector prometheus-demo-app: image: $WORKLOAD_ACCOUNT_ID.dkr.ecr.eu-west-1.amazonaws.com/demo-app ports: - "8000:8000" depends_on: - aws-otel-collector logging: driver: awslogs options: awslogs-group: ecs-xaccount-metrics-demo awslogs-region: eu-west-1 awslogs-stream-prefix: demo-app EOF 

Using ecs-cli, we will create an Amazon ECS cluster:

ecs-cli configure --cluster ecs-xaccount-metrics-demo \ --default-launch-type FARGATE \ --config-name ecs-xaccount-metrics-demo \ --region eu-west-1 ecs-cli up --cluster-config ecs-xaccount-metrics-demo

After few minutes, the cluster should be created with all necessary associated resources. Select the VPC_ID from the preceding command and get the default security group associated to the VPC:

VPC_ID= aws ec2 describe-security-groups --filters Name=vpc-id,Values=$VPC_ID \ --region eu-west-1 \ --query SecurityGroups[0].GroupId \ --output text

Edit the ecs-params.yml file needed by ecs-cli, and replace the subnet IDs and security group from the previous outputs:

version: 1 task_definition: ecs_network_mode: awsvpc task_role_arn: ecs-xaccount-task-role task_execution_role: ecs-xaccount-task-execution-role task_size: mem_limit: 0.5GB cpu_limit: 256 run_params: network_configuration: awsvpc_configuration: subnets: - "subnet-" - "subnet-" security_groups: - "sg-" assign_public_ip: ENABLED

Finally, run the following script to deploy the application:

ecs-cli compose --project-name ecs-xaccount-metrics-demo \ service up \ --cluster-config ecs-xaccount-metrics-demo \ --create-log-groups

After few minutes, the Amazon ECS service should be up and running. You can verify the logs of the aws-otel-collector on the Amazon CloudWatch Logs console, with the log group ecs-xaccount-metrics-demo.

Monitoring account: Visualize metrics

Back in the monitoring account, let’s visualize our metrics using an Amazon Managed Grafana workspace. Refer to the documentation to set up Amazon Managed Grafana.

We can view metrics coming from the application endpoint:

And the Amazon ECS cluster metrics:

Clean up

Workload account

WORKLOAD_ACCOUNT_ID= # stop and deletes ecs service ecs-cli compose --project-name ecs-xaccount-metrics-demo service down --cluster-config ecs-xaccount-metrics-demo # delete ecs cluster ecs-cli down --cluster-config ecs-xaccount-metrics-demo # delete task role aws iam detach-role-policy --role-name ecs-xaccount-task-role --policy-arn arn:aws:iam::$WORKLOAD_ACCOUNT_ID:policy/xaccount-amp-write aws iam delete-policy --policy-arn arn:aws:iam::$WORKLOAD_ACCOUNT_ID:policy/xaccount-amp-write aws iam delete-role --role-name ecs-xaccount-task-role # delete task execution role aws iam detach-role-policy --role-name ecs-xaccount-task-execution-role --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy aws iam delete-role --role-name ecs-xaccount-task-execution-role

Central account

WORKSPACE_ID= # delete role aws iam detach-role-policy --role-name ECS-AMP-Central-Role --policy-arn arn:aws:iam::aws:policy/AmazonPrometheusRemoteWriteAccess aws iam delete-role --role-name ECS-AMP-Central-Role # delete workspace aws amp delete-workspace --workspace-id $WORKSPACE_ID

Conclusion

In this post, we explained how to use the AWS Distro for OpenTelemetry (ADOT) agent to collect application and platform metrics for workloads running on Amazon ECS.

You can use ADOT on other platforms, such as Amazon EKS, Amazon Elastic Compute Cloud (Amazon EC2), or on-premises. Additionally, you can use ADOT to collect distributed traces data and have multiple heterogeneous workload accounts sending metrics centrally to AMP and other platforms. Also, you can set up private connectivity with VPC endpoints and VPC peering, according to your needs.

Visit the ADOT, AMP, and Amazon Managed Grafana sites to learn more.