DEV Community

John Ajera
John Ajera

Posted on

Fixing Fluent Bit on EKS: Solving the "NoCredentialProviders" Error

#eks #fluentbit #observability #terraform

Fixing Fluent Bit on EKS: IRSA, Helm, and the "NoCredentialProviders" Error

If you're deploying Fluent Bit to Amazon EKS with Terraform and Helm, you might hit this frustrating error:

[cloudwatch 0] NoCredentialProviders: no valid providers in chain ... EC2MetadataError: failed to make EC2Metadata request
Enter fullscreen mode Exit fullscreen mode

Even if everything looks correct — the service account is annotated with an IAM role, and the DaemonSet is running — Fluent Bit still can’t write logs to CloudWatch.

Let's walk through how to troubleshoot and fix this.

🔍 Step 1: Understand the Root Cause

Fluent Bit uses AWS credentials to ship logs to CloudWatch. These credentials usually come from:

  • IRSA (IAM Roles for Service Accounts)
  • EC2/EKS metadata (instance profile)
  • Environment variables

When using the aws-for-fluent-bit Helm chart, if you don’t explicitly configure the service account, Helm will create a new one without any IAM role annotation.

You can confirm this with:

kubectl get pod -n amazon-cloudwatch -l app.kubernetes.io/instance=<your-release-name> -o jsonpath="{.items[*].spec.serviceAccountName}"
Enter fullscreen mode Exit fullscreen mode

If it returns something like:

fluent-bit-aws-for-fluent-bit
Enter fullscreen mode Exit fullscreen mode

Instead of your IRSA-linked service account name (e.g., fluent-bit), Fluent Bit is running without credentials.

✅ Step 2: Ensure a Proper IRSA Setup

Make sure the IRSA role exists and is attached to a manually created Kubernetes service account:

resource "kubernetes_service_account" "fluentbit" { metadata { name = "fluent-bit" namespace = "amazon-cloudwatch" annotations = { "eks.amazonaws.com/role-arn" = aws_iam_role.fluentbit_irsa.arn } } }
Enter fullscreen mode Exit fullscreen mode

And the IAM role should have at least these permissions:

resource "aws_iam_policy" "fluentbit_logs" { policy = jsonencode({ Version = "2012-10-17", Statement = [ { Effect = "Allow", Action = [ "logs:PutLogEvents", "logs:DescribeLogStreams", "logs:DescribeLogGroups", "logs:CreateLogStream", "logs:CreateLogGroup" ], Resource = "*" } ] }) }
Enter fullscreen mode Exit fullscreen mode

Attach it to the IRSA role.

✅ Step 3: Update the Helm Release in Terraform

In your helm_release resource, add the following set blocks:

set { name = "serviceAccount.name" value = "fluent-bit" } set { name = "serviceAccount.create" value = "false" }
Enter fullscreen mode Exit fullscreen mode

That tells Helm to use your IRSA-linked SA and not to create its own.

Here’s a full example:

resource "helm_release" "fluentbit" { name = "fluent-bit" namespace = "amazon-cloudwatch" repository = "https://aws.github.io/eks-charts" chart = "aws-for-fluent-bit" version = "0.1.35" set { name = "serviceAccount.name" value = "fluent-bit" } set { name = "serviceAccount.create" value = "false" } set { name = "cloudWatch.enabled" value = "true" } set { name = "cloudWatch.region" value = data.aws_region.current.name } set { name = "cloudWatch.logGroupName" value = "/aws/eks/${var.cluster_name}/fluent-bit" } set { name = "cloudWatch.autoCreateGroup" value = "false" } }
Enter fullscreen mode Exit fullscreen mode

🔄 Step 4: Apply and Restart

Apply the changes:

terraform apply
Enter fullscreen mode Exit fullscreen mode

Then restart the Fluent Bit pod:

kubectl delete pod -n amazon-cloudwatch -l app.kubernetes.io/instance=fluent-bit
Enter fullscreen mode Exit fullscreen mode

Check logs:

kubectl logs -n amazon-cloudwatch -l app.kubernetes.io/instance=fluent-bit
Enter fullscreen mode Exit fullscreen mode

No more NoCredentialProviders 🎉

✅ Recap

✔️ Confirm you're using an IRSA-annotated service account
✔️ Tell Helm to use it with serviceAccount.name and create=false
✔️ Give the IAM role the correct CloudWatch permissions
✔️ Restart pods to pick up changes

Now Fluent Bit can log to CloudWatch securely using IAM — no access keys needed.

Ready to take this up a notch? Turn it into a Terraform module or plug it straight into your CI/CD and forget it ever broke in the first place. 😎

Top comments (0)