Reconcile errors on EKS cluster with Hybrid nodes #4315
Description
Bug Description
Logs and TargetGroupBinding objects are flooded with errors when target pods are deployed on hybrid nodes.
Steps to Reproduce
- Step-by-step guide to reproduce the bug:
- Deploy EKS cluster with hybrid nodes as per: https://docs.aws.amazon.com/eks/latest/userguide/hybrid-nodes-overview.html
- Configure CNI and Ingress
- Manifests applied while reproducing the issue:
apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: replicas: 1 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: nodeSelector: eks.amazonaws.com/compute-type: hybrid containers: - name: nginx image: nginx:latest ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: nginx spec: selector: app: nginx ports: - protocol: TCP port: 80 targetPort: 80 type: ClusterIP --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: nginx labels: name: nginx annotations: alb.ingress.kubernetes.io/scheme: internet-facing alb.ingress.kubernetes.io/target-type: ip spec: ingressClassName: alb rules: - http: paths: - pathType: Prefix path: "/" backend: service: name: nginx port: number: 80 --- - Controller logs/error messages while reproducing the issue:
{"level":"error","ts":"2025-08-21T13:17:46Z","msg":"Requesting network requeue due to error from ReconcileForPodEndpoints","tgb":{"name":"k8s-default-nginx-1234567890","namespace":"default"},"error":"providerID eks-hybrid:///ap-northeast-1/eks-test/mi-01c49bb1234567890 is invalid for EC2 instances, node: mi-01c49bb1234567890"} {"level":"info","ts":"2025-08-21T13:17:46Z","msg":"Requeue for networking requeue","tgb":{"name":"k8s-default-nginx-1234567890","namespace":"default"}} Expected Behavior
ALB forwarding traffic to Nginx pod, no errors in controller logs, TargetGroupBinding resource is healthy,
Actual Behavior
It works, but controller logs and k8s events are flooded with reconcile errors.
Current Workarounds
Environment
- AWS Load Balancer controller version: v2.13.3
- Kubernetes version: 1.32
- Using EKS (yes/no), if so version?: Yes, 1.32
- Using Service or Ingress: Ingress
Possible Solution (Optional)
Hybrid nodes use different ProviderID and InstanceID format (mi-)
Hybrid node: eks-hybrid:///ap-northeast-1/eks-test/mi-03a43a81234567890
EC2 node: aws:///ap-northeast-1c/i-05c8de21234567890
So this check is not working:
Hybrid pods and nodes don't have ENIs, so the resolver should account for this.
Contribution Intention (Optional)
- Yes, I'm willing to submit a PR to fix this issue
- No, I cannot work on a PR at this time
Additional Context