ALB rule limit is reached and the controller can no longer delete or edit any ingress #4373
Description
Bug Description
I'm using the ALB controller along with the group.name annotation. When the ALB reaches the rules limit, the alb controller fails to reconcile even if I delete an ingress to remove rules.
When it happens, I try to delete an ingress but it will never work. It only works if I manually delete the rules from the ALB.
How it usually happens:
- I add a new ingress and it never reconciles, never get the ALB address.
- I check the group.name, check the ALB and it has reached the limit
- In my case, we've extended it to 200 rules per ALB
- I try to delete the recent created or any other ingress but it never happens.
- I go to the aws console and manually remove the rules.
My theory
I'm not completely sure how's the reconciling process but looks like when the ingress is deleted the alb controller move its rule orders (higher than 49900), re-create the left rules with the new priority and then delete the ingress rules. As the ALB has no more space left, it fails.
Steps to Reproduce
- Step-by-step guide to reproduce the bug:
- Manifests applied while reproducing the issue:
- Controller logs/error messages while reproducing the issue:
- Considering the default ALB limit, which is 100 rules:
- Create 20 ingress with 5 host + path based rules each
- All within the same
group.name - It's better to apply 18 or less at once and then one by one apply the other until it reaches the limit
- Wait for it to reach the rules limit the alb controller reconciliation
- Get any rule from the middle of the rules list, identify its ingress
- It will fail too if a rule (via alb controller when deleting the the ingress) from the beginning of the list is deleted
- It WILL NOT fail if a rule from the end of the list is deleted (via alb controller when deleting the the ingress) .
- Delete the ingress
- The alb controller will start to log the error
Example of a manifest:
apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-west-2:xxxxx:certificate/xxxxxx5d alb.ingress.kubernetes.io/group.name: alb-ingress alb.ingress.kubernetes.io/healthcheck-interval-seconds: "15" alb.ingress.kubernetes.io/healthcheck-path: /health-check alb.ingress.kubernetes.io/healthcheck-port: traffic-port alb.ingress.kubernetes.io/healthcheck-protocol: HTTP alb.ingress.kubernetes.io/healthcheck-timeout-seconds: "5" alb.ingress.kubernetes.io/healthy-threshold-count: "2" alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS": 443}]' alb.ingress.kubernetes.io/load-balancer-attributes: access_logs.s3.enabled=true,access_logs.s3.bucket=alb-access-logs alb.ingress.kubernetes.io/scheme: internal alb.ingress.kubernetes.io/security-groups: alb alb.ingress.kubernetes.io/success-codes: "200" alb.ingress.kubernetes.io/tags: Environment=dev alb.ingress.kubernetes.io/target-group-attributes: deregistration_delay.timeout_seconds=60 alb.ingress.kubernetes.io/target-type: ip alb.ingress.kubernetes.io/unhealthy-threshold-count: "2" kubernetes.io/ingress.class: alb labels: app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: testing-ingress app.kubernetes.io/version: 4eb84e2 helm.sh/chart: 0.1.1 name: testing-ingress namespace: default spec: rules: - host: testing-ingress1.mycompany.com http: paths: - backend: service: name: testing-ingress-svc port: number: 8080 path: /* pathType: ImplementationSpecific - host: testing-ingress2.mycompany.com http: paths: - backend: service: name: testing-ingress-svc port: number: 8080 path: /* pathType: ImplementationSpecific - host: testing-ingress3.mycompany.com http: paths: - backend: service: name: testing-ingress-svc port: number: 8080 path: /* pathType: ImplementationSpecific - host: testing-ingress4.mycompany.com http: paths: - backend: service: name: testing-ingress-svc port: number: 8080 path: /* pathType: ImplementationSpecific - host: testing-ingress5.mycompany.com http: paths: - backend: service: name: testing-ingress-svc port: number: 8080 path: /* pathType: ImplementationSpecific It will look like this
FailedDeployModel(combined from similar events): Failed deploy model due to failed to create listener rule: operation error Elastic Load Balancing v2: CreateRule, https response error StatusCode: 400, RequestID: 3ac6077d-b729-4044-b11e-3a6d0a30e300, TooManyRules: The maximum number of rules on load balancer arn:aws:elasticloadbalancing:us-west-2:xxxxx:loadbalancer/app/alb-06f89267c4/de3ebac377442033 has been reached Expected Behavior
It should not break when the ALB limit is reached.
If the limit is 100 rules, it won't apply new ingresses to the same ALB when the limit is reached. Then, it should capable to remove rules even if the limit is reached.
Actual Behavior
Whenever the limit is reached, the alb controller is no longer capable of deleting, updating and reconciling that specific ALB.
Other ALBs are not affected.
Regression
Was the functionality working correctly in a previous version ?
I don't know
Current Workarounds
We have to manually deleted the rules of the specific ingress on the AWS Console.
Environment
-
AWS Load Balancer controller version: 2.13.4
-
Kubernetes version: 1.30
-
Using EKS (yes/no), if so version?: yes, eks.44
-
Using Service or Ingress: ingress
-
AWS region: us-west-2
-
How was the aws-load-balancer-controller installed:
It was deployed via Argo. The values file is pretty simple.
- Current state of the Controller configuration:
kubectl -n <controllernamespace> describe deployment aws-load-balancer-controller
Name: aws-load-balancer-controller Namespace: kube-system Labels: app.kubernetes.io/instance=aws-load-balancer-controller app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=aws-load-balancer-controller app.kubernetes.io/version=v2.13.4 helm.sh/chart=aws-load-balancer-controller-1.13.4 Annotations: argocd.argoproj.io/tracking-id: aws-load-balancer-controller:apps/Deployment:kube-system/aws-load-balancer-controller deployment.kubernetes.io/revision: 15 Selector: app.kubernetes.io/instance=aws-load-balancer-controller,app.kubernetes.io/name=aws-load-balancer-controller Replicas: 2 desired | 2 updated | 2 total | 2 available | 0 unavailable StrategyType: RollingUpdate MinReadySeconds: 0 RollingUpdateStrategy: 25% max unavailable, 25% max surge Pod Template: Labels: app.kubernetes.io/instance=aws-load-balancer-controller app.kubernetes.io/name=aws-load-balancer-controller Annotations: kubectl.kubernetes.io/restartedAt: 2025-07-17T18:06:57-07:00 Service Account: aws-load-balancer-controller Containers: aws-load-balancer-controller: Image: registry/aws-load-balancer-controller:v2.13.4 Ports: 8080/TCP Host Ports: 0/TCP, 0/TCP Args: --cluster-name=dev --ingress-class=alb --aws-region=us-west-2 --aws-vpc-id=vpc-xxx Liveness: http-get http://:61779/healthz delay=30s timeout=10s period=10s #success=1 #failure=2 Readiness: http-get http://:61779/readyz delay=10s timeout=10s period=10s #success=1 #failure=2 Mounts: /tmp/k8s-webhook-server/serving-certs from cert (ro) Volumes: cert: Type: Secret (a volume populated by a Secret) SecretName: aws-load-balancer Optional: false Priority Class Name: system-cluster-critical Node-Selectors: <none> Tolerations: <none> Conditions: Type Status Reason ---- ------ ------ Progressing True NewReplicaSetAvailable Available True MinimumReplicasAvailable OldReplicaSets: aws-load-balancer-controller-6456c88599 (0/0 replicas created), aws-load-balancer-controller-6f85cb9fd8 (0/0 replicas created), aws-load-balancer-controller-5fdf9f7769 (0/0 replicas created), aws-load-balancer-controller-57dd747cfb (0/0 replicas created), aws-load-balancer-controller-7c79747cb9 (0/0 replicas created), aws-load-balancer-controller-7755f96d5d (0/0 replicas created), aws-load-balancer-controller-79565cff89 (0/0 replicas created), aws-load-balancer-controller-777bc7cdcb (0/0 replicas created), aws-load-balancer-controller-6c9897877c (0/0 replicas created), aws-load-balancer-controller-59d9967f87 (0/0 replicas created) NewReplicaSet: aws-load-balancer-controller-5c44fbbf97 (2/2 replicas created) Events: <none> - Current state of the Ingress/Service configuration:
Name: alb Labels: app.kubernetes.io/instance=aws-load-balancer-controller app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=aws-load-balancer-controller app.kubernetes.io/version=v2.13.4 helm.sh/chart=aws-load-balancer-controller-1.13.4 Annotations: argocd.argoproj.io/tracking-id: aws-load-balancer-controller:networking.k8s.io/IngressClass:kube-system/alb Controller: ingress.k8s.aws/alb Events: <none> Possible Solution (Optional)
Contribution Intention (Optional)
- Yes, I'm willing to submit a PR to fix this issue
- No, I cannot work on a PR at this time
Additional Context