Nginx Controller does not update configuration while recreating service. #11963
Description
What happened:
Nginx Controller does not update configuration while recreating service.
The problem reproduces under certain conditions:
- Ingress object with tsl-passthrough
- Delete deployment and service
- Recreate deployment and service (as fast as possible)
What you expected to happen:
Application accesable via ingress url
NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):
v1.10.1
v1.11.2
and any other
Kubernetes version (use kubectl version):
root@vm1:~# kubectl version
Client Version: v1.30.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.0
It is also Reproduced on 1.27 and other
Environment:
Dev, test, prod
OS
root@vm1:~# cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.1 LTS"
-
Kernel (e.g.
uname -a):
root@vm1:~# uname -a
Linux vm1 5.15.0-119-generic Improve how errors connecting to api-server are handled #129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux -
How was the ingress-nginx-controller installed:
You can reproduce problen on minikube. -
Current State of the controller:
root@vm1:~# kubectl describe ingressclasses Name: nginx Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx app.kubernetes.io/name=ingress-nginx Annotations: ingressclass.kubernetes.io/is-default-class: true Controller: k8s.io/ingress-nginx Events: <none> root@vm1:~# kubectl -n ingress-nginx get all -A -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES ingress-nginx pod/ingress-nginx-admission-create-slvdf 0/1 Completed 0 29m 10.244.0.5 minikube <none> <none> ingress-nginx pod/ingress-nginx-admission-patch-vv6gt 0/1 Completed 1 29m 10.244.0.4 minikube <none> <none> ingress-nginx pod/ingress-nginx-controller-5b787686df-gwzj6 1/1 Running 0 9m45s 10.244.0.7 minikube <none> <none> kube-system pod/coredns-7db6d8ff4d-d5q7q 1/1 Running 0 30m 10.244.0.2 minikube <none> <none> kube-system pod/coredns-7db6d8ff4d-mbw2q 1/1 Running 0 30m 10.244.0.3 minikube <none> <none> kube-system pod/etcd-minikube 1/1 Running 0 30m 192.168.49.2 minikube <none> <none> kube-system pod/kube-apiserver-minikube 1/1 Running 0 30m 192.168.49.2 minikube <none> <none> kube-system pod/kube-controller-manager-minikube 1/1 Running 0 31m 192.168.49.2 minikube <none> <none> kube-system pod/kube-proxy-fdgdt 1/1 Running 0 30m 192.168.49.2 minikube <none> <none> kube-system pod/kube-scheduler-minikube 1/1 Running 0 30m 192.168.49.2 minikube <none> <none> kube-system pod/storage-provisioner 1/1 Running 1 (30m ago) 30m 192.168.49.2 minikube <none> <none> NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR default service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 31m <none> ingress-nginx service/ingress-nginx-controller NodePort 10.100.205.88 <none> 80:31454/TCP,443:30567/TCP 29m app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx ingress-nginx service/ingress-nginx-controller-admission ClusterIP 10.111.234.227 <none> 443/TCP 29m app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx kube-system service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 31m k8s-app=kube-dns NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE CONTAINERS IMAGES SELECTOR kube-system daemonset.apps/kube-proxy 1 1 1 1 1 kubernetes.io/os=linux 31m kube-proxy registry.k8s.io/kube-proxy:v1.30.0 k8s-app=kube-proxy NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR ingress-nginx deployment.apps/ingress-nginx-controller 1/1 1 1 29m controller registry.k8s.io/ingress-nginx/controller:v1.10.1@sha256:e24f39d3eed6bcc239a56f20098878845f62baa34b9f2be2fd2c38ce9fb0f29e app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx kube-system deployment.apps/coredns 2/2 2 2 31m coredns registry.k8s.io/coredns/coredns:v1.11.1 k8s-app=kube-dns NAMESPACE NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR ingress-nginx replicaset.apps/ingress-nginx-controller-5b787686df 1 1 1 9m46s controller registry.k8s.io/ingress-nginx/controller:v1.10.1@sha256:e24f39d3eed6bcc239a56f20098878845f62baa34b9f2be2fd2c38ce9fb0f29e app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=5b787686df ingress-nginx replicaset.apps/ingress-nginx-controller-768f948f8f 0 0 0 29m controller registry.k8s.io/ingress-nginx/controller:v1.10.1@sha256:e24f39d3eed6bcc239a56f20098878845f62baa34b9f2be2fd2c38ce9fb0f29e app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=768f948f8f kube-system replicaset.apps/coredns-7db6d8ff4d 2 2 2 30m coredns registry.k8s.io/coredns/coredns:v1.11.1 k8s-app=kube-dns,pod-template-hash=7db6d8ff4d NAMESPACE NAME STATUS COMPLETIONS DURATION AGE CONTAINERS IMAGES SELECTOR ingress-nginx job.batch/ingress-nginx-admission-create Complete 1/1 36s 29m create registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.1@sha256:36d05b4077fb8e3d13663702fa337f124675ba8667cbd949c03a8e8ea6fa4366 batch.kubernetes.io/controller-uid=e80da85a-d5b7-41cb-a1d2-310539779154 ingress-nginx job.batch/ingress-nginx-admission-patch Complete 1/1 35s 29m patch registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.1@sha256:36d05b4077fb8e3d13663702fa337f124675ba8667cbd949c03a8e8ea6fa4366 batch.kubernetes.io/controller-uid=3463fa08-6ccc-42ac-9d65-f9908d84a943 root@vm1:~# kubectl -n ingress-nginx describe po ingress-nginx-controller-5b787686df-gwzj6 Name: ingress-nginx-controller-5b787686df-gwzj6 Namespace: ingress-nginx Priority: 0 Service Account: ingress-nginx Node: minikube/192.168.49.2 Start Time: Tue, 10 Sep 2024 17:54:37 +0000 Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx app.kubernetes.io/name=ingress-nginx gcp-auth-skip-secret=true pod-template-hash=5b787686df Annotations: <none> Status: Running IP: 10.244.0.7 IPs: IP: 10.244.0.7 Controlled By: ReplicaSet/ingress-nginx-controller-5b787686df Containers: controller: Container ID: docker://154dd2e5c09ae5440a3bf4d9f5047a5e9794aaef41013d93511f614952fbea04 Image: registry.k8s.io/ingress-nginx/controller:v1.10.1@sha256:e24f39d3eed6bcc239a56f20098878845f62baa34b9f2be2fd2c38ce9fb0f29e Image ID: docker-pullable://registry.k8s.io/ingress-nginx/controller@sha256:e24f39d3eed6bcc239a56f20098878845f62baa34b9f2be2fd2c38ce9fb0f29e Ports: 80/TCP, 443/TCP, 8443/TCP Host Ports: 80/TCP, 443/TCP, 0/TCP Args: /nginx-ingress-controller --election-id=ingress-nginx-leader --controller-class=k8s.io/ingress-nginx --watch-ingress-without-class=true --configmap=$(POD_NAMESPACE)/ingress-nginx-controller --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services --udp-services-configmap=$(POD_NAMESPACE)/udp-services --validating-webhook=:8443 --validating-webhook-certificate=/usr/local/certificates/cert --validating-webhook-key=/usr/local/certificates/key --enable-ssl-passthrough=true State: Running Started: Tue, 10 Sep 2024 17:54:41 +0000 Ready: True Restart Count: 0 Requests: cpu: 100m memory: 90Mi Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5 Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3 Environment: POD_NAME: ingress-nginx-controller-5b787686df-gwzj6 (v1:metadata.name) POD_NAMESPACE: ingress-nginx (v1:metadata.namespace) LD_PRELOAD: /usr/local/lib/libmimalloc.so Mounts: /usr/local/certificates/ from webhook-cert (ro) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-sr6jw (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready True ContainersReady True PodScheduled True Volumes: webhook-cert: Type: Secret (a volume populated by a Secret) SecretName: ingress-nginx-admission Optional: false kube-api-access-sr6jw: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true QoS Class: Burstable Node-Selectors: kubernetes.io/os=linux minikube.k8s.io/primary=true Tolerations: node-role.kubernetes.io/master:NoSchedule node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 10m default-scheduler 0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod. Normal Scheduled 10m default-scheduler Successfully assigned ingress-nginx/ingress-nginx-controller-5b787686df-gwzj6 to minikube Normal Pulled 10m kubelet Container image "registry.k8s.io/ingress-nginx/controller:v1.10.1@sha256:e24f39d3eed6bcc239a56f20098878845f62baa34b9f2be2fd2c38ce9fb0f29e" already present on machine Normal Created 10m kubelet Created container controller Normal Started 10m kubelet Started container controller Normal RELOAD 10m nginx-ingress-controller NGINX reload triggered due to a change in configuration root@vm1:~# kubectl -n ingress-nginx describe svc ingress-nginx-controller Name: ingress-nginx-controller Namespace: ingress-nginx Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx app.kubernetes.io/name=ingress-nginx Annotations: <none> Selector: app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx Type: NodePort IP Family Policy: SingleStack IP Families: IPv4 IP: 10.100.205.88 IPs: 10.100.205.88 Port: http 80/TCP TargetPort: http/TCP NodePort: http 31454/TCP Endpoints: 10.244.0.7:80 Port: https 443/TCP TargetPort: https/TCP NodePort: https 30567/TCP Endpoints: 10.244.0.7:443 Session Affinity: None External Traffic Policy: Cluster Events: <none> How to reproduce this issue**:
#Install minikube
minikube start --force
#Install Ingress
minikube addons enable ingress
#Edit ingress deployment for enable passthrough
kubectl edit deployments.apps -n ingress-nginx ingress-nginx-controller
Add arg
--enable-ssl-passthrough=true
Install an application
kubectl create ns test kubectl create deployment web --image=gcr.io/google-samples/hello-app:1.0 -n test kubectl expose deployment web --port=8080 -n test Create an ingress (please add any additional annotation required)
echo " apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: nginx.ingress.kubernetes.io/backend-protocol: 'HTTPS' nginx.ingress.kubernetes.io/enable-access-log: 'true' nginx.ingress.kubernetes.io/ssl-passthrough: 'true' name: example-ingress-https namespace: test spec: ingressClassName: nginx rules: - host: hello-world-https.example http: paths: - backend: service: name: web port: number: 8080 path: / pathType: Prefix " | kubectl apply -f - make a request
Check that its working
curl -k --resolve "hello-world-https.example:443:$( minikube ip )" -i https://hello-world-https.example
curl: (35) error:0A00010B:SSL routines::wrong version number
SSL error because our application is running over HTTP, this is normal.
We will prepare the manifests:
cat <<EOF > manifests.yaml --- apiVersion: apps/v1 kind: Deployment metadata: labels: app: web name: web namespace: test spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app: web strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: creationTimestamp: null labels: app: web spec: containers: - image: gcr.io/google-samples/hello-app:1.0 imagePullPolicy: IfNotPresent name: hello-app resources: {} terminationMessagePath: /dev/termination-log terminationMessagePolicy: File dnsPolicy: ClusterFirst restartPolicy: Always schedulerName: default-scheduler securityContext: {} terminationGracePeriodSeconds: 30 --- apiVersion: v1 kind: Service metadata: labels: app: web name: web namespace: test spec: internalTrafficPolicy: Cluster ipFamilies: - IPv4 ipFamilyPolicy: SingleStack ports: - port: 8080 protocol: TCP targetPort: 8080 selector: app: web sessionAffinity: None type: ClusterIP EOF cat <<EOF > run.sh kubectl delete deployment web -n test kubectl delete service web -n test kubectl create -f manifests.yaml EOF Now, we cat start test.
- Check service
kubectl get svc -n test
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
web ClusterIP 10.101.12.157 8080/TCP 3m15s
IP 10.101.12.157
Run our bash script:
bash run.sh
deployment.apps "web" deleted
service "web" deleted
deployment.apps/web created
service/web created
Check again:
curl -k --resolve "hello-world-https.example:443:$( minikube ip )" -i https://hello-world-https.example
curl: (35) error:0A000126:SSL routines::unexpected eof while reading
If we run tcpdump, we can see, that packets send to old service IP
tcpdump -nnvvS -i any host 10.101.12.157
tcpdump: data link type LINUX_SLL2
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
18:56:27.059554 vethda6e844 P IP (tos 0x0, ttl 63, id 50339, offset 0, flags [DF], proto TCP (6), length 60)
192.168.49.2.57792 > 10.101.12.157.8080: Flags [S], cksum 0x08db (incorrect -> 0x20bc), seq 3739768812, win 64240, options [mss 1460,sackOK,TS val 4031445813 ecr 0,nop,wscale 7], length 0
18:56:27.059554 br-8f45197f04bc In IP (tos 0x0, ttl 63, id 50339, offset 0, flags [DF], proto TCP (6), length 60)
192.168.49.2.57792 > 10.101.12.157.8080: Flags [S], cksum 0x08db (incorrect -> 0x20bc), seq 3739768812, win 64240, options [mss 1460,sackOK,TS val 4031445813 ecr 0,nop,wscale 7], length 0
18:56:27.059571 enp0s3 Out IP (tos 0x0, ttl 62, id 50339, offset 0, flags [DF], proto TCP (6), length 60)
10.0.2.15.57792 > 10.101.12.157.8080: Flags [S], cksum 0x233f (incorrect -> 0x0658), seq 3739768812, win 64240, options [mss 1460,sackOK,TS val 4031445813 ecr 0,nop,wscale 7], length 0
18:56:28.066013 vethda6e844 P IP (tos 0x0, ttl 63, id 50340, offset 0, flags [DF], proto TCP (6), length 60)
192.168.49.2.57792 > 10.101.12.157.8080: Flags [S], cksum 0x08db (incorrect -> 0x1ccd), seq 3739768812, win 64240, options [mss 1460,sackOK,TS val 4031446820 ecr 0,nop,wscale 7], length 0
18:56:28.066013 br-8f45197f04bc In IP (tos 0x0, ttl 63, id 50340, offset 0, flags [DF], proto TCP (6), length 60)
192.168.49.2.57792 > 10.101.12.157.8080: Flags [S], cksum 0x08db (incorrect -> 0x1ccd), seq 3739768812, win 64240, options [mss 1460,sackOK,TS val 4031446820 ecr 0,nop,wscale 7], length 0
If you change Ingress after that script, then It will work.