Operator recreates additional vminsert service over and over again #1550
Description
Description
Victoria-metrics-operator is recreating the vminsert-vm-victoria-metrics-k8s-stack-additional-service every ~10-15 seconds for an unknown reason. what is also strange is, we are currently moving from Azure AKS to AWS eks, and this is not happening on AKS but on EKS. I would be really thankful if someone can help me with this.
Environment information
k8s version: v1.33.5
helm chart: victoria-metrics-k8s-stack-0.60.1
vm operator version: v0.63.0
cloud: aws
Config
Snippet of the code for vminsert
vminsert: replicaCount: 2 serviceSpec: metadata: annotations: external-dns.alpha.kubernetes.io/hostname: "vminsert-aws.{{ requiredEnv "DOMAIN" }}." service.beta.kubernetes.io/aws-load-balancer-scheme: internal service.beta.kubernetes.io/aws-load-balancer-subnets: {{ $terraformOutputs.pods_subnet_ids.value | join "," }} spec: type: LoadBalancerSnippet of the code for vmoperator
- victoria-metrics-operator: logLevel: "debug" crd: cleanup: enabled: false extraArgs: controller.maxConcurrentReconciles: 5 env: - name: VM_ENABLESTRICTSECURITY value: true - name: VM_VMAGENTDEFAULT_CONFIGRELOADERMEMORY value: 50Mi - name: VM_USECUSTOMCONFIGRELOADER value: true podSecurityContext: seccompProfile: type: "RuntimeDefault" securityContext: allowPrivilegeEscalation: false runAsNonRoot: true runAsUser: 65534 runAsGroup: 65534 capabilities: drop: - "ALL" operator: disable_prometheus_converter: false enable_converter_ownership: true admissionWebhooks: certManager: enabled: trueVictoria-metrics-operator logs
{"level":"info","ts":"2025-10-17T06:18:12Z","logger":"controller.VMCluster","msg":"check if pod update needed to desiredVersion=vmstorage-vm-victoria-metrics-k8s-stack-75c85747d8, podMustRecreate=false","vmcluster":"vm-victoria-metrics-k8s-stack","namespace":"victoria-metrics"} {"level":"info","ts":"2025-10-17T06:18:12Z","logger":"controller.VMCluster","msg":"no pod needs to be updated","vmcluster":"vm-victoria-metrics-k8s-stack","namespace":"victoria-metrics"} {"level":"info","ts":"2025-10-17T06:18:12Z","logger":"controller.VMAlertmanager","msg":"check if pod update needed to desiredVersion=vmalertmanager-vm-victoria-metrics-k8s-stack-f94878d9c, podMustRecreate=false","vmalertmanager":"vm-victoria-metrics-k8s-stack","namespace":"victoria-metrics"} {"level":"info","ts":"2025-10-17T06:18:12Z","logger":"controller.VMAlertmanager","msg":"no pod needs to be updated","vmalertmanager":"vm-victoria-metrics-k8s-stack","namespace":"victoria-metrics"} {"level":"info","ts":"2025-10-17T06:18:17Z","logger":"controller.VMCluster","msg":"check if pod update needed to desiredVersion=vmselect-vm-victoria-metrics-k8s-stack-56b9b9f5bf, podMustRecreate=false","vmcluster":"vm-victoria-metrics-k8s-stack","namespace":"victoria-metrics"} {"level":"info","ts":"2025-10-17T06:18:17Z","logger":"controller.VMCluster","msg":"no pod needs to be updated","vmcluster":"vm-victoria-metrics-k8s-stack","namespace":"victoria-metrics"} {"level":"info","ts":"2025-10-17T06:18:17Z","logger":"controller.VMCluster","msg":"creating new Service vminsert-vm-victoria-metrics-k8s-stack-additional-service","vmcluster":"vm-victoria-metrics-k8s-stack","namespace":"victoria-metrics"} {"level":"info","ts":"2025-10-17T06:18:22Z","logger":"controller.VMCluster","msg":"check if pod update needed to desiredVersion=vmstorage-vm-victoria-metrics-k8s-stack-75c85747d8, podMustRecreate=false","vmcluster":"vm-victoria-metrics-k8s-stack","namespace":"victoria-metrics"} {"level":"info","ts":"2025-10-17T06:18:22Z","logger":"controller.VMCluster","msg":"no pod needs to be updated","vmcluster":"vm-victoria-metrics-k8s-stack","namespace":"victoria-metrics"} {"level":"info","ts":"2025-10-17T06:18:27Z","logger":"controller.VMAlert","msg":"selected VMRules count=69, invalid rules count=0, namespaced names victoria-metrics/rga-next-alerts,victoria-metrics/vm-victoria-metrics-k8s-stack-general.rules,victoria-metrics/vm-victoria-metrics-k8s-stack-k8s.rules.containermemoryswap,victoria-metrics/vm-victoria-metrics-k8s-stack-k8s.rules.podowner,victoria-metrics/vm-victoria-metrics-k8s-stack-kube-prometheus-general.rules,victoria-metrics/ingress-nginx-alerts,victoria-metrics/vm-victoria-metrics-k8s-stack-k8s.rules.containermemorycache,victoria-metrics/vm-victoria-metrics-k8s-stack-k8s.rules.containermemoryworkingsetbytes,victoria-metrics/vm-victoria-metrics-k8s-stack-kube-apiserver-slos,victoria-metrics/argocd-alerts,victoria-metrics/pod-resource-alerts,victoria-metrics/vm-victoria-metrics-k8s-stack-k8s.rules.containercpuusagesecondstotal,victoria-metrics/vm-victoria-metrics-k8s-stack-k8s.rules.containermemoryrequests,victoria-metrics/vm-victoria-metrics-k8s-stack-vmcluster,victoria-metrics/cnpg-alerts-custom,victoria-metrics/core-shared-alerts,victoria-metrics/vm-victoria-metrics-k8s-stack-vm-health,loki/loki-scalable-loki-alerts,victoria-metrics/vm-victoria-metrics-k8s-stack-k8s.rules.containermemoryrss,victoria-metrics/vm-victoria-metrics-k8s-stack-kubernetes-system,victoria-metrics/vm-victoria-metrics-k8s-stack-kubernetes-system-apiserver,victoria-metrics/vm-victoria-metrics-k8s-stack-kube-state-metrics,victoria-metrics/vm-victoria-metrics-k8s-stack-node-exporter.rules,victoria-metrics/vm-victoria-metrics-k8s-stack-vmsingle,victoria-metrics/loki-alerts,victoria-metrics/sealed-secrets-alerts,victoria-metrics/tempo-alerts,victoria-metrics/vm-victoria-metrics-k8s-stack-kube-apiserver-availability.rules,victoria-metrics/vm-victoria-metrics-k8s-stack-kube-apiserver-histogram.rules,victoria-metrics/vm-victoria-metrics-k8s-stack-kubernetes-system-kubelet,victoria-metrics/vm-victoria-metrics-k8s-stack-vmagent,loki/loki-scalable-loki-rules,velero/velero-alerts,victoria-metrics/vm-victoria-metrics-k8s-stack-kubernetes-resources,victoria-metrics/vm-victoria-metrics-k8s-stack-kubernetes-storage,victoria-metrics/vm-victoria-metrics-k8s-stack-node-exporter,victoria-metrics/loki-alerts-custom,victoria-metrics/vm-victoria-metrics-k8s-stack-kube-apiserver-burnrate.rules,victoria-metrics/vm-victoria-metrics-k8s-stack-node-network,victoria-metrics/vm-victoria-metrics-k8s-stack-vmoperator,victoria-metrics/artifact-archiving-alerts,victoria-metrics/external-secrets-operator-alerts,victoria-metrics/rga-de-single-sign-out-dispatcher-alerts,victoria-metrics/postgres-alerts,victoria-metrics/vm-victoria-metrics-k8s-stack-k8s.rules.containercpulimits,victoria-metrics/vm-victoria-metrics-k8s-stack-k8s.rules.containermemorylimits,victoria-metrics/vm-victoria-metrics-k8s-stack-kubelet.rules,victoria-metrics/cert-manager-alerts,victoria-metrics/kube-proxy-alerts,victoria-metrics/prometheus-alerts,victoria-metrics/tempo-rules,victoria-metrics/vm-victoria-metrics-k8s-stack-alertmanager.rules,victoria-metrics/rga-de-one-configuration-provider-alerts,victoria-metrics/victoria-metrics-k8s-stack-kubecost,victoria-metrics/api-integrations-shared-alerts,victoria-metrics/persistentvolume-alerts,victoria-metrics/vm-victoria-metrics-k8s-stack-kube-prometheus-node-recording.rules,victoria-metrics/external-dns-alerts,victoria-metrics/node-exporter-alerts,victoria-metrics/vm-victoria-metrics-k8s-stack-kubernetes-apps,grafana-alloy/alloy-logs,victoria-metrics/grafana-alloy-alerts,victoria-metrics/rga-de-bridge-gateway-alerts,victoria-metrics/rbac-manager-alerts,victoria-metrics/vm-victoria-metrics-k8s-stack-k8s.rules.containercpurequests,victoria-metrics/cnpg-alerts,victoria-metrics/kube-api-alerts,victoria-metrics/kubernetes-node-alerts,victoria-metrics/vm-victoria-metrics-k8s-stack-node.rules","vmalert":"vm-victoria-metrics-k8s-stack","namespace":"victoria-metrics"} {"level":"info","ts":"2025-10-17T06:18:27Z","logger":"controller.VMCluster","msg":"check if pod update needed to desiredVersion=vmselect-vm-victoria-metrics-k8s-stack-56b9b9f5bf, podMustRecreate=false","vmcluster":"vm-victoria-metrics-k8s-stack","namespace":"victoria-metrics"} {"level":"info","ts":"2025-10-17T06:18:27Z","logger":"controller.VMCluster","msg":"discovered already updated pods=1, pods needed to be update=0","vmcluster":"vm-victoria-metrics-k8s-stack","namespace":"victoria-metrics"} {"level":"info","ts":"2025-10-17T06:18:27Z","logger":"controller.VMCluster","msg":"checking ready status for already updated pod vmselect-vm-victoria-metrics-k8s-stack-0 to revision version=\"vmselect-vm-victoria-metrics-k8s-stack-56b9b9f5bf\"","vmcluster":"vm-victoria-metrics-k8s-stack","namespace":"victoria-metrics"} {"level":"info","ts":"2025-10-17T06:19:07Z","logger":"controller.VMCluster","msg":"finished statefulset update from revision=\"vmselect-vm-victoria-metrics-k8s-stack-7898b8b45d\" to revision=\"vmselect-vm-victoria-metrics-k8s-stack-56b9b9f5bf\"","vmcluster":"vm-victoria-metrics-k8s-stack","namespace":"victoria-metrics"} {"level":"info","ts":"2025-10-17T06:19:07Z","logger":"controller.VMCluster","msg":"recreating new Service vminsert-vm-victoria-metrics-k8s-stack-additional-service","vmcluster":"vm-victoria-metrics-k8s-stack","namespace":"victoria-metrics"} {"level":"error","ts":"2025-10-17T06:19:07Z","logger":"manager","msg":"Reconciler error","controller":"vmcluster","controllerGroup":"operator.victoriametrics.com","controllerKind":"VMCluster","VMCluster":{"name":"vm-victoria-metrics-k8s-stack","namespace":"victoria-metrics"},"namespace":"victoria-metrics","name":"vm-victoria-metrics-k8s-stack","reconcileID":"3c91ee97-66f0-4a80-840d-a09e17458cf3","error":"failed create or update vmcluster: cannot reconcile vminsert additional service: cannot create service at recreate: object is being deleted: services \"vminsert-vm-victoria-metrics-k8s-stack-additional-service\" already exists"} {"level":"info","ts":"2025-10-17T06:19:08Z","logger":"controller.VMAgent","msg":"selected VMServiceScrape count=36, invalid rules count=0, namespaced names argocd/argocd-application-controller,argocd/argocd-notifications-controller,argocd/argocd-redis-ha,argocd/argocd-redis-ha-haproxy,argocd/argocd-repo-server,argocd/argocd-server,cert-manager/cert-manager,external-dns/global-cf-external-dns,external-secrets-operator/external-secrets-operator-cert-controller-metrics,external-secrets-operator/external-secrets-operator-metrics,external-secrets-operator/external-secrets-operator-webhook-metrics,grafana-alloy/grafana-alloy-aws,grafana-alloy/grafana-alloy-logs-daemonset,grafana-alloy/grafana-alloy-tempo,ingress-nginx/internal-nginx-ingress-nginx-controller,ingress-nginx/root-ingress-nginx-controller,loki/loki-scalable,node-exporter/vm-prometheus-node-exporter,sealed-secrets-controller/sealed-secrets-controller,tempo/grafana-tempo,traefik/traefik,velero/velero,victoria-metrics/vm-kube-state-metrics,victoria-metrics/vm-victoria-metrics-k8s-stack-core-dns,victoria-metrics/vm-victoria-metrics-k8s-stack-grafana,victoria-metrics/vm-victoria-metrics-k8s-stack-kube-api-server,victoria-metrics/vm-victoria-metrics-k8s-stack-kube-proxy,victoria-metrics/vm-victoria-metrics-k8s-stack-kube-state-metrics,victoria-metrics/vm-victoria-metrics-k8s-stack-prometheus-node-exporter,victoria-metrics/vm-victoria-metrics-operator,victoria-metrics/vmagent-vm-victoria-metrics-k8s-stack,victoria-metrics/vmalert-vm-victoria-metrics-k8s-stack,victoria-metrics/vmalertmanager-vm-victoria-metrics-k8s-stack,victoria-metrics/vminsert-vm-victoria-metrics-k8s-stack,victoria-metrics/vmselect-vm-victoria-metrics-k8s-stack,victoria-metrics/vmstorage-vm-victoria-metrics-k8s-stack","vmagent":"vm-victoria-metrics-k8s-stack","namespace":"victoria-metrics"} {"level":"info","ts":"2025-10-17T06:19:08Z","logger":"controller.VMAgent","msg":"selected VMPodScrapes count=2, invalid rules count=0, namespaced names cnpg-system/cnpg-cloudnative-pg,victoria-metrics/postgres-cluster","vmagent":"vm-victoria-metrics-k8s-stack","namespace":"victoria-metrics"} {"level":"info","ts":"2025-10-17T06:19:08Z","logger":"controller.VMAgent","msg":"selected VMNodeScrapes count=4, invalid rules count=0, namespaced names victoria-metrics/vm-victoria-metrics-k8s-stack-cadvisor,victoria-metrics/vm-victoria-metrics-k8s-stack-kubelet,victoria-metrics/vm-victoria-metrics-k8s-stack-probes,victoria-metrics/vm-victoria-metrics-k8s-stack-resources","vmagent":"vm-victoria-metrics-k8s-stack","namespace":"victoria-metrics"} {"level":"info","ts":"2025-10-17T06:19:08Z","logger":"controller.VMAgent","msg":"using cluster version of VMAgent with shards count=2","vmagent":"vm-victoria-metrics-k8s-stack","namespace":"victoria-metrics"} {"level":"info","ts":"2025-10-17T06:19:12Z","logger":"controller.VMCluster","msg":"check if pod update needed to desiredVersion=vmstorage-vm-victoria-metrics-k8s-stack-75c85747d8, podMustRecreate=false","vmcluster":"vm-victoria-metrics-k8s-stack","namespace":"victoria-metrics"} {"level":"info","ts":"2025-10-17T06:19:12Z","logger":"controller.VMCluster","msg":"no pod needs to be updated","vmcluster":"vm-victoria-metrics-k8s-stack","namespace":"victoria-metrics"}