Skip to content

Conversation

maelvls
Copy link
Member

@maelvls maelvls commented Mar 14, 2025

Ref: VC-33564

Thanks to the change made in client-go (kubernetes/kubernetes#126387) which was released in v0.33.0-alpha.1, we can now hide most of the annoying logs "the server could not find the requested resource" that we are seeing coming from client-go (as Richard started doing in #624). These messages are now gone:

W0923 13:41:33.351182 1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.3/tools/cache/reflector.go:232: failed to list awspca.cert-manager.io/v1beta1, Resource=awspcaclusterissuers: the server could not find the requested resource 

I propose that we also group and rephrase the "Failed to complete initial sync" messages. Instead of:

E0314 11:52:37.513403 68680 run.go:224] "Failed to complete initial sync of DataGatherer" err="timed out waiting for Kubernetes caches to sync" logger="Run" kind="k8s-dynamic" name="k8s/googlecasissuers" E0314 11:52:37.513536 68680 run.go:224] "Failed to complete initial sync of DataGatherer" err="timed out waiting for Kubernetes caches to sync" logger="Run" kind="k8s-dynamic" name="k8s/googlecasclusterissuers" E0314 11:52:37.513608 68680 run.go:224] "Failed to complete initial sync of DataGatherer" err="timed out waiting for Kubernetes caches to sync" logger="Run" kind="k8s-dynamic" name="k8s/awspcaissuer" E0314 11:52:37.513631 68680 run.go:224] "Failed to complete initial sync of DataGatherer" err="timed out waiting for Kubernetes caches to sync" logger="Run" kind="k8s-dynamic" name="k8s/awspcaclusterissuers" E0314 11:52:37.513655 68680 run.go:224] "Failed to complete initial sync of DataGatherer" err="timed out waiting for Kubernetes caches to sync" logger="Run" kind="k8s-dynamic" name="k8s/gateways" E0314 11:52:37.513671 68680 run.go:224] "Failed to complete initial sync of DataGatherer" err="timed out waiting for Kubernetes caches to sync" logger="Run" kind="k8s-dynamic" name="k8s/virtualservices" E0314 11:52:37.513686 68680 run.go:224] "Failed to complete initial sync of DataGatherer" err="timed out waiting for Kubernetes caches to sync" logger="Run" kind="k8s-dynamic" name="k8s/routes" E0314 11:52:37.513699 68680 run.go:224] "Failed to complete initial sync of DataGatherer" err="timed out waiting for Kubernetes caches to sync" logger="Run" kind="k8s-dynamic" name="k8s/venaficlusterissuers" E0314 11:52:37.513715 68680 run.go:224] "Failed to complete initial sync of DataGatherer" err="timed out waiting for Kubernetes caches to sync" logger="Run" kind="k8s-dynamic" name="k8s/venafiissuers" E0314 11:52:37.513728 68680 run.go:224] "Failed to complete initial sync of DataGatherer" err="timed out waiting for Kubernetes caches to sync" logger="Run" kind="k8s-dynamic" name="k8s/fireflyissuers" 

I propose this:

I0314 11:23:23.729133 1 run.go:234] "Skipping datagatherers for CRDs that can't be found in Kubernetes" logger="Run" datagatherers=["k8s/googlecasissuers","k8s/googlecasclusterissuers","k8s/awspcaissuer","k8s/awspcaclusterissuers","k8s/gateways","k8s/virtualservices","k8s/routes","k8s/venaficlusterissuers","k8s/venafiissuers","k8s/fireflyissuers"] 

You will still see the permissions errors if the RBAC rules are configured properly:

I0314 11:15:39.060659 1 dynamic.go:283] "datagatherer informer has failed and is backing off" groupVersionResource="batch/v1, Resource=cronjobs" reason="failed to list batch/v1, Resource=cronjobs: cronjobs.batch is forbidden: User \"system:serviceaccount:venafi:venafi-kubernetes-agent\" cannot list resource \"cronjobs\" in API group \"batch\" at the cluster scope" 

Before:

I0314 10:23:37.815137 51397 run.go:59] "Starting" logger="Run" version="development" commit="" I0314 10:23:37.816949 51397 config.go:491] "ignoring the venafi-cloud.uploader_id field in the config file. This field is not needed in Venafi Cloud Key Pair Service Account mode." logger="Run" I0314 10:23:37.816961 51397 config.go:543] "Using period from config" logger="Run" period="5s" I0314 10:23:37.816968 51397 config.go:762] "Loading upload_path from \"venafi-cloud\" configuration." logger="Run" I0314 10:23:37.817201 51397 run.go:117] "Healthz endpoints enabled" logger="Run.APIServer" addr=":8081" path="/healthz" I0314 10:23:37.817211 51397 run.go:121] "Readyz endpoints enabled" logger="Run.APIServer" addr=":8081" path="/readyz" E0314 10:23:37.818092 51397 run.go:267] "Error messages will not show in the pod's events because the POD_NAME environment variable is empty" logger="Run" W0314 10:23:37.843138 51397 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list cas-issuer.jetstack.io/v1beta1, Resource=googlecasissuers: the server could not find the requested resource W0314 10:23:37.843552 51397 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list cas-issuer.jetstack.io/v1beta1, Resource=googlecasclusterissuers: the server could not find the requested resource W0314 10:23:37.843954 51397 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list awspca.cert-manager.io/v1beta1, Resource=awspcaissuers: the server could not find the requested resource W0314 10:23:37.844313 51397 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list awspca.cert-manager.io/v1beta1, Resource=awspcaclusterissuers: the server could not find the requested resource W0314 10:23:37.846003 51397 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list networking.istio.io/v1alpha3, Resource=gateways: the server could not find the requested resource W0314 10:23:37.847596 51397 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list networking.istio.io/v1alpha3, Resource=virtualservices: the server could not find the requested resource W0314 10:23:37.848910 51397 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list route.openshift.io/v1, Resource=routes: the server could not find the requested resource W0314 10:23:37.853975 51397 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list jetstack.io/v1alpha1, Resource=venaficlusterissuers: the server could not find the requested resource W0314 10:23:37.855899 51397 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list jetstack.io/v1alpha1, Resource=venafiissuers: the server could not find the requested resource W0314 10:23:37.856528 51397 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list firefly.venafi.com/v1, Resource=issuers: the server could not find the requested resource W0314 10:23:38.775155 51397 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list cas-issuer.jetstack.io/v1beta1, Resource=googlecasclusterissuers: the server could not find the requested resource W0314 10:23:38.950758 51397 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list jetstack.io/v1alpha1, Resource=venafiissuers: the server could not find the requested resource W0314 10:23:39.032244 51397 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list networking.istio.io/v1alpha3, Resource=virtualservices: the server could not find the requested resource W0314 10:23:39.077548 51397 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list networking.istio.io/v1alpha3, Resource=gateways: the server could not find the requested resource W0314 10:23:39.325673 51397 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list awspca.cert-manager.io/v1beta1, Resource=awspcaclusterissuers: the server could not find the requested resource W0314 10:23:39.346444 51397 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list cas-issuer.jetstack.io/v1beta1, Resource=googlecasissuers: the server could not find the requested resource W0314 10:23:39.374397 51397 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list firefly.venafi.com/v1, Resource=issuers: the server could not find the requested resource W0314 10:23:39.377718 51397 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list route.openshift.io/v1, Resource=routes: the server could not find the requested resource W0314 10:23:39.384133 51397 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list jetstack.io/v1alpha1, Resource=venaficlusterissuers: the server could not find the requested resource W0314 10:23:39.423535 51397 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list awspca.cert-manager.io/v1beta1, Resource=awspcaissuers: the server could not find the requested resource W0314 10:23:40.797040 51397 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list jetstack.io/v1alpha1, Resource=venafiissuers: the server could not find the requested resource W0314 10:23:40.872109 51397 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list networking.istio.io/v1alpha3, Resource=gateways: the server could not find the requested resource W0314 10:23:41.188538 51397 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list route.openshift.io/v1, Resource=routes: the server could not find the requested resource W0314 10:23:41.243909 51397 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list jetstack.io/v1alpha1, Resource=venaficlusterissuers: the server could not find the requested resource W0314 10:23:41.323751 51397 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list cas-issuer.jetstack.io/v1beta1, Resource=googlecasissuers: the server could not find the requested resource W0314 10:23:41.839867 51397 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list networking.istio.io/v1alpha3, Resource=virtualservices: the server could not find the requested resource W0314 10:23:41.898455 51397 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list awspca.cert-manager.io/v1beta1, Resource=awspcaissuers: the server could not find the requested resource W0314 10:23:41.939627 51397 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list cas-issuer.jetstack.io/v1beta1, Resource=googlecasclusterissuers: the server could not find the requested resource W0314 10:23:42.026522 51397 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list firefly.venafi.com/v1, Resource=issuers: the server could not find the requested resource W0314 10:23:42.167457 51397 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list awspca.cert-manager.io/v1beta1, Resource=awspcaclusterissuers: the server could not find the requested resource I0314 10:23:43.676680 51397 run.go:405] "Data sent successfully" logger="Run.gatherAndOutputData.postData" W0314 10:23:44.598935 51397 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list cas-issuer.jetstack.io/v1beta1, Resource=googlecasissuers: the server could not find the requested resource W0314 10:23:45.359057 51397 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list route.openshift.io/v1, Resource=routes: the server could not find the requested resource W0314 10:23:45.386967 51397 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list awspca.cert-manager.io/v1beta1, Resource=awspcaclusterissuers: the server could not find the requested resource W0314 10:23:45.573294 51397 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list firefly.venafi.com/v1, Resource=issuers: the server could not find the requested resource W0314 10:23:46.019229 51397 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list jetstack.io/v1alpha1, Resource=venafiissuers: the server could not find the requested resource W0314 10:23:46.490152 51397 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list jetstack.io/v1alpha1, Resource=venaficlusterissuers: the server could not find the requested resource W0314 10:23:47.230397 51397 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list networking.istio.io/v1alpha3, Resource=gateways: the server could not find the requested resource W0314 10:23:47.339176 51397 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list awspca.cert-manager.io/v1beta1, Resource=awspcaissuers: the server could not find the requested resource W0314 10:23:47.747392 51397 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list cas-issuer.jetstack.io/v1beta1, Resource=googlecasclusterissuers: the server could not find the requested resource W0314 10:23:48.150193 51397 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list networking.istio.io/v1alpha3, Resource=virtualservices: the server could not find the requested resource I0314 10:23:48.887302 51397 run.go:405] "Data sent successfully" logger="Run.gatherAndOutputData.postData"

After:

I0314 11:25:37.899805 1 run.go:59] "Starting" logger="Run" version="v1.4.0-18-g4de9e9bc53c65d-dirty" commit="4de9e9bc53c65db0acded209ef03a899bba55675" I0314 11:25:37.900582 1 config.go:491] "ignoring the venafi-cloud.uploader_id field in the config file. This field is not needed in Venafi Cloud Key Pair Service Account mode." logger="Run" I0314 11:25:37.900591 1 config.go:543] "Using period from config" logger="Run" period="1m0s" I0314 11:25:37.900598 1 config.go:762] "Loading upload_path from \"venafi-cloud\" configuration." logger="Run" I0314 11:25:37.900674 1 run.go:108] "Metrics endpoints enabled" logger="Run.APIServer" addr=":8081" path="/metrics" I0314 11:25:37.900707 1 run.go:117] "Healthz endpoints enabled" logger="Run.APIServer" addr=":8081" path="/healthz" I0314 11:25:37.900715 1 run.go:121] "Readyz endpoints enabled" logger="Run.APIServer" addr=":8081" path="/readyz" I0314 11:25:42.906096 1 run.go:234] "Skipping datagatherers for CRDs that can't be found in Kubernetes" logger="Run" datagatherers=["k8s/certificates","k8s/certificaterequests","k8s/issuers","k8s/clusterissuers","k8s/googlecasissuers","k8s/googlecasclusterissuers","k8s/awspcaissuer","k8s/awspcaclusterissuers","k8s/gateways","k8s/virtualservices","k8s/routes","k8s/venaficlusterissuers","k8s/venafiissuers","k8s/fireflyissuers"] I0314 11:25:43.675886 1 run.go:413] "Data sent successfully" logger="Run.gatherAndOutputData.postData"

Manual Testing

There is no automated test for this change. Thus, I went ahead and manually tested the feature.

I've used the tenant https://ven-tlspk.venafi.cloud/. To access the API key, use the user system.admin@tlspk.qa.venafi.io and the password is visible in the page Production Accounts (private to Venafi). Then go to the settings and find the API key.

export APIKEY=...

Create the Private Key JWT:

venctl iam service-account agent create --name "$USER temp" \ --vcp-region US \ --output json \ --owning-team $(curl -sS https://api.venafi.cloud/v1/teams -H "tppl-api-key: $APIKEY" | jq '.teams[0].id') \ --output-file /tmp/agent-credentials.json \ --api-key $APIKEY

Build image and chart:

make docker-tarball-preflight \ oci_preflight_image_name_development=jetstack.local/preflight \ oci_preflight_image_tag=v0.0.1-dev \ oci_platforms=linux/arm64 kind load image-archive _bin/scratch/image/oci-layout-preflight.v0.0.1-dev.docker.tar rm -f _bin/scratch/image/venafi-kubernetes-agent-v0.0.1-dev.tgz make helm-chart \ oci_preflight_image_name=jetstack.local/preflight \ oci_preflight_image_tag=v0.0.1-dev \ helm_chart_version=v0.0.1-dev

Then, install the chart:

helm upgrade -i -n venafi --create-namespace venafi-kubernetes-agent _bin/scratch/image/venafi-kubernetes-agent-v0.0.1-dev.tgz \ --set config.clusterName="$USER temp" --set config.clientId="$(jq -r .client_id /tmp/agent-credentials.json)" --set image.pullPolicy=Never kubectl create secret generic -n venafi agent-credentials --from-literal=privatekey.pem="$(jq -r .private_key /tmp/agent-credentials.json)" \ --dry-run=client -o yaml | kubectl apply -f -

See that the annoying logs aren't shown anymore:

kubectl logs -n venafi -l app.kubernetes.io/instance=venafi-kubernetes-agent 
@maelvls maelvls force-pushed the VC-33564-hide-spurious-log-lines branch from e30960b to 08bd016 Compare March 14, 2025 11:40
@maelvls maelvls changed the title bump client-go to v0.33.0-alpha.1 which no longer shows annoying logs Hide the annoying logs coming from client-go Mar 14, 2025
@maelvls maelvls changed the title Hide the annoying logs coming from client-go Hide the annoying logs coming from client-go ("the server could not find the requested resource") Mar 14, 2025
@maelvls maelvls force-pushed the VC-33564-hide-spurious-log-lines branch 2 times, most recently from 08bd016 to 24d6b92 Compare March 14, 2025 12:23
go.mod Outdated
go 1.22.0
go 1.23.0

toolchain go1.23.4
Copy link
Member Author

@maelvls maelvls Mar 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The toolchain stanza kept re-appearing after running go mod tidy.

With the vendored version of Go, though, it wouldn't appear:

$ make vendor-go $ _bin/tools/go version  go version go1.23.4 darwin/arm64 $ make go-tidy
maelvls added 2 commits March 14, 2025 14:59
…ync" This commit groups the log messages about the datagatherers not being able to be synced, and also makes the log message less alarming.
@maelvls maelvls force-pushed the VC-33564-hide-spurious-log-lines branch from 24d6b92 to ae44ffc Compare March 14, 2025 13:59
@maelvls maelvls marked this pull request as ready for review March 14, 2025 14:02
Copy link
Contributor

@SgtCoDFish SgtCoDFish left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve
/hold

Looks reasonable to me. The obvious thing which stands out is relying on an alpha version of client-go, but that doesn't seem like a huge risk to me.

I added a hold in case you wanted further review, but I'd be happy to merge this!

@maelvls
Copy link
Member Author

maelvls commented Mar 14, 2025

Looks reasonable to me. The obvious thing which stands out is relying on an alpha version of client-go, but that doesn't seem like a huge risk to me.

Good point... I don't know if it makes it better or not, but controller-runtime's latest release uses v0.33.0-alpha.3... 😅

@maelvls maelvls merged commit fbab5e1 into master Mar 14, 2025
2 checks passed
@maelvls maelvls deleted the VC-33564-hide-spurious-log-lines branch March 14, 2025 14:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

2 participants