api_server.cc:184 Metadata request unsuccessful: Server responded with 'Bad Request' (400): Transport endpoint is not connected

Question

We are using google cloud instances with container option, we are seeing below error every minute in stack driver logs. we are not sure what is this error for.

api_server.cc:184 Metadata request unsuccessful: Server responded with 'Bad Request' (400): Transport endpoint is not connected

I'm thinking this is released cloud instance metadata service api, where instance details will available. Also in our one of use cases we were using gcloud command line tool in instance(inside docker container), gcloud tool does not have to access to cloud apis for first 2-5 minutes even after docker container started. in those 2-5 minutes it shows something like service account is not available.

I would like to know about this error, unable to find any relevant details in google search.

Answer 1

We've seen this similar behavior as well. I sent the following email to google:

Hello,

I’ve been seeing some inconsistent behavior when using gcloud beta compute instances create-with-container and was wondering if you may have seen something like this before:

I have a docker image (Dockerfile below) that I build and push to a container registry. In the entrypoint to the docker container I execute a script that uses å gcloud kms command to decrypt a ciphertext and gcloud.compute.instances.delete to delete an instance. If I try to run the image using gcloud beta compute instances create-with-container immediately after pushing a new image. The gcloud command will error with an error saying something like:

"\u001b[1;31mERROR:\u001b[0m (gcloud.kms.decrypt) The required property [project] is not currently set.\r"

"You may set it for your current workspace by running:\r"

"\r"

" $ gcloud config set project VALUE\r"

"\r"

"or it can be set temporarily by the environment variable [CLOUDSDK_CORE_PROJECT]\r"

Or

"\u001b[1;31mERROR:\u001b[0m (gcloud.compute.instances.delete) You do not currently have an active account selected.\r"

"Please run:\r"

"\r"

" $ gcloud auth login\r"

"\r"

"to obtain new credentials, or if you have already logged in with a\r"

"different account:\r"

"\r"

" $ gcloud config set account ACCOUNT\r"

"\r"

"to select an already authenticated account to use.\r"

If I wait approx. 3-4 minutes and run the exact same image with the exact same command, then the script will successfully run as expected. It seems to me that there is some delay in setting up the authentication for gcloud – is that the case? And do you have any recommended way to mitigate this behavior?

And got the following Response:

Thanks for reaching out and detailed report. We'll make sure to investigate this further on our end as it's quite likely there is some misalignment between setting up accounts and starting the container. Unfortunately, I don't yet have a good workaround for you other than adding a simple "sleep" command before you issue the first gcloud call from a newly created container-in-VM. I will follow up when I know more or have a solution for this issue.