0

I have a few microservices running in Container Engine (GKE) and I'm trying to get trace information in Google Console, but something goes wrong.

Here is my checklist:

  • Stackdriver Trace API is enabled in API Manager.
  • API Manager dashboard shows 99.98% error ratio.
  • GKE has permissions:
    • Stackdriver Trace: Write Only
    • Stackdriver Logging API: Write Only
    • Stackdriver Monitoring API: Full
    • Service Control: Enabled
  • There are no errors in logs
  • I used the following manuals to integrate Trace API:

Did I miss something? Thanks in advance.

api manager errors

Update: I was able to query trace api manually via curl from GKE pod:

kubectl exec -it POD -- /bin/bash curl "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token" -H "Metadata-Flavor: Google" curl --verbose -d '{"traces": [{"projectId":"xxxx","traceId":"12345678901234567890123456789053","spans":[{"spanId":3,"name":"test"}]}]}' -X PATCH https://cloudtrace.googleapis.com/v1/projects/xxxx/traces -H "Content-Type: application/json" -H "Authorization":"Bearer TOKEN"

curl output:

  • Hostname was NOT found in DNS cache
  • Trying 173.194.202.95...
  • Connected to cloudtrace.googleapis.com (173.194.202.95) port 443 (#0)
  • successfully set certificate verify locations:
  • CAfile: none CApath: /etc/ssl/certs
  • SSLv3, TLS handshake, Client hello (1):
  • SSLv3, TLS handshake, Server hello (2):
  • SSLv3, TLS handshake, CERT (11):
  • SSLv3, TLS handshake, Server key exchange (12):
  • SSLv3, TLS handshake, Server finished (14):
  • SSLv3, TLS handshake, Client key exchange (16):
  • SSLv3, TLS change cipher, Client hello (1):
  • SSLv3, TLS handshake, Finished (20):
  • SSLv3, TLS change cipher, Client hello (1):
  • SSLv3, TLS handshake, Finished (20):
  • SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
  • Server certificate:
  • subject: C=US; ST=California; L=Mountain View; O=Google Inc; CN=*.googleapis.com
  • start date: 2017-07-05 08:20:33 GMT
  • expire date: 2017-09-27 08:09:00 GMT
  • subjectAltName: cloudtrace.googleapis.com matched
  • issuer: C=US; O=Google Inc; CN=Google Internet Authority G2
  • SSL certificate verify ok.

    PATCH /v1/projects/line-b/traces HTTP/1.1 User-Agent: curl/7.38.0 Host: cloudtrace.googleapis.com Accept: / Content-Type: application/json Authorization:Bearer TOKEN Content-Length: 118

  • upload completely sent off: 118 out of 118 bytes < HTTP/1.1 200 OK < Content-Type: application/json; charset=UTF-8 ...

Dashboard also updated:

enter image description here

Improve this question
2
  • Can you provide your actual code where you are using the Stackdriver Trace API. Also provide your project ID if you are comfortable sharing that, as it will help on my end to see if I can uncover anything. If not, you can open an issue report to Google and provide the above information. Commented Jul 7, 2017 at 19:30
  • @Jordan created issue report - 63444729 Commented Jul 7, 2017 at 19:59

1 Answer 1

Reset to default
4

You can look directly at the API reporting graphs to see that your calls to google.devtools.cloudtrace.v1.TraceService.PatchTraces fail with 403 Forbidden errors.

403 Forbidden is caused when a request to a server is not authorized. Therefore, your calls from your Container Engine (GKE) cluster to 'cloudtrace.PatchTraces' are not authorized.

This could be due to the limited Stackdriver Trace 'Write Only' permission you set. Also, ensure that you added the 'trace.append' scope when creating the cluster.

Concerning the missing logs in Stackdriver. Once Stackdriver Logging is enabled for your cluster you should be able to simply write to 'STDOUT' and 'STDERR'. Fluentd should take care of sending this to Stackdriver and the output should be available in the Log Viewer under the 'GKE Container' dropdown.

Alternatively you can always use the Stackdriver Logging Client Library to directly write to Stackdriver, just as you are doing with the Stackdriver Trace API. You should also set the Log Severity to make filtering easier in the Log Viewer to find specific error logs (like the 403s you are seeing).

Improve this answer
4
  • this is not permission problem, all instances has "Allow full access to all Cloud APIs". Moreover I was able to send PatchTraces manually via curl. As for logging it' set to "Write Only" which should be enough to send logs (I receive logs data from containers). Commented Jul 14, 2017 at 0:42
  • How are you authenticating the cURL request? Could you also provide the sample cURL code that is working for you in your original answer. Commented Jul 14, 2017 at 17:00
  • yes, see update Commented Jul 15, 2017 at 22:00
  • Since you have confirmed that it is indeed working with the Default Service Account, I ask that you triple check that you have properly followed all of the steps for Setting Up Cloud Trace in C#. Check that you have properly configured the Client Library and enabled the access scope. If it still doesn't work, try manually setting the 'GOOGLE_APPLICATION_CREDENTIALS' environment variable as described at the bottom on the setup guide. Commented Jul 17, 2017 at 13:48

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.