4

I need the jsonPayload in the logs of a Google Cloud Function instead of the textPayload. My aim is to use the keys of the dictionary as labels (see Log-based metrics labels) for Log-Based Metrics so that these can be addressed in Grafana.

I am using Python's logging module, but I can also switch to something else if needed.

I need as output in the logs:

jsonPayload: `{'key1':value1, 'key2':value2}`

But I get a textPayload output instead, the whole next line is a string:

"2022-02-08 15:43:32,460 [INFO]: {"key1": value1, "key2": value2}"

Real example from the logs, in the middle, you see the textPayload:

enter image description here

The picture as text:

{ insertId: "000000-1b431ffd-e42d-4f83-xyz" labels: {1} logName: "projects/MY_PROJECT/logs/cloudfunctions.googleapis.com%2Fcloud-functions" receiveTimestamp: "2022-02-08T15:43:41.808217166Z" resource: {2} textPayload: "2022-02-08 15:43:32,460 [INFO]: {"json_metadata": {"countrows": 736203, "countcolumns": 6, "size": 48261360, "gcs_stamp": "2022-02-08 15:43:32.451000+00:00", "python_stamp": "2022-02-08 15:43:31.055538"}}" timestamp: "2022-02-08T15:43:32.460Z" trace: "projects/MY_PROJECT/traces/dd97759176248586a3d3xyz" }

First tries

Reading from https://cloud.google.com/logging/docs/structured-logging:

In Cloud Logging, structured logs refer to log entries that use the jsonPayload field to add structure to their payloads. Structured logging applies to user-written logs.

I tried to get this "structured logging" following Writing structured logs by

logging.info(json.dumps(json_for_gcp_lbm))

but to no avail.

Further in the links: there is a built-in Logging agent from GCP that uses fluentd as to About the Logging agent seems to be available only for Google Kubernetes Engine or the App Engine, not in a Google Cloud Function:

If you're using Google Kubernetes Engine or the App Engine flexible environment, you can write structured logs as JSON objects serialized on a single line to stdout or stderr. The Logging agent then sends the structured logs to Cloud Logging as the jsonPayload of the LogEntry structure.

How can I get the jsonPayload in this output?

Improve this question
6
  • Do I understand correctly - the output you provided is from using the example? What python version are you using? Commented Feb 9, 2022 at 12:50
  • @mdobrucki Yes, I have used logging.info(json.dumps(json_for_gcp_lbm)) which is taken from the example of that link, but I only used the very last line of that example, hoping that this would already make it. I use a Python 3.9 cloud function. Commented Feb 9, 2022 at 12:56
  • If you are using python 3.9 the example provided in docs should do the trick, make sure to follow it correctly and structure your data accordingly. Have you considered using Logging client libraries with this example? Commented Feb 9, 2022 at 13:22
  • @mdobrucki No, I must have overseen that since I have searched directly for "structured json" instead. I will check it with the google library and use the example's logger. Commented Feb 9, 2022 at 13:24
  • Let me know what you've established. Commented Feb 9, 2022 at 13:26

1 Answer 1

Reset to default
2

You can set up structured logging following the example in the docs. Make sure your python version is 3.8 or above. Another way to achieve that is to use Logging client libraries.

Example from the OP for the second method:

Put into your requirements.txt:

google-cloud-logging==3.0.0

as the most recent version at the time of writing.

from google.cloud import logging as gclogger

(or name it as you like)

and call the json logging with:

json_for_gcp_lbm = {MY_JSON_HERE}

 from google.cloud import logging as gclogger ... logging_client = gclogger.Client() logger_name = "MY_LOG_NAME_OF_FREE_CHOICE" # Saved in the google cloud logs logger = logging_client.logger(logger_name) # I do not know how it would work with `import logging` as the # built-in Python module, but I got it to work with adding # Following line does not work # logging.info(json.dumps(json_for_gcp_lbm, default=str)) # Instead, we need # https://cloud.google.com/logging/docs/samples/logging-write-log-entry#code-sample logger.log_struct(json_for_gcp_lbm)

Which leads to a jsonPayload output:

enter image description here

And you can then choose any label from the json in the "Create logs metric" (Log-Based Metrics / LBM) menu:

enter image description here

Improve this answer
2
  • I do not know how it would get the jsonPayload using import logging as the built-in Python module in the first link, though :). When using only the last line of the example, it gives only textPayload instead. Likely, I need the request part with that global_log_fields = {} to make it work. I think that import google.cloud.logging is more "pythonic" then, even if I have to install and import the google cloud logging for this. Commented Feb 9, 2022 at 14:33
  • Small bonus, I had not asked this, saw this by chance: there is a passage further down on the same page of your first link about how I could also access the new google.cloud.logging made logs with its jsonPayload entries from inside the CF, see Using the Logging API. Commented Feb 9, 2022 at 17:25

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.