Configure logging to Cloud Logging

This document describes how to customize logging by using a ConfigMap on your target workload cluster.

Available logs

By default, Migrate to Containers writes entries from the following sources:

  • Entries written to stdout of init, the parent of all Linux processes.
  • Contents from /var/log/syslog.

In Cloud Logging, entries from the supported log files will each appear on a line in the log in the following form. Labels are defined in a ConfigMap that lists log files.

 date time label log-output

The following example illustrates the form with an entry from MySQL:

2019-09-22 12:43:08.681193976 +0000 UTC mysql log-output

You can opt to extend the log forwarding feature to propagate logs for other processes and applications running in the migrated container.

Forward application logs to Cloud Logging

In order to log to Cloud Logging from applications on VMs migrated with Migrate to Containers, you must have the following in place:

  • Create a ConfigMap on your target workload cluster that specifies a list of the file system log files whose entries should be written to Cloud Logging.

    You can use the sample ConfigMap included in the following section as a reference covering many common log files.

  • The ConfigMap must be in the same namespace as your container's Deployment or StatefulSet.

  • The Deployment or StatefulSet must reference ConfigMap.

  • If you're migrating to Google Distributed Cloud, install Connect on your Google Distributed Cloud cluster. Connect is required to pass data back to Cloud Logging.

 apiVersion: apps/v1 kind: Deployment metadata: creationTimestamp: null labels: app: my-app migrate-for-anthos-type: workload name: test-ibsc-appenhmutimiy spec: replicas: 1 selector: matchLabels: app: test-ibsc-appenhmutimiy migrate-for-anthos-type: workload strategy: {} template: metadata: creationTimestamp: null labels: app: test-ibsc-appenhmutimiy migrate-for-anthos-type: workload spec: containers: - image: gcr.io/myproject/my-container-image:v1.0.0 name: test-ibsc-appenhmutimiy readinessProbe: exec: command: - /code/ready.sh resources: {} securityContext: privileged: true volumeMounts: - mountPath: /sys/fs/cgroup name: cgroups  - mountPath: /code/config/logs/ name: logs-config volumes: - hostPath: path: /sys/fs/cgroup type: Directory name: cgroups  - configMap: name: <ConfigMap name> name: logs-config

Use ConfigMap for common application logs

The reference ConfigMap YAML in the following section covers a list of common application log files and their application labels. You can copy the reference ConfigMap YAML as is, or modify it to remove entries or add your custom application logs.

To edit the list of log files, edit the elements following the logs: line. A log sub-element consists of the following structure:

label:  - path-with-simple-wildcards

For example:

tomcat:  - /var/log/tomcat*/catalina.out  - /var/log/tomcat*/localhost.*.log

Reference ConfigMap for common apps

apiVersion: v1 kind: ConfigMap metadata:  creationTimestamp: null  name: <ConfigMap name> data:  logs.yaml: |  logs:  mysql:  - /var/log/mysql.log  - /var/log/mysql/mysql.log  mongodb:  - /var/log/mongodb/*.log  memcached:  - /var/log/memcached.log  redis:  - /var/log/redis*.log  - /var/log/redis/*.log  rabbitmq-startup:  - /var/log/rabbitmq/startup_log  rabbitmq-startup_err:  - /var/log/rabbitmq/startup_err  rabbitmq-sasl:  - /var/log/rabbitmq/*-sasl.log  sugarcrm:  - /var/www/*/sugarcrm.log  tomcat-localhost_access_log:  - /var/log/tomcat*/localhost_access_log*.txt  tomcat:  - /var/log/tomcat*/catalina.out  - /var/log/tomcat*/localhost.*.log  apache-access:  - /var/log/apache*/access.log  - /var/log/apache*/access_log  - /var/log/httpd/access.log  - /var/log/httpd/access_log  apache-error:  - /var/log/apache*/error.log  - /var/log/apache*/error_log  - /var/log/httpd/error.log  - /var/log/httpd/error_log  cassandra:  - /var/log/cassandra/system.log  - /var/log/cassandra/cassandra.log  - /var/log/cassandra/output.log  chef-bookshelf:  - /var/log/chef-server/bookshelf/current  chef-expander:  - /var/log/chef-server/chef-expander/current  chef-pedant-http-traffic:  - /var/log/chef-server/chef-pedant/http-traffic.log  chef-server-webui:  - /var/log/chef-server/chef-server-webui/current  chef-solr:  - /var/log/chef-server/chef-solr/current  chef-erchef-current:  - /var/log/chef-server/erchef/current  chef-erchef:  - /var/log/chef-server/erchef/erchef.log.1  chef-nginx-access:  - /var/log/chef-server/nginx/access.log  chef-nginx-error:  - /var/log/chef-server/nginx/error.log  chef-nginx-rewrite-port-80:  - /var/log/chef-server/nginx/rewrite-port-80.log  chef-postgresql:  - /var/log/chef-server/postgresql/current  chef-rabbitmq:  - /var/log/chef-server/rabbitmq/current  postgresql:  - /var/log/postgres*/*.log  - /var/log/pgsql/*.log  puppet-access:  - /var/log/pe-httpd/access.log  puppet-puppetmasteraccess:  - /var/log/pe-httpd/puppetmasteraccess.log  puppet-activemq:  - /var/log/pe-activemq/activemq.log  puppet-activemq-wrapper:  - /var/log/pe-activemq/wrapper.log  puppet-mcollective:  - /var/log/pe-mcollective/mcollective.log  puppet-mcollective-audit:  - /var/log/pe-mcollective/mcollective_audit.log  puppet-puppetdb:  - /var/log/pe-puppetdb/pe-puppetdb.log  puppet-dashboard-error:  - /var/log/pe-httpd/puppetdashboard.error.log  puppet-dashboard-mcollective-client:  - /var/log/pe-puppet-dashboard/mcollective_client.log  puppet-dashboard-production:  - /var/log/pe-puppet-dashboard/production.log  puppet-dashboard-event-inspector:  - /var/log/pe-puppet-dashboard/event-inspector.log  puppet-dashboard-certificate-manager:  - /var/log/pe-puppet-dashboard/certificate_manager.log  puppet-dashboard-live-management:  - /var/log/pe-puppet-dashboard/live-management.log  puppet-console-cas-client:  - /var/log/pe-console-auth/cas_client.log  puppet-console-auth-cas:  - /var/log/pe-console-auth/cas.log  puppet-console-auth:  - /var/log/pe-console-auth/auth.log  puppet-dashboard-access:  - /var/log/pe-httpd/puppetdashboard.access.log  puppet-dashboard-failed-reports:  - /var/log/pe-puppet-dashboard/failed_reports.log  puppet-error:  - /var/log/pe-httpd/error.log  puppet-other-vhosts-access:  - /var/log/pe-httpd/other_vhosts_access.log  puppet-masterhttp:  - /var/log/pe-puppet/masterhttp.log  - /var/log/puppet/masterhttp.log  puppet-rails:  - /var/log/pe-puppet/rails.log  puppet-http:  - /var/log/puppet/http.log  gitlab-application:  - /home/git/gitlab/log/application.log  gitlab-production:  - /home/git/gitlab/log/production.log  gitlab-sidekiq:  - /home/git/gitlab/log/sidekiq.log  gitlab-unicorn-stdout:  - /home/git/gitlab/log/unicorn.stdout.log  gitlab-unicorn-stderr:  - /home/git/gitlab/log/unicorn.stderr.log  gitlab-githost:  - /home/git/gitlab/log/githost.log  gitlab-satellites:  - /home/git/gitlab/log/satellites.log  gitlab-shell:  - /home/git/gitlab-shell/gitlab-shell.log  jenkins:  - /var/log/jenkins/jenkins.log  jetty-request:  - /var/log/jetty/*.request.log  jetty-stderrout:  - /var/log/jetty/*.stderrout.log  jetty-out:  - /var/log/jetty/out.log  joomla:  - /var/www/joomla/logs/*.log  magento-system:  - /var/www/magento/var/log/system.log  magento-exception:  - /var/www/magento/var/log/exception.log  magento-report:  - /var/www/magento/var/report/*  mediawiki:  - /var/log/mediawiki/*.log  nginx-access:  - /var/log/nginx/access.log  nginx-error:  - /var/log/nginx/error.log  redmine:  - /var/log/redmine/*.log  salt-master:  - /var/log/salt/master  salt-minion:  - /var/log/salt/minion  salt-key:  - /var/log/salt/key  salt-syndic:  - /var/log/salt/syndic.loc  solr:  - /var/log/solr/*.log  sugarcrm:  - /var/www/*/sugarcrm.log  tomcat-localhost_access_log:  - /var/log/tomcat*/localhost_access_log.*.txt  tomcat:  - /var/log/tomcat*/catalina.out,/var/log/tomcat*/localhost.*.log

What's next