Uploaded byMartin Etmajer
Monitoring, Logging and Tracing on Kubernetes
The document discusses monitoring, logging and tracing tools for Kubernetes including Heapster, Grafana, Fluentd, Elastic Stack, Jolokia and OpenTracing. It provides examples of deploying Heapster with InfluxDB and Grafana for metrics collection, Fluentd to ingest container logs into Elasticsearch, and using Jolokia and OpenTracing for remote access to JMX metrics and distributed tracing functionality.
Monitoring, Logging and Tracing on Kubernetes
- 1. #Perform2017 with Heapster, Grafana,Fluentd, Elastic Stack, Jolokia and OpenTracing Martin Etmajer @metmajer Dynatrace, Technology Lead Monitoring, Logging and Tracing on Kubernetes
- 3.
- 4. The „Jay Z“of classical music
- 5.
- 6.
- 7.
- 8. #Perform2017 Monitoring with Kubernetes Dashboard,Heapster, influxdb & Grafana
- 10. $ git clonehttps://github.com/kubernetes/dashboard.git $ kubectl create –f src/deploy/kubernetes-dashboard.yaml Example: Deploy Kubernetes Dashboard likely to be present in your kube-system already
- 14.
- 16.
- 17. collects resources usage,etc. exposed via REST
- 18. Heapster runs ina Pod
- 19.
- 20. Example: Heapster (standalone) source Kubernetes RESTclient Kubernetes Dashboard via built-in client
- 22. Example: Deploy Heapster(standalone) $ git clone https://github.com/kubernetes/heapster.git $ kubectl create –f heapster/deploy/kube-config/standalone/
- 25. Example: Heapster (withinfluxdb and Grafana) source Kubernetes sinks influxdb grafana query
- 27. Example: Deploy Heapster(with influxdb and Grafana) $ git clone https://github.com/kubernetes/heapster.git $ kubectl create –f heapster/deploy/kube-config/influxdb/
- 29. #Perform2017 Logging with Fluentd andElastic Stack
- 30. continuous stream oflog data for each container
- 31.
- 33.
- 34. Example: Fluentd (forcontainer logs to elasticsearch) watch container logs elasticsearch kibana queryingest via configuration
- 36. Example: Deploy Fluentd(for container logs to elasticsearch) $ git clone https://github.com/kubernetes/kubernetes.git $ kubectl create –f kubernetes/cluster/addons/fluentd-elasticsearch/
- 40.
- 41. remote access toJMX MBeans
- 42. remote access toJMX MBeans exposed via REST
- 43. Java Management Extensions(JMX) to manage and monitor applications Source: https://en.wikipedia.org/wiki/Java_Management_Extensions Dynamic MBean MXBean Standard MBean Instrumentation Level instrument resources MBeanServerAgent Level agent for registered MBeans Connectors Adaptors Remote Management Level provides access to MBeanServerRMI, HTTP/SOAP HTML, SNMP
- 44. Example: Jolokia (getlist of accessible MBeans) $ curl http://localhost:8080/jolokia/list includes memory, threading, gc, etc.
- 46. Example: Jolokia (readmemory usage data) $ curl http://localhost:8080/jolokia/read/java.lang:type=Memory name of MBean
- 48. Example: Jolokia (readthreading data) $ curl http://localhost:8080/jolokia/read/java.lang:type=Threading name of MBean
- 50.
- 51. a formally specifiedstandard API for reporting with various API implementations
- 53.
- 54. Why distributed (end-to-end)tracing? Source: https://research.google.com/pubs/pub36356.html initiator services remote calls
- 55. Why distributed (end-to-end)tracing? Source: https://research.google.com/pubs/pub36356.html initiator services remote calls What does the architecture look like?What does the architecture look like?
- 56. Why distributed (end-to-end)tracing? Source: https://research.google.com/pubs/pub36356.html initiator services remote calls Where is most of the time spent?
- 57. Why distributed (end-to-end)tracing? Source: https://research.google.com/pubs/pub36356.html initiator services remote calls How are services invoked?
- 58. Why distributed (end-to-end)tracing? Source: https://research.google.com/pubs/pub36356.html initiator services remote calls Which services aren’t invoked?
- 60.
- 64.
Editor's Notes
- #10 get an overview of the applications running on the Kubernetes cluster create, modify and delete Kubernetes resources troubleshooting (logs) https://kubernetes.io/docs/user-guide/ui/
- #20 https://github.com/kubernetes/heapster/blob/master/docs/source-configuration.md https://github.com/kubernetes/heapster/blob/master/docs/sink-configuration.md https://github.com/kubernetes/heapster/blob/master/docs/model.md