Download to read offline
![• kubectl get persistentvolumeclaims • kubectl describe nodes • Error logs of the database 2018-11-28T00:03:26.322318+06:00 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 4188ms. The settings might not be optimal. (flushed=692 and evicted=0, during the time.) Disk Diagnostics 21 © Copyright 2023 Percona ® LLC. All rights reserved](https://image.slidesharecdn.com/svetasmirnovak8database-230611132611-f8501a0a/75/Database-in-Kubernetes-Diagnostics-and-Monitoring-21-2048.jpg)
![• kubectl get persistentvolumeclaims • kubectl describe nodes • Error logs of the database 2018-11-28T00:03:26.322318+06:00 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 4188ms. The settings might not be optimal. (flushed=692 and evicted=0, during the time.) • Counters in the database Disk Diagnostics 22 © Copyright 2023 Percona ® LLC. All rights reserved](https://image.slidesharecdn.com/svetasmirnovak8database-230611132611-f8501a0a/75/Database-in-Kubernetes-Diagnostics-and-Monitoring-22-2048.jpg)
Uploaded bySveta Smirnova
Database in Kubernetes: Diagnostics and Monitoring
The document discusses diagnostics and monitoring for databases within Kubernetes, focusing on memory, disk, CPU, and network usage. It highlights tools and techniques such as the Percona Monitoring and Management (PMM) client, resource limits, and specialized commands for troubleshooting. Additionally, it emphasizes the importance of log management and automated data collection features provided by the pt-k8s-debug-collector.
Database in Kubernetes: Diagnostics and Monitoring
- 1. Database in Kubernetes: Diagnosticsand Monitoring Sveta Smirnova Principal Support Engineering Coordinator
- 2. • MySQL SupportEngineer • Author MySQL Troubleshooting MySQL Cookbook, 4th Edition • JSON UDF functions • FILTER clause for MySQL • Speaker • Percona Live, OOW, Fosdem, DevConf, HighLoad... Sveta Smirnova
- 3. Database Performance Influencers 3 ©Copyright 2023 Percona ® LLC. All rights reserved
- 4. • Standard databasetools: EXPLAIN • Interactive tools • Log files Query Performance Diagnostics 4 © Copyright 2023 Percona ® LLC. All rights reserved
- 5. • Interactive tools •Same as on bare metal Concurrency Diagnostics 5 © Copyright 2023 Percona ® LLC. All rights reserved
- 6. • Managed byKubernetes • Limits, set by Kubernetes • DBA does not have access to tools, available on bare hardware Hardware 6 © Copyright 2023 Percona ® LLC. All rights reserved
- 7. • Supported PMM 7 © Copyright2023 Percona ® LLC. All rights reserved
- 8. • Supported • AllPercona operators are shipped with PMM client PMM 8 © Copyright 2023 Percona ® LLC. All rights reserved
- 9. • Supported • AllPercona operators are shipped with PMM client • PMM server needs to be installed separately • Install via Helm is available PMM 9 © Copyright 2023 Percona ® LLC. All rights reserved
- 10. • Supported • AllPercona operators are shipped with PMM client • PMM server needs to be installed separately • Install via Helm is available • Add pmm.enabled=true into deploy/cr.yaml, configure PMM server host and API key, then apply PMM 10 © Copyright 2023 Percona ® LLC. All rights reserved
- 11. Database Performance Troubleshooting 11 ©Copyright 2023 Percona ® LLC. All rights reserved
- 12. Database Performance Troubleshooting 12 ©Copyright 2023 Percona ® LLC. All rights reserved
- 13. • Global buffersto cache "hot" data • InnoDB Buffer Pool • WiredTiger Cache • Shared buffers in PostgreSQL • Operation and thread-specific buffers • "Internal" buffers not under user control How Memory Used by Databases? 13 © Copyright 2023 Percona ® LLC. All rights reserved
- 14. • Physically availablefor nodes • Limited by the Operator resources: requests: memory: 32G ... # limits: # memory: 32G Memory Limits 14 © Copyright 2023 Percona ® LLC. All rights reserved
- 15. • kubectl toppods • kubectl top nodes • kubectl describe nodes Memory Diagnostics 15 © Copyright 2023 Percona ® LLC. All rights reserved
- 16. • kubectl toppods • kubectl top nodes • kubectl describe nodes • Database-specific instruments • Performance Schema in MySQL Memory Diagnostics 16 © Copyright 2023 Percona ® LLC. All rights reserved
- 17. • Table data •Indexes • Log files • Temporary data How Disk Used by Databases? 17 © Copyright 2023 Percona ® LLC. All rights reserved
- 18. volumeSpec: persistentVolumeClaim: resources: requests: storage: 2G Disk Limitations 18 ©Copyright 2023 Percona ® LLC. All rights reserved
- 19. • kubectl getpersistentvolumeclaims Disk Diagnostics 19 © Copyright 2023 Percona ® LLC. All rights reserved
- 20. • kubectl getpersistentvolumeclaims • kubectl describe nodes Disk Diagnostics 20 © Copyright 2023 Percona ® LLC. All rights reserved
- 21. • kubectl getpersistentvolumeclaims • kubectl describe nodes • Error logs of the database 2018-11-28T00:03:26.322318+06:00 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 4188ms. The settings might not be optimal. (flushed=692 and evicted=0, during the time.) Disk Diagnostics 21 © Copyright 2023 Percona ® LLC. All rights reserved
- 22. • kubectl getpersistentvolumeclaims • kubectl describe nodes • Error logs of the database 2018-11-28T00:03:26.322318+06:00 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 4188ms. The settings might not be optimal. (flushed=692 and evicted=0, during the time.) • Counters in the database Disk Diagnostics 22 © Copyright 2023 Percona ® LLC. All rights reserved
- 23. • Handle connections •Threads in MySQL, MongoDB • Sub-processes in PostgreSQL • Perform calculations • Perform IO operations How CPU Used by Databases? 23 © Copyright 2023 Percona ® LLC. All rights reserved
- 24. resources: requests: cpu: "16" limits: cpu: "32" CPULimits 24 © Copyright 2023 Percona ® LLC. All rights reserved
- 25. • kubectl toppods • kubectl top nodes • kubectl describe nodes CPU Diagnostics 25 © Copyright 2023 Percona ® LLC. All rights reserved
- 26. • kubectl toppods • kubectl top nodes • kubectl describe nodes • Counters in the database CPU Diagnostics 26 © Copyright 2023 Percona ® LLC. All rights reserved
- 27. • Communication betweenserver and client • Regular client • Application • Replicas • Synchronous nodes How Databases Use Network? 27 © Copyright 2023 Percona ® LLC. All rights reserved
- 28. • kubectl describenodes Network Diagnostics 28 © Copyright 2023 Percona ® LLC. All rights reserved
- 29. • kubectl describenodes • Logs and statuses of the namespace kube-system and your CNI Network Diagnostics 29 © Copyright 2023 Percona ® LLC. All rights reserved
- 30. • kubectl describenodes • Logs and statuses of the namespace kube-system and your CNI • Latency in databases Network Diagnostics 30 © Copyright 2023 Percona ® LLC. All rights reserved
- 31. • By default,log files are gone after pod failure • Critical in such situations as database crash Log Files Rotation 31 © Copyright 2023 Percona ® LLC. All rights reserved
- 32. • By default,log files are gone after pod failure • Critical in such situations as database crash • Solution • PXC Operator: separate pod to store logs • Other operators: K8-recommended solutions such as Grafana Loki Log Files Rotation 32 © Copyright 2023 Percona ® LLC. All rights reserved
- 33. • Created forSupport team to automate K8 troubleshooting collections • Owned by Support team • In active progress • New featurs each version pt-k8s-debug-collector 33 © Copyright 2023 Percona ® LLC. All rights reserved
- 34. pt-stalk • Default forbare metal • Bare hardware • Such tools as • iostat • vmstat • dmesg • top pt-k8s-debug-collector • Default for K8 • kubectl • Such commands as • get pods • get storageclasses • get replicasets • get logs Batch Data Collection 34 © Copyright 2023 Percona ® LLC. All rights reserved
- 35. • Basic batchcollections per databases • MySQL and PXC pt-summary pt-mysql-summary Log files • MongoDB pt-mongodb-summary • PostgreSQL pg_gather Features in pt-k8s-debug-collector 35 © Copyright 2023 Percona ® LLC. All rights reserved
- 36. • Basic batchcollections per databases • MySQL and PXC pt-summary pt-mysql-summary Log files • MongoDB pt-mongodb-summary • PostgreSQL pg_gather • kubectl get Features in pt-k8s-debug-collector 36 © Copyright 2023 Percona ® LLC. All rights reserved
- 37. • Basic batchcollections per databases • MySQL and PXC pt-summary pt-mysql-summary Log files • MongoDB pt-mongodb-summary • PostgreSQL pg_gather • kubectl get • kubectl logs Features in pt-k8s-debug-collector 37 © Copyright 2023 Percona ® LLC. All rights reserved
- 38. • pt-stalk • Morekubectl commands • Extended MongoDB support • Nicer output format for summaries Future of the pt-k8s-debug-collector 38 © Copyright 2023 Percona ® LLC. All rights reserved
- 39. • Mind limitsin cr.yaml • Work with K8 administrators for setting up policies, allowing you to view hardware usage • Setup Loki or similar solution to save the log files after restart • Enable online database instrumentation • Use PMM or similar monitoring solution • Plan in advance Conclusion 39 © Copyright 2023 Percona ® LLC. All rights reserved
- 40. Percona K8 Operators PMMDemo pt-k8s-debug-collector pt-stalk pg_gather Grafana Loki Tools Mentioned 40 © Copyright 2023 Percona ® LLC. All rights reserved
- 41. Thank you! /svetsmirnova /svetasmirnova /svetsmirnova/SvetaSmirnova