DEV Community

Cover image for Backing up Prometheus using TSDB Snapshots : Kubernetes/OpenShift
karan singh
karan singh

Posted on

Backing up Prometheus using TSDB Snapshots : Kubernetes/OpenShift

#prometheus #backup #kubernetes

These are my quick-and-dirty brain-dump notes to myself on how to backup prometheus database running on k8s or OpenShift

  • Get Token for API Authentication and Prometheus API Route URL 
 oc whoami -t oc get route -n openshift-monitoring | grep -i prometheus
Enter fullscreen mode Exit fullscreen mode
  • Run sample curl request 
 curl -ks -H ‘Authorization: Bearer 0za4LjX9xPcqDjhWaufkgcQGo4grqA7ws4zvHrqgfY4’ ‘[https://prometheus-k8s-openshift-monitoring.apps.ocp4.cp4d.com/api/v1/query?query=ALERTS'](https://prometheus-k8s-openshift-monitoring.apps.ocp4.cp4d.com/api/v1/query?query=ALERTS') | python -m json.tool
Enter fullscreen mode Exit fullscreen mode
  • Create TSDB Snapshot 
 curl -X ‘POST’ -ks -H ‘Authorization: Bearer 0za4LjX9xPcqDjhWaufkgcQGo4grqA7ws4zvHrqgfY4’ ‘[https://prometheus-k8s-openshift-monitoring.apps.ocp4.cp4d.com/api/v2/admin/tsdb/snapshot'](https://prometheus-k8s-openshift-monitoring.apps.ocp4.cp4d.com/api/v2/admin/tsdb/snapshot') | python -m json.tool
Enter fullscreen mode Exit fullscreen mode
  • You might get an error, so you first need to enable Admin APIs 
 { “error”: “Admin APIs are disabled”, “message”: “Admin APIs are disabled”, “code”: 14 }
Enter fullscreen mode Exit fullscreen mode
  • Enable AdminAPI 
 oc -n openshift-monitoring patch prometheus k8s \ — type merge — patch ‘{“spec”:{“enableAdminAPI”:true}}’
Enter fullscreen mode Exit fullscreen mode
  • Verify Admin API is enabled 
 oc describe po prometheus-k8s-1 | grep -i admin — web.enable-admin-api
Enter fullscreen mode Exit fullscreen mode
  • Hit TSDB snapshot API to take snapshot 
 curl -X ‘POST’ -ks -H ‘Authorization: Bearer 0za4LjX9xPcqDjhWaufkgcQGo4grqA7ws4zvHrqgfY4’ ‘[https://prometheus-k8s-openshift-monitoring.apps.ocp4.cp4d.com/api/v2/admin/tsdb/snapshot'](https://prometheus-k8s-openshift-monitoring.apps.ocp4.cp4d.com/api/v2/admin/tsdb/snapshot') | python -m json.tool
Enter fullscreen mode Exit fullscreen mode 
 { “name”: “20210512T162601Z-33415dbd315ae6af” }
Enter fullscreen mode Exit fullscreen mode

  • Find the snapshot and copy it locally

  • The default folder is /prometheus/snapshots/ but you can find the data folder by finding the --storage.tsdb.path config in your deployment.

 curl -X ‘POST’ -ks -H ‘Authorization: Bearer 0za4LjX9xPcqDjhWaufkgcQGo4grqA7ws4zvHrqgfY4’ ‘[https://prometheus-k8s-openshift-monitoring.apps.ocp4.cp4d.com/api/v1/admin/tsdb/snapshot'](https://prometheus-k8s-openshift-monitoring.apps.ocp4.cp4d.com/api/v1/admin/tsdb/snapshot') | python -m json.tool
Enter fullscreen mode Exit fullscreen mode
  • List the the snapshot directory 
 oc -n openshift-monitoring exec -it prometheus-k8s-0 -c prometheus — /bin/sh -c “ls /prometheus/snapshots/20210512T162601Z-33415dbd315ae6af”
Enter fullscreen mode Exit fullscreen mode
  • Copy the Snapshot from prometheus container to local machine 
 oc project openshift-monitoring oc rsync prometheus-k8s-0:/prometheus/snapshots/ /home/prometheus
Enter fullscreen mode Exit fullscreen mode

And this is how to back Prometheus Snapshot to a local machine

Top comments (0)