DEV Community

Cover image for Clean up resources with kube-janitor
Ashok Nagaraj
Ashok Nagaraj

Posted on • Edited on

Clean up resources with kube-janitor

#kubernetes #cleanup #tooling #tip
About

Clean up (delete) Kubernetes resources after a configured TTL (time to live)

Implementation

The application is deployed as a deployment with escalated privileges. It listens to API requests to the API server, has an internal schedule queue (probably) and then deletes resources upon a rule match.

Reference repo

https://codeberg.org/hjacobs/kube-janitor.git

Installation 
# pick files from https://codeberg.org/hjacobs/kube-janitor/src/branch/main/deploy/ # update rules.yaml as necessary > cat rules.yaml rules: # remove deployments and statefulsets with a "demo" label set after 3 days - id: cleanup-demo-objects resources: - deployments - statefulsets jmespath: "(spec.template.metadata.labels.demo)" ttl: 3d # remove all deployments and jobs named "pr-*" after 6 hours - id: cleanup-pr-deployments resources: - deployments - jobs jmespath: "starts_with(metadata.name, 'pr-')" ttl: 6h # delete all resources within the "temp-*" namespace after 3 days - id: cleanup-temp-namespaces resources: - namespaces jmespath: "starts_with(metadata.name, 'temp-')" ttl: 3d # delete all PVCs which are not mounted and not referenced by StatefulSets after 4 days - id: remove-unused-pvcs resources: - persistentvolumeclaims jmespath: "_context.pvc_is_not_mounted && _context.pvc_is_not_referenced" ttl: 4d > kubectl apply -k .
Enter fullscreen mode Exit fullscreen mode
Configuration

There are 3 ways of using this tool

  1. Annotate the object with a janitor/ttl annotation. Useful for CI/CD scenario
    > kubectl annotate deploy test-app-dep janitor/ttl=24h

  2. Annotate the object with a janitor/expires annotation. Useful for dangling jobs/cronjobs
    > kubectl annotate deploy nginx janitor/expires=2022-03-31

  3. Update the rules file (edit it and (re)deploy or > kubectl edit configmap kube-janitor). This is the server side automation applicable for policy enforcement

  4. refer rules.yaml for example

  5. use jmespath for writing rules (refer this)

Note:

  • namespace level cleanup is not working and there is a PR to add this enhancement
More info

Fascinating to see a python application working almost as a kubernetes CRD without all the complexities.

Top comments (0)