This runbook will show you how to perform a volume downsize. The usual operation is to extend a volume, but in some cases you might have over-dimensioned your volumes and might need to downsize your volumes, in order to reduce costs.
Assume you have a StackGres cluster with:
3ongres-dbongres-db20Gi$ kubectl exec -it -n ongres-db ongres-db-2 -c patroni -- patronictl list + Cluster: ongres-db (6918002883456245883) -------+----+-----------+ | Member | Host | Role | State | TL | Lag in MB | +-------------+----------------+--------+---------+----+-----------+ | ongres-db-0 | 10.0.7.11:7433 | Leader | running | 3 | | | ongres-db-1 | 10.0.0.10:7433 | | running | 3 | 0 | | ongres-db-2 | 10.0.6.9:7433 | | running | 3 | 0 | +-------------+----------------+--------+---------+----+-----------+ Verify the PVC’s:
$ kubectl get pvc -n ongres-db NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE distributedlogs-data-distributedlogs-0 Bound pvc-9bab7a68-a209-4d9a-93f7-871a217a28b1 50Gi RWO standard 162m ongres-db-data-ongres-db-0 Bound pvc-a2aa5198-c553-4e0d-a1e1-914669abb69f 20Gi RWO gp2-data 11m ongres-db-data-ongres-db-1 Bound pvc-c724b2bf-cf17-4f57-a882-3a5da6947f44 20Gi RWO gp2-data 10m ongres-db-data-ongres-db-2 Bound pvc-5124b9d2-ec35-46d7-9eda-7543d9ed7148 20Gi RWO gp2-data 4m47s Assuming the disk size is over-dimensioned, and you need to perform a downsize to 15Gi.
Perform a switchover to the pod with the higher index number (ongres-db-2).
Execute:
kubectl exec -it -n ongres-db ongres-db-0 -c patroni -- patronictl switchover
Master [ongres-db-0]: Candidate ['ongres-db-1', 'ongres-db-2'] []: ongres-db-2 When should the switchover take place (e.g. 2021-01-15T16:40 ) [now]: Current cluster topology + Cluster: ongres-db (6918002883456245883) -------+----+-----------+ | Member | Host | Role | State | TL | Lag in MB | +-------------+----------------+--------+---------+----+-----------+ | ongres-db-0 | 10.0.7.11:7433 | Leader | running | 3 | | | ongres-db-1 | 10.0.0.10:7433 | | running | 3 | 0 | | ongres-db-2 | 10.0.6.9:7433 | | running | 3 | 0 | +-------------+----------------+--------+---------+----+-----------+ Are you sure you want to switchover cluster ongres-db, demoting current master ongres-db-0? [y/N]:y 2021-01-15 15:41:11.93457 Successfully switched over to "ongres-db-2" + Cluster: ongres-db (6918002883456245883) -------+----+-----------+ | Member | Host | Role | State | TL | Lag in MB | +-------------+----------------+--------+---------+----+-----------+ | ongres-db-0 | 10.0.7.11:7433 | | stopped | | unknown | | ongres-db-1 | 10.0.0.10:7433 | | running | 3 | 0 | | ongres-db-2 | 10.0.6.9:7433 | Leader | running | 3 | | +-------------+----------------+--------+---------+----+-----------+ Now, check the cluster state:
$ kubectl exec -it -n ongres-db ongres-db-2 -c patroni -- patronictl list + Cluster: ongres-db (6918002883456245883) -------+----+-----------+ | Member | Host | Role | State | TL | Lag in MB | +-------------+----------------+--------+---------+----+-----------+ | ongres-db-0 | 10.0.7.11:7433 | | running | 4 | 0 | | ongres-db-1 | 10.0.0.10:7433 | | running | 4 | 0 | | ongres-db-2 | 10.0.6.9:7433 | Leader | running | 4 | | +-------------+----------------+--------+---------+----+-----------+ As the downsize is not a common situation, it is necessary to temporary remove the StackGres operator validating-webhook, so first, create a backup of the yaml manifest:
Execute:
kubectl get validatingwebhookconfigurations.admissionregistration.k8s.io stackgres-operator -o yaml > validating-webhook-stackgres-operator.yaml Now delete the StackGres operator validating-webhook exucting:
kubectl delete validatingwebhookconfigurations.admissionregistration.k8s.io stackgres-operator WARNING: Note that removing the validating webhook might potentially lead to some error in the resources that you may have to solve manually later if after restarting the operator Pod any error arise during the update of the existing resources that the operator executes on bootstrap. Usually manual intervention is not needed, but you should be aware of this.
Now, edit the StackGres cluster volume definition to the new size:
kubectl patch sgclusters -n ongres-db ongres-db --type='json' -p '[{ "op": "replace", "path": "/spec/pods/persistentVolume/size", "value": "10Gi" }]' You’ll get the following message:
sgcluster.stackgres.io/ongres-db patched Now, if you check the events you will see an error like:
kubectl get events -n ongres-db .... Failure executing: PATCH at: https://10.96.0.1/apis/apps/v1/namespaces/ongres-db/statefulsets/ongres-db. Message: StatefulSet.apps "ongres-db" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=spec, message=Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden, reason=FieldValueForbidden, additionalProperties={})], group=apps, kind=StatefulSet, name=ongres-db, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=StatefulSet.apps "ongres-db" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}). .... This is expected because is forbidden to change the spec of a stateful set.
Delete the stateful set and let the StackGres operator recreate it:
$ kubectl delete sts -n ongres-db ongres-db --cascade=orphan Important Note: Do not forget the parameter
--cascade=orphanbecause this will keep the existing pods.
Verify that the stateful set now has the new volume size:
$ kubectl describe sts -n ongres-db ongres-db | grep -i capacity Capacity: 15Gi At this moment it is recommended to resotre the StackGres operator validating-webhook:
kubectl create -f validating-webhook-stackgres-operator.yaml Edit the replica size to 1:
$ kubectl patch sgclusters -n ongres-db ongres-db --type='json' -p '[{ "op": "replace", "path": "/spec/instances", "value": 1 }]' Once you decrease the replicas, you’ll see something like:
$ kubectl get pods -n ongres-db NAME READY STATUS RESTARTS AGE distributedlogs-0 2/2 Running 0 3h4m ongres-db-2 6/6 Running 0 27m Proceed to delete the unused PVCs ongres-db-data-ongres-db-0 and ongres-db-data-ongres-db-1:
$ kubectl delete pvc -n ongres-db ongres-db-data-ongres-db-0 persistentvolumeclaim "ongres-db-data-ongres-db-0" deleted $ kubectl delete pvc -n ongres-db ongres-db-data-ongres-db-1 persistentvolumeclaim "ongres-db-data-ongres-db-1" deleted This will release the persistent volumes and then you can proceed to delete them:
$ kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-5124b9d2-ec35-46d7-9eda-7543d9ed7148 20Gi RWO Retain Bound ongres-db/ongres-db-data-ongres-db-2 gp2-data 32m pvc-9bab7a68-a209-4d9a-93f7-871a217a28b1 50Gi RWO Delete Bound ongres-db/distributedlogs-data-distributedlogs-0 standard 3h10m pvc-a2aa5198-c553-4e0d-a1e1-914669abb69f 20Gi RWO Retain Released ongres-db/ongres-db-data-ongres-db-0 gp2-data 39m pvc-c724b2bf-cf17-4f57-a882-3a5da6947f44 20Gi RWO Retain Released ongres-db/ongres-db-data-ongres-db-1 gp2-data 38m Delete the disks with Released status:
$ kubectl delete pv pvc-a2aa5198-c553-4e0d-a1e1-914669abb69f persistentvolume "pvc-a2aa5198-c553-4e0d-a1e1-914669abb69f" deleted $ kubectl delete pv pvc-c724b2bf-cf17-4f57-a882-3a5da6947f44 persistentvolume "pvc-c724b2bf-cf17-4f57-a882-3a5da6947f44" deleted Increase the replica size to 2:
$ kubectl patch sgclusters -n ongres-db ongres-db --type='json' -p '[{ "op": "replace", "path": "/spec/instances", "value": 2 }]' Now, your cluster will have 2 pods:
$ kubectl get pods -n ongres-db NAME READY STATUS RESTARTS AGE distributedlogs-0 2/2 Running 0 3h15m ongres-db-0 6/6 Running 0 49s ongres-db-2 6/6 Running 0 37m Check again the cluster state:
$ kubectl exec -it -n ongres-db ongres-db-2 -c patroni -- patronictl list + Cluster: ongres-db (6918002883456245883) -------+----+-----------+ | Member | Host | Role | State | TL | Lag in MB | +-------------+----------------+--------+---------+----+-----------+ | ongres-db-0 | 10.0.7.12:7433 | | running | 4 | 0 | | ongres-db-2 | 10.0.6.9:7433 | Leader | running | 4 | | +-------------+----------------+--------+---------+----+-----------+ And the new pod will have the new disk size:
$ kubectl get pvc -n ongres-db NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE distributedlogs-data-distributedlogs-0 Bound pvc-9bab7a68-a209-4d9a-93f7-871a217a28b1 50Gi RWO standard 3h17m ongres-db-data-ongres-db-0 Bound pvc-37d96872-b132-4a89-a579-d87f8cf1fa92 15Gi RWO gp2-data 2m47s ongres-db-data-ongres-db-2 Bound pvc-5124b9d2-ec35-46d7-9eda-7543d9ed7148 20Gi RWO gp2-data 39m Perform another switchover, this time to node ongres-db-0:
$ kubectl exec -it -n ongres-db ongres-db-2 -c patroni -- patronictl switchover Master [ongres-db-2]: Candidate ['ongres-db-0'] []: ongres-db-0 When should the switchover take place (e.g. 2021-01-15T17:12 ) [now]: Current cluster topology + Cluster: ongres-db (6918002883456245883) -------+----+-----------+ | Member | Host | Role | State | TL | Lag in MB | +-------------+----------------+--------+---------+----+-----------+ | ongres-db-0 | 10.0.7.12:7433 | | running | 4 | 0 | | ongres-db-2 | 10.0.6.9:7433 | Leader | running | 4 | | +-------------+----------------+--------+---------+----+-----------+ Are you sure you want to switchover cluster ongres-db, demoting current master ongres-db-2? [y/N]: y 2021-01-15 16:12:57.14561 Successfully switched over to "ongres-db-0" + Cluster: ongres-db (6918002883456245883) -------+----+-----------+ | Member | Host | Role | State | TL | Lag in MB | +-------------+----------------+--------+---------+----+-----------+ | ongres-db-0 | 10.0.7.12:7433 | Leader | running | 4 | | | ongres-db-2 | 10.0.6.9:7433 | | stopped | | unknown | +-------------+----------------+--------+---------+----+-----------+ This will delete the pod ongres-db-2 and create the pod ongres-db-1
NAME READY STATUS RESTARTS AGE distributedlogs-0 2/2 Running 0 3h19m ongres-db-0 6/6 Running 0 4m51s ongres-db-1 6/6 Running 0 41s You can proceed to delete the PVC and PV of ongres-db-2
$ kubectl delete pvc -n ongres-db ongres-db-data-ongres-db-2 persistentvolumeclaim "ongres-db-data-ongres-db-2" deleted $ kubectl delete pv pvc-5124b9d2-ec35-46d7-9eda-7543d9ed7148 persistentvolume "pvc-5124b9d2-ec35-46d7-9eda-7543d9ed7148" deleted Now, your cluster will have the new, reduced disk size:
$ kubectl get pvc -n ongres-db NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE distributedlogs-data-distributedlogs-0 Bound pvc-9bab7a68-a209-4d9a-93f7-871a217a28b1 50Gi RWO standard 3h24m ongres-db-data-ongres-db-0 Bound pvc-37d96872-b132-4a89-a579-d87f8cf1fa92 15Gi RWO gp2-data 9m21s ongres-db-data-ongres-db-1 Bound pvc-46c1433b-26e8-422c-aecf-145b1bb5aac1 15Gi RWO gp2-data 5m11s As you temporary removed the validating-webhook it is necessary to restart the StackGres Operator pod.
Execute:
kubectl delete pod -n stackgres -l app=stackgres-operator Check the pod started successfully:
Execute:
kubectl get pod -n stackgres -l app=stackgres-operator The output should be like:
NAME READY STATUS RESTARTS AGE stackgres-operator-85df9c556c-c242s 1/1 Running 0 79s