Skip to content

4.5.0 BackrestBackups loop until partitions full #2106

@v3nturetheworld

Description

@v3nturetheworld

In Crunchy 4.4.0 and below I noticed the default behavior in regards to backrest backups being made was one backup was created on cluster creation prior to replica, then any following backup would be made from a failover or some pgo command. In Crunchy 4.5.0 I see backrestBackups being created every 10-15 minutes, and without settings limits in a custom pgBackRest Custom Configuration, it will repeatedly make backups until the local partitions are full and postgres will enter a non function state until all of the wal files are deleted in the local storage folder on each node. I'm not sure if there's some operator installation config that I missed when upgrading from 4.4 or if the automatic backups are intended behavior but I was wondering if there was a way to ensure backups only occur like in 4.4 which is on cluster creation, failover and from pgo commands rather then periodic.

I confirmed that this issue occurs when no custom configs used and and no data was ever added to the database by just running:

pgo create cluster test-cluster -n crunchy-test --cpu 15m --memory 690Mi --memory-limit 2Gi --storage-config testlocalstorage --replica-storage-config testlocalstorage --pgbackrest-cpu 30m --pgbackrest-memory 20Mi --pgbackrest-memory-limit 60Mi 

I'm guessing this shouldn't be happening, as it doesnt happen with the same creation method on 4.4, and I haven't seen any issues similar to this.

Please tell us about your environment:

  • Operating System:
  • Where is this running ( Local , Cloud Provider): 3 node local cluster
  • Storage being used (NFS, Hostpath, Gluster, etc): primary and replica use localstorage, backrest-shared-repo uses openebs-sc
  • Container Image Tag: centos7-13.0-4.5.0
  • PostgreSQL Version: 13.0
  • Platform (Docker, Kubernetes, OpenShift): OpenShift
  • Platform Version: 3.11

If possible please run the following on the kubernetes or OpenShift (oc) commands and provide the result:

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions