Sheduled kubernetes pg-backup objects are not automatically cleaned up

[konoox]

Report

Sheduled kubernetes pg-backup objects are not automatically cleaned up

More about the problem

Since the 2.3.0 k8s Percona Postgres operator upgrade, pg-backup resources are created for each scheduled backup (K8SPG-410)
The pgbackrest backups and related items saved in the storage are still clean-up up properly following the retention rules defined in the spec.backups.pgbackrest.global.-retention-full* attributes of the pg cluster definition.
BUT the pg-backup related k8s resources are well created but never cleaned up even with the retention attributes properly defined.
This problem lead to a quick accumulation of pg-backups and jobs k8s resources in the namespaces using a pg cluster with scheduled backups.

Steps to reproduce

1. Create a pg cluster with a backup storage and schedule section defined and a retention period
2. On the scheduled time, pg-backup resource is created with the related job, pod, pgbackrest backup and items in the storage
3. At the end of the retention period, pgbackrest backup and items are removed from the storage but none of the k8s resources (pg-backup and job) are not deleted

Versions

1. Kubernetes - v1.27.6
2. Operator - Percona for PostgreSQL 2.3.1
3. Database - PostgreSQL 15.5

Anything else?

No response

[hors]

Hi @konoox, we have a task about it https://perconadev.atlassian.net/browse/K8SPG-502. We will fix this bug in the next release.

[mfilipe]

I have this problem using 2.4.0. pg-backup are being deleted but jobsin Kubernetes are not being cleaned up.

Should I set spec.backups.pgbackrest.jobs.ttlSecondsAfterFinished?

[Alemiz112]

Hi, I am facing the same problem with version 2.5.0.

The backups are configured as following:

  backups:
    pgbackrest:
      image: perconalab/percona-postgresql-operator:main-ppg16-pgbackrest
      configuration:
        - secret:
            name: main-pg-cluster-backup-secrets

      global:
        repo1-retention-full: "4" # keep 4 full backups = 4 weeks
        repo1-retention-full-type: count
        repo1-retention-diff: "14" # keep diff backups for 14 days

      jobs:
        ttlSecondsAfterFinished: 259200 # keep jobs for 3 days

This seems to correctly delete backups from repo1 as well as deletes associated Job resources, however PerconaPGBackups are not being deleted.

[afms-jsnouffer]

I am also encountering this issue with 2.5.0 with jobs.ttlSecondsAfterFinished configured. Defining ttlSecondsAfterFinished to allow Jobs to be auto-cleaned up seems to break the operator's ability to cleanup the associated PerconaPGBackup object.

The percona operator logs print the following: failed to cleanup outdated backups {"controller": "perconapgcluster", "controllerGroup": "pgv2.percona.com", "controllerKind": "PerconaPGCluster", "PerconaPGCluster": {"name":"...","namespace":"..."}, "namespace": "...", "name": "...", "reconcileID": "...", "error": "get backup job: Job.batch \"...\" not found" ....

This problematic check seems to be here.