TODO: Add guide to cleaning up backup resources #401
Description
Environment
- OS: Ubuntu 24.04 LTS
- k3s version: v1.29.5+k3s1 (4e53a323)
- AWX Operator: 2.19.0
Description
I have had a backup script running forever, that follows the backup guide. It has been working flawlessly, but it has started failing every time since 2024-11-27. There have been made no modifications from the time since the backup job was functioning perfectly, until now where it has been failing every time I attempt to make a backup.
Step to Reproduce
Unsure, have not attempted to reproduce on a clean setup as daily backups have worked fine.
The command that fails is kubectl apply -f "{{ awx_k3s_repo_dir }}/awx-on-k3s/backup/awxbackup.yaml"
Logs
When running kubectl apply -f "/backup/awxbackup.yaml" it finishes immediately instead of doing the backup, and logs displayed by kubectl -n awx logs -f deployments/awx-operator-controller-manager do not tell me anything I can make sense of.
kubectl -n awx logs -f deployments/awx-operator-controller-manager
{"level":"info","ts":"2024-12-18T10:31:48Z","logger":"cmd","msg":"Version","Go Version":"go1.20.12","GOOS":"linux","GOARCH":"amd64","ansible-operator":"v1.34.0","commit":"d26c43bf94960d292152862a6685696be33190fb"}
{"level":"info","ts":"2024-12-18T10:31:48Z","logger":"cmd","msg":"Watching namespaces","namespaces":["awx"]}
{"level":"info","ts":"2024-12-18T10:31:48Z","logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_AWX_AWX_ANSIBLE_COM","default":2}
{"level":"info","ts":"2024-12-18T10:31:48Z","logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_AWXBACKUP_AWX_ANSIBLE_COM","default":2}
{"level":"info","ts":"2024-12-18T10:31:48Z","logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_AWXRESTORE_AWX_ANSIBLE_COM","default":2}
{"level":"info","ts":"2024-12-18T10:31:48Z","logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_AWXMESHINGRESS_AWX_ANSIBLE_COM","default":2}
{"level":"info","ts":"2024-12-18T10:31:48Z","logger":"ansible-controller","msg":"Watching resource","Options.Group":"awx.ansible.com","Options.Version":"v1beta1","Options.Kind":"AWX"}
{"level":"info","ts":"2024-12-18T10:31:48Z","logger":"ansible-controller","msg":"Watching resource","Options.Group":"awx.ansible.com","Options.Version":"v1beta1","Options.Kind":"AWXBackup"}
{"level":"info","ts":"2024-12-18T10:31:48Z","logger":"ansible-controller","msg":"Watching resource","Options.Group":"awx.ansible.com","Options.Version":"v1beta1","Options.Kind":"AWXRestore"}
{"level":"info","ts":"2024-12-18T10:31:48Z","logger":"ansible-controller","msg":"Watching resource","Options.Group":"awx.ansible.com","Options.Version":"v1alpha1","Options.Kind":"AWXMeshIngress"}
{"level":"info","ts":"2024-12-18T10:31:48Z","logger":"proxy","msg":"Starting to serve","Address":"127.0.0.1:8888"}
{"level":"info","ts":"2024-12-18T10:31:48Z","logger":"apiserver","msg":"Starting to serve metrics listener","Address":"localhost:5050"}
{"level":"info","ts":"2024-12-18T10:31:48Z","msg":"starting server","kind":"health probe","addr":"[::]:6789"}
{"level":"info","ts":"2024-12-18T10:31:48Z","logger":"controller-runtime.metrics","msg":"Starting metrics server"}
{"level":"info","ts":"2024-12-18T10:31:48Z","logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":"127.0.0.1:8080","secure":false}
I1218 10:31:48.821221 7 leaderelection.go:250] attempting to acquire leader lease awx/awx-operator...
I1218 10:32:05.084628 7 leaderelection.go:260] successfully acquired lease awx/awx-operator
{"level":"info","ts":"2024-12-18T10:32:05Z","msg":"Starting EventSource","controller":"awx-controller","source":"kind source: *unstructured.Unstructured"}
{"level":"info","ts":"2024-12-18T10:32:05Z","msg":"Starting EventSource","controller":"awxrestore-controller","source":"kind source: *unstructured.Unstructured"}
{"level":"info","ts":"2024-12-18T10:32:05Z","msg":"Starting EventSource","controller":"awxmeshingress-controller","source":"kind source: *unstructured.Unstructured"}
{"level":"info","ts":"2024-12-18T10:32:05Z","msg":"Starting Controller","controller":"awx-controller"}
{"level":"info","ts":"2024-12-18T10:32:05Z","msg":"Starting Controller","controller":"awxmeshingress-controller"}
{"level":"info","ts":"2024-12-18T10:32:05Z","msg":"Starting Controller","controller":"awxrestore-controller"}
{"level":"info","ts":"2024-12-18T10:32:05Z","msg":"Starting EventSource","controller":"awxbackup-controller","source":"kind source: *unstructured.Unstructured"}
{"level":"info","ts":"2024-12-18T10:32:05Z","msg":"Starting Controller","controller":"awxbackup-controller"}
{"level":"info","ts":"2024-12-18T10:32:05Z","msg":"Starting workers","controller":"awx-controller","worker count":32}
{"level":"info","ts":"2024-12-18T10:32:05Z","msg":"Starting workers","controller":"awxmeshingress-controller","worker count":32}
{"level":"info","ts":"2024-12-18T10:32:05Z","msg":"Starting workers","controller":"awxrestore-controller","worker count":32}
{"level":"info","ts":"2024-12-18T10:32:05Z","msg":"Starting workers","controller":"awxbackup-controller","worker count":32}
{"level":"info","ts":"2024-12-18T10:32:05Z","logger":"KubeAPIWarningLogger","msg":"unknown field \"status.conditions[0].message\""}
{"level":"info","ts":"2024-12-18T10:32:05Z","logger":"KubeAPIWarningLogger","msg":"unknown field \"status.conditions[1].message\""}
{"level":"info","ts":"2024-12-18T10:32:05Z","logger":"KubeAPIWarningLogger","msg":"unknown field \"status.conditions[2].message\""}
I1218 10:32:06.277917 7 request.go:697] Waited for 1.028919415s due to client-side throttling, not priority and fairness, request: PUT:https://10.43.0.1:443/apis/awx.ansible.com/v1beta1/namespaces/awx/awxbackups/awx-backup-20240716080901/status
When I check the awxbackup objects I see a long list of old backup objects, and it does create a new one every time I apply the backup:
kubectl -n awx get awxbackup
NAME AGE
<100s of other awx-backup-202x objects>
awx-backup-20241218100946 22m
awx-backup-20241218102752 4m40s
Files
my awxbackup.yml is defined as
---
apiVersion: awx.ansible.com/v1beta1
kind: AWXBackup
metadata:
name: {{ filename_startswith }}-{{ current_date_utc }}
namespace: awx
spec:
deployment_name: awx
backup_pvc: awx-backup-claim