K8SPSMDB-1265: clean up basmdb-backup and use kubectl wait for wait_restore

[eleo007]

CHANGE DESCRIPTION

Problem:
Hi. The following problems should be fixed:

1. From time to time wait_restore() does not detect that object has requested state and fails, although eventually restore gets ready state.
2. The presence of the backup in minio storage fails in demand-backup-sharded.
3. The backups are not deleted from the storage after tests as a result resync requires a lot of time and tests run long.

Cause:
Short explanation of the root cause of the issue if applicable.

Solution:

1. Use kubectl wait instead of regular loop.
2. Add retry.
3. Delete backups before removing finalizers from objects.

CHECKLIST

Jira

• Is the Jira ticket created and referenced properly?
• Does the Jira ticket have the proper statuses for documentation (Needs Doc) and QA (Needs QA)?
• Does the Jira ticket link to the proper milestone (Fix Version field)?

Tests

• Is an E2E test/test case added for the new feature/change?
• Are unit tests added where appropriate?
• Are OpenShift compare files changed for E2E tests (compare/*-oc.yml)?

Config/Logging/Testability

• Are all needed new/changed options added to default YAML files?
• Are all needed new/changed options added to the Helm Chart?
• Did we add proper logging messages for operator actions?
• Did we ensure compatibility with the previous version or cluster upgrade process?
• Does the change support oldest and newest supported MongoDB version?
• Does the change support oldest and newest supported Kubernetes version?