OCPBUGS-49347: Degraded machine-config CO due to ManagedBootImages update when upgrading an OCL cluster

[djoshy]

Closes: https://issues.redhat.com/browse/OCPBUGS-49347

- What I did

• I've moved the "stamp the boot images" function call into syncMachineConfiguration. It was originally called in syncRequiredMachineConfigPools, but in cases where the master pool update(for example, OCL upgrades) took a long time and due to the ordering of the syncs, there were potential scenarios where the stamps would be never applied. By placing this into syncMachineConfiguration, this should be no longer take place.
• I've also updated the call to only explicitly check for master pools, instead of "required" pools(which has really just been master pools, but made things confusing)

- How to verify it

• This is hard to reproduce based on the bug, but OCL upgrades should be a good way to stress test this.
• Existing e2es should pass.

[openshift-ci-robot]

@djoshy: This pull request references Jira Issue OCPBUGS-49347, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.19.0) matches configured target version for branch (4.19.0)
• bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @sergiordlr

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

Closes: https://issues.redhat.com/browse/OCPBUGS-49347

- What I did

• I've moved the "stamp the boot images" function call into syncMachineConfiguration. It was originally called in syncRequiredMachineConfigPools, but in cases where the master pool update(for example, OCL upgrades) took a long time and due to the ordering of the syncs, there were potential scenarios where the stamps would be never applied. By placing this into syncMachineConfiguration, this should be no longer take place.
• I've also updated the call to only explicitly check for master pools, instead of "required" pools(which has really just been master pools, but made things confusing)

- How to verify it

• This is hard to reproduce based on the bug, but OCL upgrades should be a good way to stress test this.
• Existing e2es should pass.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[djoshy]

/test e2e-gcp-op-ocl

[sergiordlr]

Upgrade execution:
https://qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gs/qe-private-deck/logs/periodic-ci-openshift-openshift-tests-private-release-4.19-amd64-nightly-4.19-upgrade-from-stable-4.19-aws-ipi-ocl-fips-f60/1886820025159913472

New upgrade execution:
https://qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gs/qe-private-deck/logs/periodic-ci-openshift-openshift-tests-private-release-4.19-amd64-nightly-4.19-upgrade-from-stable-4.19-aws-ipi-ocl-fips-f60/1887065122124337152

New upgrade execution:
https://qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gs/qe-private-deck/logs/periodic-ci-openshift-openshift-tests-private-release-4.19-amd64-nightly-4.19-upgrade-from-stable-4.19-aws-ipi-ocl-fips-f60/1887464170945777664

[sergiordlr]

All managedbootimages 2e2 test cases passed, and an the upgrades didn't fail.

/label qe-approved

[openshift-ci-robot]

@djoshy: This pull request references Jira Issue OCPBUGS-49347, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.19.0) matches configured target version for branch (4.19.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @sergiordlr

In response to this:

Closes: https://issues.redhat.com/browse/OCPBUGS-49347

- What I did

• I've moved the "stamp the boot images" function call into syncMachineConfiguration. It was originally called in syncRequiredMachineConfigPools, but in cases where the master pool update(for example, OCL upgrades) took a long time and due to the ordering of the syncs, there were potential scenarios where the stamps would be never applied. By placing this into syncMachineConfiguration, this should be no longer take place.
• I've also updated the call to only explicitly check for master pools, instead of "required" pools(which has really just been master pools, but made things confusing)

- How to verify it

• This is hard to reproduce based on the bug, but OCL upgrades should be a good way to stress test this.
• Existing e2es should pass.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[djoshy]

/label acknowledge-critical-fixes-only

This should have minimal impact on payloads and is necessary for OCL GA.

[isabella-janssen]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: djoshy, isabella-janssen

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [djoshy,isabella-janssen]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

@djoshy: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-azure-ovn-upgrade-out-of-change	aff2c36	link	false	/test e2e-azure-ovn-upgrade-out-of-change
ci/prow/bootstrap-unit	aff2c36	link	false	/test bootstrap-unit

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[openshift-ci-robot]

/retest-required

Remaining retests: 0 against base HEAD 5ad8612 and 2 for PR HEAD aff2c36 in total

[openshift-ci-robot]

/retest-required

Remaining retests: 0 against base HEAD c26bdb0 and 1 for PR HEAD aff2c36 in total

[openshift-ci-robot]

@djoshy: Jira Issue OCPBUGS-49347: All pull requests linked via external trackers have merged:

• openshift/machine-config-operator#4810

Jira Issue OCPBUGS-49347 has been moved to the MODIFIED state.

In response to this:

Closes: https://issues.redhat.com/browse/OCPBUGS-49347

- What I did

• I've moved the "stamp the boot images" function call into syncMachineConfiguration. It was originally called in syncRequiredMachineConfigPools, but in cases where the master pool update(for example, OCL upgrades) took a long time and due to the ordering of the syncs, there were potential scenarios where the stamps would be never applied. By placing this into syncMachineConfiguration, this should be no longer take place.
• I've also updated the call to only explicitly check for master pools, instead of "required" pools(which has really just been master pools, but made things confusing)

- How to verify it

• This is hard to reproduce based on the bug, but OCL upgrades should be a good way to stress test this.
• Existing e2es should pass.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-bot]

[ART PR BUILD NOTIFIER]

Distgit: ose-machine-config-operator
This PR has been included in build ose-machine-config-operator-container-v4.19.0-202502150439.p0.gd1aebf3.assembly.stream.el9.
All builds following this will include this PR.