[Test] Add integ test to capture fragilities in cluster update workflow by gmarciani · Pull Request #7275 · aws/aws-parallelcluster

gmarciani · 2026-03-10T00:10:47Z

Description of changes

Add end to end test to capture fragilities in cluster update workflow.
The test validates the following scenarios:

a node that is bootstrapping during the update and completes the bootstrap right after the update, is able to deploy the update config.
a node that completes the bootstrap during the readiness check and does not deploy the updated config on time, is reported by the check, but does not cause failure and it is able to deploy the update config after the check.
a node that faces transient update failures, is able to recover thanks to the retries
a node that is bootstrapped right before the update is able to complete the bootstrap and deploy the updated config.

Auxiliary changes to common functions:

Add utility to wait for a set of acceptable stack statuses
Allow to search for patterns in logs with a timestamp constraint
Allow to retrieve dna.json from on login nodes
Allow to retrieve cluster config deployed on login nodes
Fixed the logic to submit a non blocking cluster update.

Tests

PENDING test_update_race_conditions

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

tests/integration-tests/tests/update/test_update_readiness_check.py

tests/integration-tests/tests/update/test_update_race_conditions.py

codecov · 2026-03-10T16:47:39Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.08%. Comparing base (d4d705d) to head (945feaa).
⚠️ Report is 50 commits behind head on develop.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #7275      +/-   ##
===========================================
+ Coverage    90.06%   90.08%   +0.01%     
===========================================
  Files          182      182              
  Lines        16646    16730      +84     
===========================================
+ Hits         14992    15071      +79     
- Misses        1654     1659       +5

Flag	Coverage Δ
unittests	`90.08% <ø> (+0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

tests/integration-tests/tests/update/test_update_race_conditions.py

gmarciani · 2026-03-11T23:42:02Z

tests/integration-tests/conftest.py

+# def pytest_collection_finish(session):
+#     _log_collected_tests(session)
+#     # Create the metadata table after the regions are collected
+#     try:
+#         region = session.config.getoption("regions") or get_all_regions(session.config.getoption("tests_config"))
+#         region = [unmarshal_az_override(az) for az in region]
+#         # Use the first element of the list of regions, since there must be at least one
+#         reporting_region = get_reporting_region(region[0])
+#         logging.info(f"Metadata reporting region {reporting_region}")
+#         # Setup the metadata table in case it doesn't exist
+#         MetadataTableManager(reporting_region, METADATA_TABLE).create_metadata_table()
+#     except Exception as exc:
+#         logging.info(f"There was a '{type(exc)}' error with '{exc}' when creating the table!")


This commit is just for testing and will be removed

tests/integration-tests/tests/update/test_update_race_conditions.py

…l testing.

github-advanced-security bot found potential problems Mar 10, 2026

View reviewed changes

gmarciani changed the title ~~Wip/mgiacomo/3150/integ test update workflow 0305 1~~ [Test] Add integ test to capture fragilities in cluster update workflow Mar 10, 2026

gmarciani added skip-changelog-update Disables the check that enforces changelog updates in PRs 3.x Test labels Mar 10, 2026

gmarciani force-pushed the wip/mgiacomo/3150/integ-test-update-workflow-0305-1 branch from d9050f3 to 82904c3 Compare March 10, 2026 15:11

github-advanced-security bot found potential problems Mar 10, 2026

View reviewed changes

tests/integration-tests/tests/update/test_update_race_conditions.py Fixed Show fixed Hide fixed

tests/integration-tests/tests/update/test_update_race_conditions.py Fixed Show fixed Hide fixed

gmarciani force-pushed the wip/mgiacomo/3150/integ-test-update-workflow-0305-1 branch from 82904c3 to 9a9da62 Compare March 10, 2026 16:29

gmarciani force-pushed the wip/mgiacomo/3150/integ-test-update-workflow-0305-1 branch 4 times, most recently from d6f88e3 to 7f56e41 Compare March 10, 2026 18:17

github-advanced-security bot found potential problems Mar 10, 2026

View reviewed changes

tests/integration-tests/tests/update/test_update_race_conditions.py Fixed Show fixed Hide fixed

tests/integration-tests/tests/update/test_update_race_conditions.py Fixed Show fixed Hide fixed

gmarciani force-pushed the wip/mgiacomo/3150/integ-test-update-workflow-0305-1 branch 5 times, most recently from b293f22 to 82b0e94 Compare March 10, 2026 20:48

github-advanced-security bot found potential problems Mar 10, 2026

View reviewed changes

gmarciani force-pushed the wip/mgiacomo/3150/integ-test-update-workflow-0305-1 branch 5 times, most recently from d3540d1 to 687a4c2 Compare March 11, 2026 17:06