chore(ci_visibility): early exit for retry mechanisms

[gnufede]

PR by Bits
View session in Datadog

Comment @DataDog to request changes

Feedback (especially what can be better) welcome in #code-gen-aka-bits-dev-feedback!

---

Description

Implements early exit logic for retry mechanisms to reduce unnecessary overhead on customer CI. This change prevents scheduling further test retries once the purpose of each retry mechanism has been fulfilled:

• Auto Test Retries: Stop retrying after the first successful test execution
• Early Flake Detection: Stop retrying once there has been at least one failed and one successful execution
• Attempt to Fix: Stop retrying after the first failed execution

The implementation adds state tracking to the Test class to record whether it has passed or failed at any point, then uses these flags in each retry handler's should_retry() method to determine early exit conditions.

Testing

Updated test expectations in test_pytest_efd.py to reflect the reduced number of retries:

• EFD flaky test case now expects 2 test runs instead of 11
• EFD skip-then-fail case now expects 3 test runs instead of 11
• All event counts and output assertions updated accordingly
• Tests verify that retries stop as soon as early exit conditions are met

Risks

Low risk. The changes are additive and only add early exit conditions without modifying existing retry logic. Behavior is only affected when early exit conditions are met, which improves efficiency without changing correctness.

Additional Notes

The implementation leverages the existing set_status() method on TestRun to track test outcomes, ensuring state is updated whenever a test status is recorded. This approach maintains a single source of truth for test status information.

[datadog-datadog-prod-us1-2]

[View session in Datadog]

Bits Dev status: ✅ Done

CI Auto-fix: Disabled | Enable

Comment @DataDog to request changes

[cit-pr-commenter-54b7da]

Codeowners resolved as

ddtrace/testing/internal/retry_handlers.py                              @DataDog/ci-app-libraries
ddtrace/testing/internal/test_data.py                                   @DataDog/ci-app-libraries
tests/testing/internal/pytest/test_pytest_efd.py                        @DataDog/ci-app-libraries