[Issue] Linux gfx950-dcgpu rocblas: — `rtest.py reports "FAILED xml test suite!" despite 0 gtest failures`

[TriveniTadapaneni]

Context

• PR: Bump rocm-libraries from 37b9981 to 39d237a TheRock#6512 — bump rocm-libraries from 37b9981ba57 to 39d237aab48
• Failing run: https://github.com/ROCm/TheRock/actions/runs/29263391220
• Failing job:
  Test rocblas (shard 5/6) (gfx950-dcgpu)
  Test rocblas (shard 3/6) (gfx950-dcgpu)
  Test rocblas (shard 1/6) (gfx950-dcgpu) — job 87062203568
• Component: rocblas
• Platform: Linux
• Affected arch: gfx950-dcgpu

Error

2: [ PASSED   ] 98937 tests.
2: [ FAILED   ] 0 tests.
2: command line: rocblas-test --gtest_output=xml --gtest_color=yes --gtest_filter=*quick*:*pre_checkin*-*known_bug*
2: FAILED xml test suite!
1/1 Test #2: rocblas-test_standard_suite ......***Failed  616.56 sec
##[error]Process completed with exit code 8.

Suspected Root Cause

Commit 8a7acc7b5a9b ("feat: [rocBLAS] expand supported ctest options #8984") adds USE_RTEST_DRIVER to apply_test_category_labels() in projects/rocblas/clients/gtest/CMakeLists.txt. This routes CTest for the standard suite through python3 rocblas_rtest.py -t ctest_standard instead of directly invoking the rocblas-test GTest binary.

rtest.py's run_cmd() scans every stdout line with re.search(r'error|fail', outstring, re.IGNORECASE). GTest always prints [ FAILED ] 0 tests. at the end of every run — even when zero tests fail — and this line matches the pattern. The text scan overrides the process exit code (0 = all pass) to 1 and prints FAILED xml test suite!.

AI Suggested Fix

In projects/rocblas/rtest.py, exclude GTest bracket-summary lines from the error scan:

# projects/rocblas/rtest.py — run_cmd(), inside the stdout readline loop
# BEFORE:
                    error = error or re.search(r'error|fail', outstring, re.IGNORECASE)

# AFTER: skip GTest summary lines like "[ FAILED   ] 0 tests."
                    if not re.match(r'^\s*[[\s\w]+]\s*\d+\s+tests?', outstring, re.IGNORECASE):
                        error = error or re.search(r'error|fail', outstring, re.IGNORECASE)

[amd-chiranjeevi]

@TorreZuk Can you please check this issue ?

[amd-chiranjeevi]

Testing the revert of suspected root cause
revert PR: #9394
Test Run: https://github.com/ROCm/TheRock/actions/runs/29329663677

[TorreZuk]

Ahh yes can adjust the regular expression, not sure why the PR CI reports success.

[TorreZuk]

Okay this is optional output but the shards that skipped tests produce it. This was a lie by AI " GTest always prints " no it is optional for rocblas and that explains why the PR passed.

[amd-chiranjeevi]

Ahh yes can adjust the regular expression, not sure why the PR CI reports success.

What was the reason it passed in the gfx94x job but failed in the gfx950 job?
Also, can you check whether these failures can be ignored and treated as non-blockers for the rocm-libraries bump?

[TorreZuk]

Ahh yes can adjust the regular expression, not sure why the PR CI reports success.

What was the reason it passed in the gfx94x job but failed in the gfx950 job? Also, can you check whether these failures can be ignored and treated as non-blockers for the rocm-libraries bump?

It is optional output, it only shows [FAILED] 0 if there are skipped tests, I didn't get any skipped locally, and the PR runs were lucky and didn't skip. As for why you are getting skipped test is another issue to investigate, but they are normal behaviour to skip if support or memory is not sufficient