Skip to content

Conversation

@aleguy02
Copy link

@aleguy02 aleguy02 commented Oct 2, 2025

Description

This PR automates uploading JUnit reports to codecov in the test pipeline. An example pipeline run with the changes can be found here (https://github.com/aleguy02/fork-pytest/actions/runs/18184970357). An example of what the codecov UI looks like with these changes can be found here (https://app.codecov.io/gh/aleguy02/fork-pytest/tests).

Closes #12689

Checklist

  • Include documentation when adding new features.
  • Include new tests or update existing tests when applicable.
  • Allow maintainers to push and squash when merging my commits. Please uncheck this if you prefer to squash the commits yourself.

Additional Notes

I didn't create a changelog since the documented behavior of pytest was not modified.

@bluetech bluetech added the skip news used on prs to opt out of the changelog requirement label Oct 3, 2025
Copy link
Member

@webknjaz webknjaz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a great start! I normally avoid adding --junitxml= (even if the path value is empty) to normal/local invocations. Instead, I like adding the entire CLI flag in the CI definition: https://github.com/cherrypy/cheroot/blob/06a6d67e549ccaefe332cd0539c21e1483031071/.github/workflows/ci-cd.yml#L786.

I've made suggestions below. Let's see if that works..


As for the change note, sometimes it's a good idea to use the contrib type for such changes.

@aleguy02
Copy link
Author

aleguy02 commented Nov 22, 2025

Do you recommend doing this because it is cleaner to have the entire CLI flag in one place instead of splitting it across the workflow and tox.ini file? If so that makes perfect sense (I'm asking for my personal learning) @webknjaz

@aleguy02 aleguy02 force-pushed the upload-junit-reports-codecov branch from 3e622d3 to 6af094b Compare November 22, 2025 18:13
@webknjaz webknjaz removed the skip news used on prs to opt out of the changelog requirement label Nov 25, 2025
@psf-chronographer psf-chronographer bot added the bot:chronographer:provided (automation) changelog entry is part of PR label Nov 25, 2025
@@ -0,0 +1 @@
automated uploading JUnit reports to codecov in the test workflow with codecov GitHub Action
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change notes are supposed to communicate the effect of upgrading since the previous release. They make up a coherent change log document, which is expected to follow consistent writing style and use full sentences and such. They shouldn't generally reference internal implementation details and be higher level and approachable to the end-users.

This is a contrib note, though, so the "end users" are other contributors (and core devs).

Here's a preview of your addition: https://pytest--13778.org.readthedocs.build/en/13778/changelog.html#contributor-facing-changes.

To me, it looks a bit out of place because it doesn't have a full stop in the sentence, nor does the sentence start with a capital letter.

This will probably fit better:

Suggested change
automated uploading JUnit reports to codecov in the test workflow with codecov GitHub Action
The test reports are now published to Codecov from GitHub Actions.
The test statistics is visible `on the web interface
<https://app.codecov.io/gh/pytest-dev/pytest/tests>`__.
-- by :user:`aleguy02`

P.S. Technically, this PR does not require a change note but since you added it, let's make it more coherent with the rest.

if: "! matrix.use_coverage"
shell: bash
env:
_PYTEST_TOX_POSARGS_JUNIT: --junitxml=junit-${{ matrix.name }}.xml
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this produces one file per job that doesn't go anywhere with this name, we can skip making the name dynamic.

Suggested change
_PYTEST_TOX_POSARGS_JUNIT: --junitxml=junit-${{ matrix.name }}.xml
_PYTEST_TOX_POSARGS_JUNIT: --junitxml=junit.xml

if: "matrix.use_coverage"
shell: bash
env:
_PYTEST_TOX_POSARGS_JUNIT: --junitxml=junit-${{ matrix.name }}.xml
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
_PYTEST_TOX_POSARGS_JUNIT: --junitxml=junit-${{ matrix.name }}.xml
_PYTEST_TOX_POSARGS_JUNIT: --junitxml=junit.xml

uses: codecov/codecov-action@5a1091511ad55cbe89839c7260b706298ca349f7
with:
fail_ci_if_error: false
files: junit-${{ matrix.name }}.xml
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
files: junit-${{ matrix.name }}.xml
files: junit.xml

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For consistency with the coverage upload step done above this one, let's use ./junit.xml.

@webknjaz
Copy link
Member

Do you recommend doing this because it is cleaner to have the entire CLI flag in one place instead of splitting it across the workflow and tox.ini file? If so that makes perfect sense (I'm asking for my personal learning)

Perhaps. But this is more due to the fact that configuration should be where it belongs. If you add a --junitxml= to tox.ini, that configuration will also be picked up in local runs, when the contributors invoke tox on their computers. Setting it only in CI prevents that.

@webknjaz webknjaz added type: infrastructure improvement to development/releases/CI structure backport 9.0.x apply to PRs at any point; backports the changes to the 9.0.x branch labels Nov 25, 2025
Copy link
Member

@nicoddemus nicoddemus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good, thanks @aleguy02.

Please see the requested changes by @webknjaz. 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport 9.0.x apply to PRs at any point; backports the changes to the 9.0.x branch bot:chronographer:provided (automation) changelog entry is part of PR type: infrastructure improvement to development/releases/CI structure

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[TODO] Upload JUnit reports to Codecov

4 participants