Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI: Generating the e2e test matrix #5763

Merged
merged 1 commit into from
Feb 4, 2025
Merged

CI: Generating the e2e test matrix #5763

merged 1 commit into from
Feb 4, 2025

Conversation

kantai
Copy link
Member

@kantai kantai commented Jan 28, 2025

This PR is a first pass at trying to generate the test matrix automatically. Right now, I'm just trying to make sure that it gets a single test to run using the excludes functions. Once I'm convinced that works, I'll update it to match the current exclusion set.

@kantai kantai requested a review from a team as a code owner January 28, 2025 22:02
@wileyj
Copy link
Collaborator

wileyj commented Jan 29, 2025

interesting!

have you considred nextest's filtersets here as well? we'd still need a matrix, but the pattern matching might be easier than what you're doing here

https://nexte.st/docs/filtersets/

@kantai
Copy link
Member Author

kantai commented Jan 29, 2025

have you considred nextest's filtersets here as well? we'd still need a matrix, but the pattern matching might be easier than what you're doing here

After looking into it, I don't think it would help that much. The filterset language doesn't support matching "ignored" only (which is how we indicate these tests), so it would only be helpful to match the binary stacks-node. But it wouldn't change the jq filter at all, because the JSON still needs to be altered to match the matrix spec.

@kantai kantai force-pushed the ci/generate-tests branch from ef5045d to 4d96cf9 Compare January 29, 2025 15:28
@kantai kantai self-assigned this Jan 29, 2025
jferrant
jferrant previously approved these changes Jan 29, 2025
* use nextest list to generate the set of e2e tests for bitcoin-tests (this switches the workflow from being an "inclusion list" to an "exclusion list").
* this changeset matches the current set of tests run in `develop`. it does not try to enable tests that were perhaps overlooked.
@kantai kantai force-pushed the ci/generate-tests branch from 0bdcd92 to b6e5b91 Compare February 3, 2025 22:28
@kantai kantai requested review from jferrant and wileyj February 4, 2025 15:14
@kantai kantai added this to the 3.1.0.0.6 milestone Feb 4, 2025
@wileyj
Copy link
Collaborator

wileyj commented Feb 4, 2025

this is neat! the only thing we should be aware of is it'll be easy to overwhelm the amount of tests in the matrix, so it's something we'll want to keep an eye on over time.

@kantai kantai requested review from jcnelson, hstove and obycode February 4, 2025 21:01
Copy link
Contributor

@obycode obycode left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM and opened #5802 for future enhancement.

@kantai kantai added this pull request to the merge queue Feb 4, 2025
Merged via the queue into develop with commit 91a1398 Feb 4, 2025
175 checks passed
@kantai kantai deleted the ci/generate-tests branch February 4, 2025 22:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Status: ✅ Done
Development

Successfully merging this pull request may close these issues.

4 participants