Fix everest fixture for caching testcase results #9928
Conversation
jonathan-eq
force-pushed
the
fix-flaky-test
branch
from
b69d5cb to
15575ad
Compare
CodSpeed Performance ReportMerging #9928 will not alter performanceComparing Summary
|
jonathan-eq
force-pushed
the
fix-flaky-test
branch
2 times, most recently
from
6066a8c to
8c2b68a
Compare
jonathan-eq
force-pushed
the
fix-flaky-test
branch
from
8c2b68a to
783d311
Compare
tests/everest/test_api_snapshots.py
Outdated
| @@ -74,6 +74,9 @@ def test_api_snapshots(config_file, snapshot, cached_example): | |||
| .strip() | |||
| + "\n" | |||
| ) | |||
| snapshot.snapshot_dir = ( | |||
There was a problem hiding this comment.
Is this part necessary? The snapshots are currently already scoped/split without this?
There was a problem hiding this comment.
You are right, it was pytest-repeat that changed the snapshot names
| cache = pytestconfig.cache | ||
|
|
||
| def run_config(test_data_case: str): | ||
| if cache.get(f"cached_example:{test_data_case}", None) is None: | ||
| if ( |
There was a problem hiding this comment.
I think this makes the everest tests take 16m to run (as opposed to currently 6-7mins) as it now has to create one set of complete runs per runner, in this case running sequential might actually be faster/cheaper than parallel. So it would be ideal to be able to share this cache thing between runners and look for another way to avoid the collision. I don't think I understand the problem this solves in full detail, nor whether this is the right way to go about it.
There was a problem hiding this comment.
The way this is solved in ert is by using --dist loadgroup to put all workflows that require the cached testdata run in the same process.
jonathan-eq
force-pushed
the
fix-flaky-test
branch
from
783d311 to
38e44e0
Compare
The fixture for caching had a race condition, where multiple workers tried setting the cache.
jonathan-eq
force-pushed
the
fix-flaky-test
branch
from
38e44e0 to
28bd0c3
Compare
|
Superseded by #10041 |
Issue
Resolves #9670
Approach
pytest-snapshot does not go well with pytest-xdist, so we have to manually set the snapshot path to avoid it using the worker-ids in the path.
(Screenshot of new behavior in GUI if applicable)
git rebase -i main --exec 'just rapid-tests')When applicable