Python toolchain cache invalidation #2283
Labels
Comments
|
Can you provide more info? In particular, reproduction steps or output that indicates something isn't being cached. |
|
Yes, the hermetic interpreters are downloaded again and all the pip
dependencies are reinstalled
…On Wed, Oct 9, 2024 at 5:31 PM Richard Levasseur ***@***.***> wrote:
Can you provide more info? In particular, reproduction steps or output
that indicates something isn't being cached.
—
Reply to this email directly, view it on GitHub
<#2283 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABACPPIXUJ45BBMQV7CYJTZ2W4F3AVCNFSM6AAAAABPVPI5P6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMBTGYYTONJUGI>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
|
Does #2256 fix the cache invalidation you are seing? |
|
It appears so - I have not been able to reproduce using that commit. Closing. |
|
@ewhauser, could you add context for reopening this issue? |
|
Think I closed this prematurely. This behavior still seems to exist in |
|
I bisected and this is the offending commit: |
|
Interesting 68c3048 would have been my last guess here. What is the toolchain matching log if you pass FYI: @rickeylev |
|
Yes, I agree. I've bisected it 3 separate times and always ended up at this commit. Pinned to the previous commit in our monorepo and the problem is solved. The other piece of information I can give is that the repro seems to be:
and the dependent repositories are invalidated. I've tried to extract this into a simple repo without luck yet. |
|
That is a weird commit to be the cause. All that local toolchain stuff isn't active. Maybe the changes to repo_utils.bzl? It introduces some getenv() and watch() calls. Maybe something about that is over-triggering? |
|
Looking through the code, the new watch/getenv calls I see are:
which() is called as part some of the runtime repo setup, iirc. In repo_utils (https://github.com/bazelbuild/rules_python/blob/main/python/private/repo_utils.bzl), if you:
Does the invalidation go away? |
|
@ewhauser, is the latest RC still having the issue? |
|
Tested this morning against the RC and have not been able to reproduce. |
|
I have removed #2406 dependency from fetching of the toolchains and I wonder if that fixed it. I'll close this as done, but feel free to reopen if you see any issues. Thanks for the testing! |
Ubehebe
added a commit
to Ubehebe/rules_python_2283_mvce
that referenced
this issue
Dec 22, 2024
|
I've kept my eye on this bug for months, with my repository's rules_python version pinned at 0.32.2, hoping it would be resolved. With the recent release of bazel 8, it's becoming harder to keep rules_python pinned at that version, so I'd like to discuss this now. I have a minimal repro here: https://github.com/Ubehebe/rules_python_2283_mvce. It's a trivial repo with a git pre-commit hook that runs git has some sneaky semantics when it runs hooks. Specifically, git exposes the value of |
|
I should also mention that I tried to patch rules_python 1.0.0, via bazel's single_version_override, to revert 68c3048, which has been identified as a culprit earlier in this discussion. The results were inconclusive. I still observed the pip dependency invalidation. However, 68c3048 doesn't revert cleanly against 1.0.0, so it's possible I incorrectly resolved some conflicts. |
|
@Ubehebe, I think what you are describing could be a separate issue even if it originatesdue to similar reasons. Could you please create a new ticket? One way to debug this could be to print bazel info instead of bazel build so that we could rule out simple discrepancies between the envs (e.g. the repository cache location). |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
🐞 bug report
Affected Rule
rules_python 0.35.0Is this a regression?
Yes
Description
The latest version of
rules_pythonis not caching the Python toolchain correctly on OSX.🔬 Minimal Reproduction
🌍 Your Environment
Operating System:
Output of
bazel version:Rules_python version:
Anything else relevant?
This occurs in both
WORKSPACEand bzlmod builds.The text was updated successfully, but these errors were encountered: