Use scratchspaces rather than .julia/conda

[IanButterworth]

scratchpaces have become the standard place for things like that which are currently stored in .julia/conda.
Notably CondaPkg.jl uses scratchspaces.

Also, using a custom dir means special handling would be needed for tools like julia-actions/cache

[Krastanov]

Moving to scratchspaces will probably also fix CI issues where for some reason DEPOT/conda/3/x86_64 does not exist. It happens very often in CI and it seems the standard fix is to add mkdir -p "DEPOT/conda/3/x86_64" to CI scripts.

These issues are not reproducible locally for me, only in CI.

E.g. these two lines are necessary in my CI scripts:

• https://github.com/QuantumSavory/QuantumClifford.jl/blob/50d3b34b6a46e36121ff0bedec8c58e1d79774e4/.github/workflows/ci.yml#L50
• https://github.com/QuantumSavory/QuantumClifford.jl/blob/50d3b34b6a46e36121ff0bedec8c58e1d79774e4/.buildkite/pipeline.yml#L34

Without them, this happens at runtime:

I learnt of this workaround from slack where other people were complaining about similar issues. Copied here for archival:

Here is the link to the Zygote issue:

• https://github.com/FluxML/Zygote.jl/pull/1392/files

[IanButterworth]

Yes, either:

1. This issue needs to be fixed
2. Allow including custom depot directories julia-actions/cache#72 should be implemented
3. Conda.jl should somehow detect at runtime, after precompilation, if it needs to reinstall things

But note that 2) would require users to opt-in.

Doing 3) would probably be a good thing either way.

Until then users of Conda.jl are going to have failing because CI caching is becoming more widespread, so it would be wise to consider this an urgent issue

[Krastanov]

Just a minor addendum: some of these CI issues I mentioned above are independent of julia-actions/cache (e.g. the ones for buildkite)

[axsk]

I was directed here with the same issue, namely
ERROR: LoadError: InitError: could not load library "/home/runner/.julia/conda/3/x86_64/lib/libpython3.10.so.1.0" /home/runner/.julia/conda/3/x86_64/lib/libpython3.10.so.1.0: cannot open shared object file: No such file or directory
on every second CI run (when the cache is loaded). Every other run the cache was deleted and it works fine.
Will look into the workaround, thank you @Krastanov