submodules are not cached

[AlexisTM]

Problem

When doing a recursive shallow clone, I expect every submodule to be also cached and fetched through git-cache.

 git cache clone -U https://github.com/PX4/PX4-Autopilot --recurse-submodules --shallow-submodules

[kaspar030]

When doing a recursive shallow clone, I expect every submodule to be also cached and fetched through git-cache.

Thanks for the suggestion. I'll see what I can do!

[kaspar030]

@AlexisTM do you mind testing this?

You'd need to use the git version (cargo install --git https://github.com/kaspar030/git-cache-rs). Hint: increase default fetch jobs, like git config --global submodule.fetchJobs 16.

[AlexisTM]

Oh, wow. You made my day.
I made a quick bash wrapper for the recursive cloning while you integrated it to the tool & made the process parallel.

If you need more ideas on missing features, I would mention:

• Remote server for the cache storage (Allows for multiple CI agents to use the same remote to use/store the cache; Sadly, a simple shared folder would cause data races)
• git cache submodule -U update --recursive (Update all remote submodules recursively and update them locally from the cache)

[AlexisTM]

It works amazingly btw. I didn't add it to Jenkins yet but locally, it went from over 20 minutes to 11 seconds for the "fresh" clones.

[kaspar030]

It works amazingly btw. I didn't add it to Jenkins yet but locally, it went from over 20 minutes to 11 seconds for the "fresh" clones.

Happy to hear, and thanks for testing!

[kaspar030]

• Remote server for the cache storage (Allows for multiple CI agents to use the same remote to use/store the cache; Sadly, a simple shared folder would cause data races)

So git-cache does do locking, using a shared .gitcache concurrently should actually work. This is depended on heavily by git-cache's initial use case, https://ci.riot-os.org/. So it does work on local file systems. If it doesn't for you, please open an issue with more details (which fs etc.!).

Being able to act as a server, or have a tree of caches, would be great. I know I'll get nerd-sniped one day.

git cache submodule -U update --recursive (Update all remote submodules recursively and update them locally from the cache)

That makes sense, I'll add that. (Not this week though. 😇)

[AlexisTM]

That is great to know! Thanks for the tool :)