gix-pack: provide access to index-less parallel pack traversal (cache::delta::Tree)

[Roberto Di Cosmo (rdicosmo)]

gix-pack already has the machinery to resolve the objects of a self-contained pack in parallel directly from a header scan: build a cache::delta::Tree and call Tree::traverse(). This is essentially the path used internally by index::File::traverse_with_index().

Since cache::delta is pub(crate) this path cannot currently be used outside the crate.

This is precious functionality for our use case (and possibly more), and I am requesting advice on how to proceed to get a public way to access it.

The use case

At Software Heritage we work on archiving and deduplicating source code at scale. We fetch packfiles from very large numbers of repositories and, for each freshly fetched self-contained, non-thin pack (ofs-delta), need to walk the pack once to compute content hashes and convert the objects. Since the packs are self-contained and we recompute and check object hashes downstream, we do not need a verified traversal, and building a complete .idx just to traverse the pack is unnecessary.

This matters at our scale: Linux-sized repositories are around 13.5M objects, Chromium around 28M, and we process millions of packs. Avoiding a complete intermediate index construction avoids significant waste of resources.

The entry graph can already be built directly from the pack header scan, by pack offset, which is precisely what the existing gitoxide traversal machinery does internally.

We just need a supported way to access this machinery.

Possible API shape

The apparently simple approach, pub mod delta, seems too broad: it exposes ~20 internal items, creates a documentation burden with #![deny(missing_docs)], and unnecessarily makes those internals part of the public API.

The smallest change I found would be a facade re-export of just what the caller needs:

pub use delta::{Tree, traverse::{Context, Options}};

Those three items are already documented, while delta itself can remain pub(crate).

But I seeat least two reasons why you may prefer something narrower:

• The new lock-free resolve scheduler in fix: resolve delta trees lazily in a lock-free pool #2852 changed Options/Context: making Options/Context public may constrain exactly this kind of future refactoring.
• Exposing Tree also exposes the hand-driven builder, not a particularly attractive public API.

The ask

We are not attached to any particulr API shape, what we need is the capability: build the delta dependency tree directly from a streaming pack header scan and resolve it in parallel, without first materialising an index.

Which approach makes sense for you? Once the decision is make on your side, I'm happy to prepare the PR, including a small doc-test anchoring the index-less traversal path.

[Sebastian Thiel (Byron)]

I will keep it short as I think I talk to Claude here, which if I were would be violating the contribution guidelines that requires clear separation of speech.

pub use delta::{Tree, traverse::{Context, Options}}; sounds good, go for it. No worries about exposing what's there or breaking changes later, that's what plumbing is there for. Never stable.

[Roberto Di Cosmo (rdicosmo)]

Thanks Sebastian Thiel (@Byron) for the quick positive answer, I will file the PR following the guidelines.
Actually, it is a human here, but for a first participation in a significant project, I thought that I had to be verbose in explaining the use case and options. Will be more concise in the PR, and thank you for this much needed work!