The slotmap turned out to be too small

[FoxLightning]

Current behavior 😯

gix is not working with repository ~2M files + LFS

Expected behavior 🤔

work as git

Git behavior

Steps to reproduce 🕹

git status
git log

[Byron]

Thanks for reporting!

This shouldn't be happening as it starts out the slotmap with num_files_to_open * 2, so there is always twice as much as it would need to map everything.

Could you list the files in .git/objects/pack/?

[FoxLightning]

/usr/bin/ls 00 0a 13 1c 24 2c 34 3d 45 4d 57 60 69 73 7e 88 90 98 a0 a9 b3 bd c6 cf db e5 ee f8 info 01 0b 14 1d 25 2d 35 3e 46 4e 58 61 6a 75 81 89 91 99 a1 aa b4 be c7 d0 dc e6 ef f9 pack 03 0c 15 1e 26 2e 36 3f 47 4f 59 62 6b 76 82 8a 92 9a a3 ab b5 bf c8 d2 dd e7 f0 fa 04 0d 16 1f 27 2f 37 40 48 50 5b 64 6c 77 83 8b 93 9b a4 ac b6 c0 c9 d4 de e9 f1 fb 05 0e 17 20 28 30 38 41 49 51 5c 65 6d 78 84 8c 94 9c a5 ae b7 c1 ca d6 e0 ea f2 fc 07 10 18 21 29 31 39 42 4a 52 5d 66 6e 79 85 8d 95 9d a6 af b8 c2 cb d7 e1 eb f4 fd 08 11 1a 22 2a 32 3a 43 4b 53 5e 67 6f 7a 86 8e 96 9e a7 b0 b9 c3 cc d8 e3 ec f5 fe 09 12 1b 23 2b 33 3c 44 4c 54 5f 68 70 7d 87 8f 97 9f a8 b2 bc c4 ce d9 e4 ed f7 ff

[FoxLightning]

its on windows

[FoxLightning]

190 items

[Byron]

Thanks, it really looks like this is a bug where it can't properly scale beyond 32 anymore.

[FoxLightning]

What about the progress of the task? How can I help you? Perhaps you need additional information?

[Byron]

This issue is top of my list, and still I didn't get to it - special times.
Indeed, I'd appreciate your help, thanks for asking.

gitoxide/gix/tests/gix/remote/fetch.rs

Lines 88 to 90 in dbf079f

	#[test]
	#[cfg(feature = "blocking-network-client")]
	fn fetch_more_packs_than_can_be_handled() -> gix_testtools::Result {

The test above could be adapted to create more than 32 packs in the local repository by fetching repeatedly. Then a default repository can be opened there, with the assumption that this reproduces the issue you are seeing after accessing the head_id().

If this doesn't reproduce, and as an immediate fix, you could also try to set the default slotmap size to 96, which would allow 96 indices and packs to be opened at the same time (i.e. nearly 200 files) - that way it should just work without truly fixing the bug. If it still doesn't work though, this would certainly also tell us something about the nature of the issue.

[Byron]

Apologies for the long wait.

However, I did change the test in the linked PR to the point where it should be able to reproduce the issue.

The hypothesis was that if there are many packs, it's unable properly configure its slotmap to be able to load all packs if needed. However, int he screenshot it seemingly failed to do that.

A test was created to create a repository that would be above a set minimal amount of packs it could read, to validate it dynamically adjusts that value. And the test works as expected.

Further, I have a repository with 145 packs, and gix status works fine there as well.

Could you retry this with the latest version of gix (CLI)?

My assumption would be that it is still reproducing for you, but ideally we could work on debugging this together to figure out what the problem is. In the screenshot, it clearly didn't see all packs and initialized the slotmap with the minimum of 32 slots, even though 95 packs (the slotmap should be 105 slots large then).
It's notable that the algorithm to figure out how many slots are needed is filtering the directory (.git/objects/packs) contents quite a bit, and that would be another reason that it can't see files that are seemingly there.

gitoxide/gix-odb/src/store_impls/dynamic/load_index.rs

Lines 457 to 460 in 52ff68b

	let indices = entries
	.filter_map(Result::ok)
	.filter_map(|e| e.metadata().map(|md| (e.path(), md)).ok())
	.filter(|(_, md)| md.file_type().is_file())

Thanks in advance for your help.