Git log parsing and export

[jannismilz]

I'm wondering a couple of things about git in general and also if gitoxide is superior:

• How fast can you implement a git log to extract all the possible data of a commit?
• Is it possible to more efficiently store this data and make it queriable?
• Does gitoxide have a git log format option and if yes, any better tham normal git?

I'm asking for insights about performance on a huge scale (milliona of repos) for research.

I'm curous about all your thoughts, ideas and any feedback.

[Byron]

I don't think one can ever say that one is superior over the other in a blanket statement, but I am sure gitoxide has some advantages that can be helpful.

How fast can you implement a git log to extract all the possible data of a commit?

You can try it for yourself by running ein t hours, which installs with cargo install gitoxide. It's as fast as I could make it, doing exactly what you mentioned.

Is it possible to more efficiently store this data and make it queriable?

Absolutely. Read the data and store it in sqlite, like ein t query does. query doesn't store commit information, but it stores the expensive information that many tools need.

Does gitoxide have a git log format option and if yes, any better tham normal git?

No, git log is a lot of work on top of a commit traversal whose performance and capability gix does not match.

I'm asking for insights about performance on a huge scale (milliona of repos) for research.

I think gitoxide is very well suited for this and Nosey Parker already shows how to do that. It decodes all Git objects though, which is what you want to be as fast as possible.

Oh, and since it's millions of repos, there is gix corpus to run algorithms on a corpus of repositories in parallel and store the results. All this is just as demonstration on what can be done, and you can mix and match.

[jannismilz]

You can try it for yourself by running ein t hours, which installs with cargo install gitoxide. It's as fast as I could make it, doing exactly what you mentioned.

I just tried it out. It's definitely super fast but I believe that's not "all the possible data" right? Or at least not displayed.

I'm asking about every data which is: commit hash, author name/email/date, committer name/email/date, message and if possible the additional file and line changes information (which increase the time for the normal git log by around 10x according to my tests).

Absolutely. Read the data and store it in sqlite, like ein t query does. query doesn't store commit information, but it stores the expensive information that many tools need.

SQLite is good for a couple of repos but for storing a couple billions of commit objects (which are 3.5TB uncompressed) it starts to become a problem to then also run expensive analytical queries on top of it. That's why I used ClickHouse but I more wondered if there would be a more efficient file storage format than the git objects but as I did some more research it already seems that that's already a pretty good architecture.

What's more interesting and what I further did research on is that you can make multiple repos use the same .git folder location which results in e.g. forks not using more disk since they can also reuse the git objects with the same hash. But that's just a sidenote.

No, git log is a lot of work on top of a commit traversal whose performance and capability gix does not match.

What work is done on top of the commit traversal? I would just love a better and more stable format to export git commits since there can be any data in these fields (already found commits with pdf binary embedded..)

I think gitoxide is very well suited for this and Nosey Parker already shows how to do that. It decodes all Git objects though, which is what you want to be as fast as possible.

Super interesting, thank you for the link. What does decoding git objects mean? Like decompress them since some part is gzipped or something (I read a quick note on that somewhere)

Oh, and since it's millions of repos, there is gix corpus to run algorithms on a corpus of repositories in parallel and store the results. All this is just as demonstration on what can be done, and you can mix and match.

Thank you for the hint, I'm going to look into it. I just found out that gix corpus doesn't work corerctly on bare repos? Even tho that's what I'm doing to save disk since I don't need the files, I only need the git commits and with them I could recreate the files.

[Byron]

I think I shared everything I could but believe you will figure out everything else from the pointers. All the best!