Skip to content

Commit 3a5416f

Browse files
authored
gj (#1)
* plumbing: format: pktline, Accept oversized pkt-lines up to 65524 bytes The canonical Git client successfully decodes sideband packets up to 65524 bytes in length (4-byte header + 65520-byte payload). The Git protocol documentation was updated in August 2016 to reduce the maximum payload size to 65516 bytes, however old implementations still exist in the wild emitting 65520-byte payloads. As there is no technical difficulty with accepting (not emitting) larger payload sizes, this change adjusts the limit check to allow successful decoding of packets up to 65524 bytes. This change increases compatibility with the current canonical Git implementation. Doc changes from August 2016: git/git@7841c48#diff-52695c8fe91b78b70cea44562ae28297L67 Current packet buffer size is still LARGE_PACKET_MAX (+1 null): https://github.com/git/git/blob/468165c1d8a442994a825f3684528361727cd8c0/sideband.c#L24 https://github.com/git/git/blob/468165c1d8a442994a825f3684528361727cd8c0/sideband.c#L36 LARGE_PACKET_MAX definition: https://github.com/git/git/blob/468165c1d8a442994a825f3684528361727cd8c0/pkt-line.h#L100 Signed-off-by: Joseph Vusich <[email protected]> * add PlainOpen variant to find .git in parent dirs This is the git tool's behavior that people are used to; if one runs a git command in a repository's subdirectory, git still works. Fixes src-d#765. Signed-off-by: Daniel Martí <[email protected]> * use bsd superset for conditional compilation Signed-off-by: wardn <[email protected]> * config: adds branches to config for tracking branches against remotes, updates clone to track when cloning a branch. Fixes src-d#313 Signed-off-by: Jeremy Chambers <[email protected]> * dotgit: ignore filenames that don't match a hash For both packfiles and object files. Issue: keybase/client#11366 Signed-off-by: Jeremy Stribling <[email protected]> * storage: dotgit, init fixtures in benchmark. Fixes src-d#770 fixtures is not initialized in BenchmarkRefMultipleTimes and caused panic. Signed-off-by: Javi Fontan <[email protected]> * git: remote, Add shallow commits instead of substituting. Fixes src-d#412 updateShallow substituted the previous shallow list with the one returned by the UploadPackResponse. If the repository had previous shallow commits these are deleted from the list. This change adds the new shallow hashes to the old ones. Signed-off-by: Javi Fontan <[email protected]> * dotgit: add test for bad file in pack directory Suggested by mcuadros. Issue: src-d#807 Signed-off-by: Jeremy Stribling <[email protected]> * Resolve full commit sha to plumbing hash Signed-off-by: antham <[email protected]> * storage: filesystem, close shallow file when read Signed-off-by: Máximo Cuadros <[email protected]> * git: worktree, Skip special git directory. Fixes src-d#814 Signed-off-by: kuba-- <[email protected]> * travis: dropping 1.8.x support due to golang.org/x/crypto/ssh requirement * Use remote name in fetch while clone Fixes src-d#827 Signed-off-by: Dustin Frisch <[email protected]> * Worktree: Provide ability to add excludes (src-d#825) Worktree: Provide ability to add excludes * Teach ResolveRevision how to look up annotated tags Signed-off-by: Mike Lundy <[email protected]> * git: remote, Do not iterate all references on update. The current code iterates all the references in the remote to check if they match the refspec. This is OK when the refspec is a wildcard but is a waste of time when they are not. A hash with references is generated for fast access before starting the update and used only when the refspec is not a wildcard. In a repository with 7800 references this meant 7800 * 7800 checks. With the current code it took 8m30s to update the references. With the new code it takes less than 0.5s. References are already extensively tested in remote_test.go. Signed-off-by: Javi Fontan <[email protected]> * idxfile: optimise allocations in readObjectNames This makes all the required Entry allocations in one go, instead of huge amounts of small individual allocations. Signed-off-by: David Symonds <[email protected]> * packfile: improve Index memory representation to be more compact Instead of using a map for offset indexing, use a sorted slice. Binary searching is fast, and a slice is much more compact. This has a negligible hit on speed, but has a significant impact on memory usage, especially for larger repos. benchmark old ns/op new ns/op delta BenchmarkIndexConstruction-12 15506506 14056098 -9.35% benchmark old allocs new allocs delta BenchmarkIndexConstruction-12 60764 60385 -0.62% benchmark old bytes new bytes delta BenchmarkIndexConstruction-12 4318145 3913169 -9.38% Signed-off-by: David Symonds <[email protected]> * config: modules, Ignore submodules with dotdot '..' path components. Fixes CVE-2018-11235 References: * https://blogs.msdn.microsoft.com/devops/2018/05/29/announcing-the-may-2018-git-security-vulnerability/ * https://security-tracker.debian.org/tracker/CVE-2018-11235 * git/git@0383bbb Signed-off-by: Joseph Vusich <[email protected]> * worktree: Don't allow .gitmodules to be a symlink. Fixes CVE-2018-11235 References: * https://blogs.msdn.microsoft.com/devops/2018/05/29/announcing-the-may-2018-git-security-vulnerability/ * https://security-tracker.debian.org/tracker/CVE-2018-11235 * git/git@10ecfa7 Signed-off-by: Joseph Vusich <[email protected]> * dotgit: Move package outside internal. Signed-off-by: Antonio Jesus Navarro Perez <[email protected]> * Remove println Signed-off-by: Antonio Jesus Navarro Perez <[email protected]> * plumbing: object, adds tree path cache to trees. Fixes src-d#793 The cache is used in Tree.FindEntry for faster path search. Signed-off-by: Javi Fontan <[email protected]> * plumbing: packfile, Don't push empty objects. Fixes src-d#840 Signed-off-by: kuba-- <[email protected]> * storage: filesystem, make ObjectStorage constructor public Signed-off-by: Miguel Molina <[email protected]> * plumbing/transport: http, Adds token authentication support [Fixes src-d#858] Signed-off-by: Eric Billingsley <[email protected]> * Fix documentation for Notes It previously said that it returned all references that are branches, but that's not true. Signed-off-by: Morgan Bazalgette <[email protected]> * packfile: optimise NewIndexFromIdxFile for a very common case Loading from an on-disk idxfile will usually already have the idxfile entries in order, so check that before wasting time on sorting. Signed-off-by: David Symonds <[email protected]> * Remote.Fetch: error on missing remote reference Signed-off-by: Máximo Cuadros <[email protected]> * storage/filesystem: avoid norwfs build flag norwfs build flag was used to work on filesystems that do not support neither opening a file in read/write mode or renaming a file (e.f. sivafs). This had two problems: - go-git could not be compiled to work properly both with regular filesystems and limited filesystems at the same time. - the norwfs trick was not available on Windows. This PR removes the norwfs build flag, as well as the windows conditional flag on the dotgit package. For the file open mode, we use the new billy capabilities, to check at runtime if the filesystem supports opening a file in read/write mode or not. For the renaming, we just try and fallback to alternative methods if billy.ErrNotSupported is returned. Signed-off-by: Santiago M. Mola <[email protected]> * utils: diff, skip useless rune->string conversion According to library documentation : https://github.com/sergi/go-diff/blob/master/diffmatchpatch/diff.go#L391 Signed-off-by: Marc Barussaud <[email protected]> * plumbing: add context to allow cancel on diff/patch computing Signed-off-by: Marc Barussaud <[email protected]> * worktree: add test for correct tree sorting (issue src-d#881) Signed-off-by: Mark Bartel <[email protected]> * worktree: sort the tree object. Fixes src-d#881 Signed-off-by: Mark Bartel <[email protected]> * worktree: address PR comments: sort imports appropriately Signed-off-by: Mark Bartel <[email protected]> * plumbing: object, expose ErrEntryNotFound in FindEntry. Fixes src-d#883 FindEntry will return ErrDirNotFound if the directory doesn't exist. But it doesn't return a public error if the entry itself is missing. This exposes the internal error ErrEntryNotFound, so users can programmatically check for this condition. Signed-off-by: James Ravn <[email protected]> * plumbing/transport/internal: common, add support of Gogs for ErrRepositoryNotFound, avoiding to get an 'unknown error: '. Add some tests for existing supported services (github, gitlab, etc...) too. Signed-off-by: Jerome Doucet <[email protected]> * plumbing/object: fix pgp signature encoder/decoder The way of reading pgp signatures was searching for pgp begin line in the header. This caused problems when this string appeared and was not part of the signature. For example if it appears in the message as an example or is part of the author name the decoder starts treating it as a signature. In this state the code was not able to notice then the header ended so it entered in an infinite loop searching for pgp end string. Now it uses the same method as original git. Searches for gpgsig section in header and starts getting all lines until the next part. In encoder the string used to add signatures was incorrect. It is now changed to the proper "gpgsig" string instead of "pgpsig". Signed-off-by: Javi Fontan <[email protected]> * plumbing/format/idxfile: add new Index and MemoryIndex Signed-off-by: Miguel Molina <[email protected]> * plumbing/packfile: add new packfile parser Signed-off-by: Javi Fontan <[email protected]> * plumbing/packfile: disable lookup by offset In one case it disables the cache and the other disables lookup when the scanner is not seekable. Could be added back later. Signed-off-by: Javi Fontan <[email protected]> * plumbing: idxfile, add idxfile.Writer with Observer interface It's still not complete: * 64 bit offsets * IdxChecksum Signed-off-by: Javi Fontan <[email protected]> * plumbing/idxfile: use Entry to hold object data Signed-off-by: Javi Fontan <[email protected]> * plumbing/idxfile: support offset64 generating indexes Signed-off-by: Javi Fontan <[email protected]> * plumbing/packfile: preallocate memory in PatchDelta Signed-off-by: Javi Fontan <[email protected]> * plumbing/idxfile: fix bug searching in MemoryIndex Signed-off-by: Javi Fontan <[email protected]> * plumbing/idxfile: add offset/hash mapping to index This functionality may be moved elsewhere in the future but is needed now to fit filesystem.ObjectStorage and the new index. Signed-off-by: Javi Fontan <[email protected]> * plumbing/idxfile: index is created only once and retrieved with Index Index is also automatically generated when OnFooter is called. Signed-off-by: Javi Fontan <[email protected]> * plumbing, storage: integrate new index Now dotgit.PackWriter uses the new packfile.Parser and index. Signed-off-by: Javi Fontan <[email protected]> * plumbing: packfile, new Packfile representation Signed-off-by: Miguel Molina <[email protected]> * plumbing, packfile: delete index_test as is no longer used Signed-off-by: Javi Fontan <[email protected]> * plumbing: fix two errors in idxfile and packfile decoder Signed-off-by: Javi Fontan <[email protected]> * storage/filesystem: add back IndexStorage Signed-off-by: Javi Fontan <[email protected]> * plumbing: packfile, lazy object reads with DiskObjects Signed-off-by: Miguel Molina <[email protected]> * plumbing/idxfile: test FindHash and writer with 64 bit offsets Signed-off-by: Javi Fontan <[email protected]> * storage/filesystem: remove duplicated IndexStorage Signed-off-by: Javi Fontan <[email protected]> * plumbing/packfile: add index generation to decoder Signed-off-by: Javi Fontan <[email protected]> * Fix wrong godoc on Tags() method. Reword Tags() method documentation. Point to TagObjects() method to get all the tags on a repository. Signed-off-by: Antonio Jesus Navarro Perez <[email protected]> * plumbing: packfile, fix package tests Signed-off-by: Miguel Molina <[email protected]> * *: use parser to populate non writable storages and bug fixes Signed-off-by: Miguel Molina <[email protected]> * Fixed cloning of a single tag Relates to src-d#870 Signed-off-by: Fedor Korotkov <[email protected]> * plumbing: packfile, allow non-seekable sources on Parser Signed-off-by: Miguel Molina <[email protected]> * plumbing: packfile, add Parse benchmark Signed-off-by: Miguel Molina <[email protected]> * plumbing: packfile, read object content only once Signed-off-by: Miguel Molina <[email protected]> * storage: filesystem, benchmark PackfileIter Signed-off-by: Miguel Molina <[email protected]> * plumbing: packfile, rename DiskObject to FSObject Signed-off-by: Miguel Molina <[email protected]> * storage: filesystem, close Packfile after iterating objects Signed-off-by: Miguel Molina <[email protected]> * storage: filesystem, add PackfileIter benchmark reading object content Signed-off-by: Miguel Molina <[email protected]> * plumbing: packfile, open and close packfile on FSObject reads Signed-off-by: Miguel Molina <[email protected]> * git: add benchmark for iterating repository objects Signed-off-by: Miguel Molina <[email protected]> * plumbing: idxfile, Crc32 to CRC32 and return ok from findHashIndex Signed-off-by: Miguel Molina <[email protected]> * plumbing: add buffer cache and use it in packfile parser It uses less memory and is faster as slices don't have to be converted from/to MemoryObject and they are indexed by offset. Signed-off-by: Javi Fontan <[email protected]> * plumbing/pacfile: tidy up objectInfo struct * a new hasher is created when needed * delete unused fields * base content is no longer kept in memory Signed-off-by: Javi Fontan <[email protected]> * plumbing/packfile: do not compute sha1 for already undeltified objects Signed-off-by: Javi Fontan <[email protected]> * added hook support Signed-off-by: noxora <[email protected]> trying a possible fix to the delete test Signed-off-by: noxora <[email protected]> still trying to fix this test Signed-off-by: noxora <[email protected]> fixes did not work, seems to be a windows env problem Signed-off-by: noxora <[email protected]> * plumbing: object, Don't add new line at end of commit signature The way that commit signatures were being written out was causing an extra newline to be written at the end of the commit when the message encoding was already taking care of this. Ultimately, this results in a corrupt object, rendering the object unverifiable with the signature in the commit. Signed-off-by: Chris Marchesi <[email protected]> * git: Add ability to PGP sign commits This adds the ability to sign commits by adding the SignKey field to CommitOptions. If present, the commit will be signed during the WorkTree.Commit call. The supplied SignKey must already be decrypted by the caller. Signed-off-by: Chris Marchesi <[email protected]> * git: Remove use of strings.Builder This was added in Go 1.10 and is not supported on Go 1.9. Switched to bytes.Buffer to ensure compatibility. Signed-off-by: Chris Marchesi <[email protected]> * git: Remove old hash validation code This will not work for a signed commit as with the GPG signature being a part of the commit, the hash is now non-deterministic. Verification of the commit is done through the validation of the signature. Signed-off-by: Chris Marchesi <[email protected]> * git: Add extra test for testing bad key error case I'm hoping this helps get codecov to a tolerable delta. :) Signed-off-by: Chris Marchesi <[email protected]> * dotgit: fix object delete test Signed-off-by: Santiago M. Mola <[email protected]> * object: fix panic when reading object header When the first line of the pgp signature is an empty line or some header is malformed it crashes as there's no data for the header element. For example, if author name is "\n". Signed-off-by: Javi Fontan <[email protected]> * Fixed an edge case for .gitignore Fixes src-d#923 Signed-off-by: Fedor Korotkov <[email protected]> * plumbing/idxfile: object iterators returns entries in offset order In the latest change the order was changed from offset order in packfiles to hash order. This makes reading all the objects not as efficient as before. It also created problems when the previous order was expected. Also added EntriesByOffset to indexes. Signed-off-by: Javi Fontan <[email protected]> * git: Add tagging support This adds a few methods: * CreateTag, which can be used to create both lightweight and annotated tags with a supplied TagObjectOptions struct. PGP signing is possible as well. * Tag, to fetch a single tag ref. As opposed to Tags or TagObjects, this will also fetch the tag object if it exists and return it along with the output. Lightweight tags just return the object as nil. * DeleteTag, to delete a tag. This simply deletes the ref. The object is left orphaned to be GCed later. I'm not 100% sure if DeleteTag is the correct behavior - looking for details on exactly *what* happens to a tag object if you delete the ref and not the tag were sparse, and groking the Git source did not really produce much insight to the untrained eye. This may be something that comes up in review. If deletion of the object is necessary, the in-memory storer may require some updates to allow DeleteLooseObject to be supported. Signed-off-by: Chris Marchesi <[email protected]> * plumbing: object, correct tag PGP encoding As with the update in ec3d2a8, tag encoding needed to be corrected to ensure extra newlines were not being added in during tag object encoding, so that it did not corrupt the object for verification. Signed-off-by: Chris Marchesi <[email protected]> * plumbing: object, don't add extra newline on PGP signature Tag encoding/decoding seems to be a lot more sensitive to requiring the exact expected format in the object, which generally includes messages canonicalized so that they have a newline on the end (even if they didn't before). As such, the message should be written with the newline (no need for an extra), and the PGP signature right after that, which will be newline split already, so there's no need to split it again. All of this means it's very important for the caller to send the message in the correct format - which I'm correcting in the next commit. Signed-off-by: Chris Marchesi <[email protected]> * git: Canonicalize incoming annotated tag messages Tag messages are highly sensitive to being in the expected format, especially when encoding/decoding for PGP verification. As such, we do a simple trimming of whitespace on the incoming message and add a newline on the end, to ensure there are no surprises here. Signed-off-by: Chris Marchesi <[email protected]> * git: Replace test signing key with one with longer expiry The old one was created with defaults, which would have caused CI failures in 2 years. The new one is valid for 10 years: > gpg --list-secret-keys /root/.gnupg/pubring.kbx ------------------------ sec rsa4096 2018-08-22 [SC] [expires: 2028-08-19] 93A17FF01E54328546087C8E029395402EFCCD53 uid [ unknown] foo bar <[email protected]> Signed-off-by: Chris Marchesi <[email protected]> * plumbing, storage: add bases to the common cache After clone only resolved deltas were added to the cache. This caused slowdowns in small repositories where most objects can be held in cache. It also makes packfiles reuse delta cache from the store. Previously it created a new delta cache each time a packfile object was created. This also slowed down a bit accessing objects and had an impact on memory consumption when bases are added to the cache. Signed-off-by: Javi Fontan <[email protected]> * git: Discern tag target type from supplied hash I figured there was a way to do this without having to have TagObjectOptions supply this in - there is. Added support for this in and removed the object type from TagObjectOptions. Signed-off-by: Chris Marchesi <[email protected]> * git: Don't return tag object with Tag, adjust docs for Tag and Tags I've mainly noticed that in using the current Tag function, that there were lots of times that I was ignoring the ref or the object, depending on what I needed. This was evident in the tests as well. As such, I think it just makes more sense for a singular tag fetcher to return just a ref, through which the caller may grab the annotation if they need it, and if it exists. Also, contrary to the docs on Tags, all tags have a ref, even if they are annotated. The difference between a lightweight tag and an annotated tag is the presence of the tag object, which the ref will point to if the tag is annotated. As such I've adjusted the docs with an example as to how one can get the annotation for a tag ref through the iterator. Source: https://git-scm.com/book/en/v2/Git-Internals-Git-References, tags section. Signed-off-by: Chris Marchesi <[email protected]> * storage/dotgit: search for incoming dir only once Search for incoming object directory was done once each time objects were accessed. This means a ReadDir of the objects path that is expensive. Now incoming directory is searched the first time an object is accessed and its name kept in DotGit to be reused. Signed-off-by: Javi Fontan <[email protected]> * storage/dotgit: use HasPrefix instead of Split Also reformatted function comment and fixed some typos. Signed-off-by: Javi Fontan <[email protected]> * Remove empty dirs when cleaning with Dir opt. Signed-off-by: kuba-- <[email protected]> * Add Status.IsUntracked function Signed-off-by: kuba-- <[email protected]> * plumbing: object: Clamp object timestamps before unix epoch to unix epoch Signed-off-by: Taru Karttunen <[email protected]> * config: add commentChar to core config struct Signed-off-by: Zaq? Wiedmann <[email protected]> * git: add Static option to PlainOpen Also adds Static configuration to Storage and DotGit. This option means that the git repository is not expected to be modified while open and enables some optimizations. Each time a file is accessed the storer tries to open an object file for the requested hash. When this is done for a lot of objects it is expensive. With Static option a list of object files is generated the first time an object is accessed and used to check if exists instead of using system calls. A similar optimization is done for packfiles. Signed-off-by: Javi Fontan <[email protected]> * git, storer: use a common storer.Options for storer and PlainOpen Signed-off-by: Javi Fontan <[email protected]> * dotgit: fix typo in comment Signed-off-by: Javi Fontan <[email protected]> * git: do not expose storage options in PlainOpen Signed-off-by: Javi Fontan <[email protected]> * plumbing/storer: rename Static option to ExclusiveAccess Signed-off-by: Javi Fontan <[email protected]> * storage/filesystem: make Storage options private Signed-off-by: Javi Fontan <[email protected]> * storage/filesystem: move Options to filesytem and dotgit Signed-off-by: Javi Fontan <[email protected]> * storage/dotgit: add ExclusiveAccess tests in dotgit This functionality was already tested in storage/filesystem. The coverage tool only takes into account files from the same package of the test. Signed-off-by: Javi Fontan <[email protected]> * storage/dotgit: add KeepDescriptors option This option maintains packfile file descriptors opened after reading objects from them. It improves performance as it does not have to be opening packfiles each time an object is needed. Also adds Close to EncodedObjectStorer to close all the files manualy. Signed-off-by: Javi Fontan <[email protected]> * plumbing/storer: do not expose Close in EncodedObjectStorer interface Signed-off-by: Javi Fontan <[email protected]> * storage/filesystem: add KeepDescriptors test Also delete Close from MockObjectStorage and memory storer. Signed-off-by: Javi Fontan <[email protected]> * storage/filesystem: compare files using offset in test Using equals to compare files it uses diff to do so. This can potentially consume lots of ram. Changed the comparison to use file offsets. If the descriptor is reused the offset is maintained. Signed-off-by: Javi Fontan <[email protected]> * plumbing/transport: ssh check if list of known_hosts files is empty Signed-off-by: kuba-- <[email protected]> * Fix fatal corrupt patch in unified diff format Signed-off-by: Antonio Jesus Navarro Perez <[email protected]> * Expose Storage cache. Signed-off-by: kuba-- <[email protected]> * git: s/fetch/returns/ on Tag function doc This is to avoid any ambiguity with the act of "fetching" in git in general. Signed-off-by: Chris Marchesi <[email protected]> * git: Add Tag objects to the list of supported objects for walking This is necessary to support pruning on Tag objects. Signed-off-by: Chris Marchesi <[email protected]> * git: Don't touch tag objects orphaned by tag deletion Deleting a tag ref for an annotated tag in normal git behavior does not delete the tag object right away. This is handled by the normal GC process. Signed-off-by: Chris Marchesi <[email protected]> * git: Add some tests for annotated tag deletion Added a couple of tests for annotated tag deletion: * The first one is a general test and should work regardless of the fixture used - the tag object could possibly be packed, so we do a prune *and* a repack operation before testing to see if the object was GCed correctly. * The second one actually creates the tag to be deleted, so that the tag object gets created as a loose, unpacked object. This is so we can effectively test that purning unpacked objects is now working 100% correctly (this was failing before because tag objects were not supported for walking). Signed-off-by: Chris Marchesi <[email protected]> * git: s/TagObjectOptions/CreateTagOptions/ Just renaming the TagObjectOptions type to CreateTagOptions so that it's consistent with the other option types. Signed-off-by: Chris Marchesi <[email protected]> * repository: fix test for new Storage constructor Signed-off-by: Máximo Cuadros <[email protected]> * *: go modules support Signed-off-by: Máximo Cuadros <[email protected]> * travis: drop go1.9 add go1.11 * Fix potential LRU cache size issue. Signed-off-by: kuba-- <[email protected]> * Remove empty space to trigger windows build. Signed-off-by: kuba-- <[email protected]> * storage/filesystem: keep packs open in PackfileIter PackfileIter was not taking into account the option KeepDescriptors and was always closing the file. This caused "file already closed" errors when iterating packfiles in with KeepDescriptors active. Signed-off-by: Javi Fontan <[email protected]> * storage/filesystem: add more doc to NewPackfileIter Signed-off-by: Javi Fontan <[email protected]> * all: remove extra 's' in "mismatch" Signed-off-by: Jongmin Kim <[email protected]> * test: improve test for urlencoded user:pass Signed-off-by: Santiago M. Mola <[email protected]> * use time.IsZero in Prune Signed-off-by: u5surf <[email protected]> * Add test for Windows local paths. Signed-off-by: Filip Navara <[email protected]> * git: Fix Status.IsClean() documentation The documentation of the IsClean Method contained a negation, so it was describing the opposite of its actual behavior. Fixes src-d#838 Signed-off-by: David Url <[email protected]> * Plumbing: object, Add support for Log with filenames. Fixes src-d#826 (src-d#979) plumbing: object, Add support for Log with filenames. Fixes src-d#826 * object: get object size without reading whole object Signed-off-by: Jeremy Stribling <[email protected]> * tree: add a Size() method for getting plaintext size Without reading the entire object into memory. Signed-off-by: Jeremy Stribling <[email protected]> * filesystem: add a new test for EncodedObjectSize Suggested by taruti. Signed-off-by: Jeremy Stribling <[email protected]> * repository: allow open non-bare repositories as bare Signed-off-by: Máximo Cuadros <[email protected]> * use remote name in fetch while clone, test Signed-off-by: Máximo Cuadros <[email protected]> * references: sort: compare author timestamps when commit timestamps are equal, test Signed-off-by: Máximo Cuadros <[email protected]> * teach ResolveRevision how to look up annotated tags, test Signed-off-by: Máximo Cuadros <[email protected]> * teach ResolveRevision how to look up annotated tags, test Signed-off-by: Máximo Cuadros <[email protected]> * packfile: add comment on GetSizeByOffset Suggested by mcuadros. Issue: src-d#982 Signed-off-by: Jeremy Stribling <[email protected]> * blame: fix edge case with missing \n in content length causing mismatched length error Signed-off-by: Máximo Cuadros <[email protected]> * repository: improve CheckoutOption.Hash doc Signed-off-by: Máximo Cuadros <[email protected]> * remote: use reference deltas on push when the remote server does not support offset deltas Signed-off-by: Benjamin Ash <[email protected]> * Fixed a typo. (src-d#989) README: Fixed a typo. * Enables building on openbsd, dragonfly bsd and solaris Signed-off-by: Yuce Tekol <[email protected]> * plumbing/format/packfile: Fix broken "thin" packfile support. Fixes src-d#991 Signed-off-by: Javier Peletier <[email protected]> * plumbing: ReferenceName constructors Signed-off-by: Máximo Cuadros <[email protected]> * examples & documentation: PlainClone with Basic Authentication (Password & Access Token) (src-d#990) examples: PlainClone with Basic Authentication (Password & Access Token) * add StackOverflow to support channels Since we are not redirecting users to StackOverflow for support questions, it makes sense to add it to the official support channels. Signed-off-by: Santiago M. Mola <[email protected]> * plumbing: transport/http, Add missing host/port on redirect. Fixes src-d#820 Signed-off-by: Dave Henderson <[email protected]> * Fix spelling and grammar in docs and example Signed-off-by: Lukasz Kokot <[email protected]> * update gcfg dependency to v1.4.0 Signed-off-by: Dave Henderson <[email protected]> * repository: added cleanup for the PlainCloneContext() Signed-off-by: Bartek Jaroszewski <[email protected]> * improve cleanup implementation, add more tests Signed-off-by: Santiago M. Mola <[email protected]> * Update LICENSE Signed-off-by: Máximo Cuadros <[email protected]> * http: improve TokenAuth documentation Users are often confused with TokenAuth, since it might look that it should be used with GitHub's OAuth tokens. But that is not the case. TokenAuth implements HTTP bearer authentication. Most git servers will use HTTP basic authentication (user+passwords) even for OAuth tokens. Signed-off-by: Santiago M. Mola <[email protected]> * plumbing: ssh, Fix flaky test TestAdvertisedReferencesNotExists. Fixes src-d#969 Signed-off-by: Colton McCurdy <[email protected]> * repository: Fix RefSpec for a single tag. Fixes src-d#960 Signed-off-by: Fedor Korotkov <[email protected]> * storage/filesystem: Added reindex method to reindex packfiles Signed-off-by: Javier Peletier <[email protected]> * plumbing/format/packfile: Added thin pack test Signed-off-by: Javier Peletier <[email protected]> * Remove unused method (src-d#1022) Signed-off-by: Antonio Jesus Navarro Perez <[email protected]> * plumbing: format/index: support for EOIE extension, by default on git v2.2.0 Signed-off-by: Máximo Cuadros <[email protected]> * repository: fix plain clone error handling regression PR src-d#1008 introduced a regression by changing the errors returned by PlainClone when a repository did not exist. This change goes back to returned errors as they were in v4.7.0. Fixes src-d#1027 Signed-off-by: Santiago M. Mola <[email protected]> * plumbing: format/packfile, performance optimizations for reading large commit histories (src-d#963) Signed-off-by: Filip Navara <[email protected]>
1 parent 01c428c commit 3a5416f

File tree

170 files changed

+10167
-2209
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

170 files changed

+10167
-2209
lines changed

.gitignore

+3
Original file line numberDiff line numberDiff line change
@@ -1 +1,4 @@
11
coverage.out
2+
*~
3+
coverage.txt
4+
profile.out

.travis.yml

+2-2
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
language: go
22

33
go:
4-
- 1.8.x
5-
- 1.9.x
4+
- "1.10"
5+
- "1.11"
66

77
go_import_path: gopkg.in/src-d/go-git.v4
88

CONTRIBUTING.md

+2-1
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,8 @@ This can be done easily using the [`-s`](https://github.com/git/git/blob/b2c150d
2121

2222
The official support channels, for both users and contributors, are:
2323

24-
- GitHub [issues](https://github.com/src-d/go-git/issues)*
24+
- [StackOverflow go-git tag](https://stackoverflow.com/questions/tagged/go-git) for user questions.
25+
- GitHub [Issues](https://github.com/src-d/go-git/issues)* for bug reports and feature requests.
2526
- Slack: #go-git room in the [source{d} Slack](https://join.slack.com/t/sourced-community/shared_invite/enQtMjc4Njk5MzEyNzM2LTFjNzY4NjEwZGEwMzRiNTM4MzRlMzQ4MmIzZjkwZmZlM2NjODUxZmJjNDI1OTcxNDAyMmZlNmFjODZlNTg0YWM)
2627

2728
*Before opening a new issue or submitting a new pull request, it's helpful to

LICENSE

+2-2
Original file line numberDiff line numberDiff line change
@@ -186,7 +186,7 @@
186186
same "printed page" as the copyright notice for easier
187187
identification within third-party archives.
188188

189-
Copyright 2017 Sourced Technologies, S.L.
189+
Copyright 2018 Sourced Technologies, S.L.
190190

191191
Licensed under the Apache License, Version 2.0 (the "License");
192192
you may not use this file except in compliance with the License.
@@ -198,4 +198,4 @@
198198
distributed under the License is distributed on an "AS IS" BASIS,
199199
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
200200
See the License for the specific language governing permissions and
201-
limitations under the License.
201+
limitations under the License.

README.md

+7-7
Original file line numberDiff line numberDiff line change
@@ -3,16 +3,16 @@
33

44
*go-git* is a highly extensible git implementation library written in **pure Go**.
55

6-
It can be used to manipulate git repositories at low level *(plumbing)* or high level *(porcelain)*, through an idiomatic Go API. It also supports several type of storage, such as in-memory filesystems, or custom implementations thanks to the [`Storer`](https://godoc.org/gopkg.in/src-d/go-git.v4/plumbing/storer) interface.
6+
It can be used to manipulate git repositories at low level *(plumbing)* or high level *(porcelain)*, through an idiomatic Go API. It also supports several types of storage, such as in-memory filesystems, or custom implementations thanks to the [`Storer`](https://godoc.org/gopkg.in/src-d/go-git.v4/plumbing/storer) interface.
77

8-
It's being actively develop since 2015 and is being use extensively by [source{d}](https://sourced.tech/) and [Keybase](https://keybase.io/blog/encrypted-git-for-everyone), and by many other libraries and tools.
8+
It's being actively developed since 2015 and is being used extensively by [source{d}](https://sourced.tech/) and [Keybase](https://keybase.io/blog/encrypted-git-for-everyone), and by many other libraries and tools.
99

1010
Comparison with git
1111
-------------------
1212

1313
*go-git* aims to be fully compatible with [git](https://github.com/git/git), all the *porcelain* operations are implemented to work exactly as *git* does.
1414

15-
*git* is a humongous project with years of development by thousands of contributors, making it challenging for *go-git* implement all the features. You can find a comparison of *go-git* vs *git* in the [compatibility documentation](COMPATIBILITY.md).
15+
*git* is a humongous project with years of development by thousands of contributors, making it challenging for *go-git* to implement all the features. You can find a comparison of *go-git* vs *git* in the [compatibility documentation](COMPATIBILITY.md).
1616

1717

1818
Installation
@@ -24,12 +24,12 @@ The recommended way to install *go-git* is:
2424
go get -u gopkg.in/src-d/go-git.v4/...
2525
```
2626

27-
> We use [gopkg.in](http://labix.org/gopkg.in) for having a versioned API, this means that when `go get` clones the package, is the latest tag matching `v4.*` cloned and not the master branch.
27+
> We use [gopkg.in](http://labix.org/gopkg.in) to version the API, this means that when `go get` clones the package, it's the latest tag matching `v4.*` that is cloned and not the master branch.
2828
2929
Examples
3030
--------
3131

32-
> Please note that the functions `CheckIfError` and `Info` used in the examples are from the [examples package](https://github.com/src-d/go-git/blob/master/_examples/common.go#L17) just to be used in the examples.
32+
> Please note that the `CheckIfError` and `Info` functions used in the examples are from the [examples package](https://github.com/src-d/go-git/blob/master/_examples/common.go#L17) just to be used in the examples.
3333
3434

3535
### Basic example
@@ -71,7 +71,7 @@ r, err := git.Clone(memory.NewStorage(), nil, &git.CloneOptions{
7171

7272
CheckIfError(err)
7373

74-
// Gets the HEAD history from HEAD, just like does:
74+
// Gets the HEAD history from HEAD, just like this command:
7575
Info("git log")
7676

7777
// ... retrieves the branch pointed by HEAD
@@ -110,7 +110,7 @@ Date: Fri Nov 11 13:23:22 2016 +0100
110110
...
111111
```
112112

113-
You can find this [example](_examples/log/main.go) and many others at the [examples](_examples) folder
113+
You can find this [example](_examples/log/main.go) and many others in the [examples](_examples) folder.
114114

115115
Contribute
116116
----------

_examples/README.md

+4
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,10 @@ Here you can find a list of annotated _go-git_ examples:
66
- [showcase](showcase/main.go) - A small showcase of the capabilities of _go-git_
77
- [open](open/main.go) - Opening a existing repository cloned by _git_
88
- [clone](clone/main.go) - Cloning a repository
9+
- [username and password](clone/auth/basic/username_password/main.go) - Cloning a repository
10+
using a username and password
11+
- [personal access token](clone/auth/basic/access_token/main.go) - Cloning
12+
a repository using a GitHub personal access token
913
- [commit](commit/main.go) - Commit changes to the current branch to an existent repository
1014
- [push](push/main.go) - Push repository to default remote (origin)
1115
- [pull](pull/main.go) - Pull changes from a remote repository

_examples/branch/main.go

+1-1
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ func main() {
2828

2929
// Create a new plumbing.HashReference object with the name of the branch
3030
// and the hash from the HEAD. The reference name should be a full reference
31-
// name and now a abbreviated one, as is used on the git cli.
31+
// name and not an abbreviated one, as is used on the git cli.
3232
//
3333
// For tags we should use `refs/tags/%s` instead of `refs/heads/%s` used
3434
// for branches.

_examples/checkout/main.go

+2-2
Original file line numberDiff line numberDiff line change
@@ -38,8 +38,8 @@ func main() {
3838
})
3939
CheckIfError(err)
4040

41-
// ... retrieving the commit being pointed by HEAD, it's shows that the
42-
// repository is poiting to the giving commit in detached mode
41+
// ... retrieving the commit being pointed by HEAD, it shows that the
42+
// repository is pointing to the giving commit in detached mode
4343
Info("git show-ref --head HEAD")
4444
ref, err = r.Head()
4545
CheckIfError(err)
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
package main
2+
3+
import (
4+
"fmt"
5+
"os"
6+
7+
git "gopkg.in/src-d/go-git.v4"
8+
. "gopkg.in/src-d/go-git.v4/_examples"
9+
"gopkg.in/src-d/go-git.v4/plumbing/transport/http"
10+
)
11+
12+
func main() {
13+
CheckArgs("<url>", "<directory>", "<github_access_token>")
14+
url, directory, token := os.Args[1], os.Args[2], os.Args[3]
15+
16+
// Clone the given repository to the given directory
17+
Info("git clone %s %s", url, directory)
18+
19+
r, err := git.PlainClone(directory, false, &git.CloneOptions{
20+
// The intended use of a GitHub personal access token is in replace of your password
21+
// because access tokens can easily be revoked.
22+
// https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/
23+
Auth: &http.BasicAuth{
24+
Username: "abc123", // yes, this can be anything except an empty string
25+
Password: token,
26+
},
27+
URL: url,
28+
Progress: os.Stdout,
29+
})
30+
CheckIfError(err)
31+
32+
// ... retrieving the branch being pointed by HEAD
33+
ref, err := r.Head()
34+
CheckIfError(err)
35+
// ... retrieving the commit object
36+
commit, err := r.CommitObject(ref.Hash())
37+
CheckIfError(err)
38+
39+
fmt.Println(commit)
40+
}
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
package main
2+
3+
import (
4+
"fmt"
5+
"os"
6+
7+
git "gopkg.in/src-d/go-git.v4"
8+
. "gopkg.in/src-d/go-git.v4/_examples"
9+
"gopkg.in/src-d/go-git.v4/plumbing/transport/http"
10+
)
11+
12+
func main() {
13+
CheckArgs("<url>", "<directory>", "<github_username>", "<github_password>")
14+
url, directory, username, password := os.Args[1], os.Args[2], os.Args[3], os.Args[4]
15+
16+
// Clone the given repository to the given directory
17+
Info("git clone %s %s", url, directory)
18+
19+
r, err := git.PlainClone(directory, false, &git.CloneOptions{
20+
Auth: &http.BasicAuth{
21+
Username: username,
22+
Password: password,
23+
},
24+
URL: url,
25+
Progress: os.Stdout,
26+
})
27+
CheckIfError(err)
28+
29+
// ... retrieving the branch being pointed by HEAD
30+
ref, err := r.Head()
31+
CheckIfError(err)
32+
// ... retrieving the commit object
33+
commit, err := r.CommitObject(ref.Hash())
34+
CheckIfError(err)
35+
36+
fmt.Println(commit)
37+
}

_examples/commit/main.go

+3-3
Original file line numberDiff line numberDiff line change
@@ -12,13 +12,13 @@ import (
1212
"gopkg.in/src-d/go-git.v4/plumbing/object"
1313
)
1414

15-
// Basic example of how to commit changes to the current branch to an existent
15+
// Basic example of how to commit changes to the current branch to an existing
1616
// repository.
1717
func main() {
1818
CheckArgs("<directory>")
1919
directory := os.Args[1]
2020

21-
// Opens an already existent repository.
21+
// Opens an already existing repository.
2222
r, err := git.PlainOpen(directory)
2323
CheckIfError(err)
2424

@@ -44,7 +44,7 @@ func main() {
4444

4545
fmt.Println(status)
4646

47-
// Commits the current staging are to the repository, with the new file
47+
// Commits the current staging area to the repository, with the new file
4848
// just created. We should provide the object.Signature of Author of the
4949
// commit.
5050
Info("git commit -m \"example go-git commit\"")

_examples/log/main.go

+1-1
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ func main() {
2323
})
2424
CheckIfError(err)
2525

26-
// Gets the HEAD history from HEAD, just like does:
26+
// Gets the HEAD history from HEAD, just like this command:
2727
Info("git log")
2828

2929
// ... retrieves the branch pointed by HEAD

_examples/open/main.go

+1-1
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ func main() {
1414
CheckArgs("<path>")
1515
path := os.Args[1]
1616

17-
// We instance a new repository targeting the given path (the .git folder)
17+
// We instanciate a new repository targeting the given path (the .git folder)
1818
r, err := git.PlainOpen(path)
1919
CheckIfError(err)
2020

_examples/pull/main.go

+1-1
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ func main() {
1313
CheckArgs("<path>")
1414
path := os.Args[1]
1515

16-
// We instance a new repository targeting the given path (the .git folder)
16+
// We instance\iate a new repository targeting the given path (the .git folder)
1717
r, err := git.PlainOpen(path)
1818
CheckIfError(err)
1919

_examples/showcase/main.go

+1-1
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ import (
1616
// - Get the HEAD reference
1717
// - Using the HEAD reference, obtain the commit this reference is pointing to
1818
// - Print the commit content
19-
// - Using the commit, iterate all its files and print them
19+
// - Using the commit, iterate over all its files and print them
2020
// - Print all the commit history with commit messages, short hash and the
2121
// first line of the commit message
2222
func main() {

_examples/storage/README.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66

77

88
### and what this means ...
9-
*git* has as very well defined storage system, the `.git` directory, present on any repository. This is the place where `git` stores al the [`objects`](https://git-scm.com/book/en/v2/Git-Internals-Git-Objects), [`references`](https://git-scm.com/book/es/v2/Git-Internals-Git-References) and [`configuration`](https://git-scm.com/docs/git-config#_configuration_file). This information is stored in plain files.
9+
*git* has a very well defined storage system, the `.git` directory, present on any repository. This is the place where `git` stores all the [`objects`](https://git-scm.com/book/en/v2/Git-Internals-Git-Objects), [`references`](https://git-scm.com/book/es/v2/Git-Internals-Git-References) and [`configuration`](https://git-scm.com/docs/git-config#_configuration_file). This information is stored in plain files.
1010

1111
Our original **go-git** version was designed to work in memory, some time after we added support to read the `.git`, and now we have added support for fully customized [storages](https://godoc.org/gopkg.in/src-d/go-git.v4/storage#Storer).
1212

_examples/tag/main.go

+1-1
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ func main() {
1515
CheckArgs("<path>")
1616
path := os.Args[1]
1717

18-
// We instance a new repository targeting the given path (the .git folder)
18+
// We instanciate a new repository targeting the given path (the .git folder)
1919
r, err := git.PlainOpen(path)
2020
CheckIfError(err)
2121

blame.go

+24-6
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ import (
66
"fmt"
77
"strconv"
88
"strings"
9+
"time"
910
"unicode/utf8"
1011

1112
"gopkg.in/src-d/go-git.v4/plumbing"
@@ -106,24 +107,41 @@ type Line struct {
106107
Author string
107108
// Text is the original text of the line.
108109
Text string
110+
// Date is when the original text of the line was introduced
111+
Date time.Time
112+
// Hash is the commit hash that introduced the original line
113+
Hash plumbing.Hash
109114
}
110115

111-
func newLine(author, text string) *Line {
116+
func newLine(author, text string, date time.Time, hash plumbing.Hash) *Line {
112117
return &Line{
113118
Author: author,
114119
Text: text,
120+
Hash: hash,
121+
Date: date,
115122
}
116123
}
117124

118125
func newLines(contents []string, commits []*object.Commit) ([]*Line, error) {
119-
if len(contents) != len(commits) {
120-
return nil, errors.New("contents and commits have different length")
126+
lcontents := len(contents)
127+
lcommits := len(commits)
128+
129+
if lcontents != lcommits {
130+
if lcontents == lcommits-1 && contents[lcontents-1] != "\n" {
131+
contents = append(contents, "\n")
132+
} else {
133+
return nil, errors.New("contents and commits have different length")
134+
}
121135
}
122-
result := make([]*Line, 0, len(contents))
136+
137+
result := make([]*Line, 0, lcontents)
123138
for i := range contents {
124-
l := newLine(commits[i].Author.Email, contents[i])
125-
result = append(result, l)
139+
result = append(result, newLine(
140+
commits[i].Author.Email, contents[i],
141+
commits[i].Author.When, commits[i].Hash,
142+
))
126143
}
144+
127145
return result, nil
128146
}
129147

blame_test.go

+32
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@ package git
22

33
import (
44
"gopkg.in/src-d/go-git.v4/plumbing"
5+
"gopkg.in/src-d/go-git.v4/plumbing/object"
56

67
. "gopkg.in/check.v1"
78
"gopkg.in/src-d/go-git-fixtures.v3"
@@ -13,6 +14,31 @@ type BlameSuite struct {
1314

1415
var _ = Suite(&BlameSuite{})
1516

17+
func (s *BlameSuite) TestNewLines(c *C) {
18+
h := plumbing.NewHash("ce9f123d790717599aaeb76bc62510de437761be")
19+
lines, err := newLines([]string{"foo"}, []*object.Commit{{
20+
Hash: h,
21+
Message: "foo",
22+
}})
23+
24+
c.Assert(err, IsNil)
25+
c.Assert(lines, HasLen, 1)
26+
c.Assert(lines[0].Text, Equals, "foo")
27+
c.Assert(lines[0].Hash, Equals, h)
28+
}
29+
30+
func (s *BlameSuite) TestNewLinesWithNewLine(c *C) {
31+
lines, err := newLines([]string{"foo"}, []*object.Commit{
32+
{Message: "foo"},
33+
{Message: "bar"},
34+
})
35+
36+
c.Assert(err, IsNil)
37+
c.Assert(lines, HasLen, 2)
38+
c.Assert(lines[0].Text, Equals, "foo")
39+
c.Assert(lines[1].Text, Equals, "\n")
40+
}
41+
1642
type blameTest struct {
1743
repo string
1844
rev string
@@ -32,6 +58,10 @@ func (s *BlameSuite) TestBlame(c *C) {
3258
obt, err := Blame(commit, t.path)
3359
c.Assert(err, IsNil)
3460
c.Assert(obt, DeepEquals, exp)
61+
62+
for i, l := range obt.Lines {
63+
c.Assert(l.Hash.String(), Equals, t.blames[i])
64+
}
3565
}
3666
}
3767

@@ -53,6 +83,8 @@ func (s *BlameSuite) mockBlame(c *C, t blameTest, r *Repository) (blame *BlameRe
5383
l := &Line{
5484
Author: commit.Author.Email,
5585
Text: lines[i],
86+
Date: commit.Author.When,
87+
Hash: commit.Hash,
5688
}
5789
blamedLines = append(blamedLines, l)
5890
}

0 commit comments

Comments
 (0)