Skip to content
This repository was archived by the owner on Apr 15, 2020. It is now read-only.

Commit fe5057c

Browse files
toxeustoxeus
committed
implement gsv
gsv can now fetch a specified package and all its recursive dependencies. This is the most critical feature since recursive steps are always annoying to do manually. All other typical dependency management can be done using plain git commands. Still, the goal is to minimize the need for plain git commands over time and integrate full dependency management into gsv on a high and user friendly layer.
1 parent ecaf5d7 commit fe5057c

File tree

8 files changed

+390
-0
lines changed

8 files changed

+390
-0
lines changed

‎.gitmodules

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
[submodule "vendor/github.com/constabulary/gb"]
2+
path = vendor/github.com/constabulary/gb
3+
url = https://github.com/constabulary/gb
4+
[submodule "vendor/golang.org/x/tools"]
5+
path = vendor/golang.org/x/tools
6+
url = https://go.googlesource.com/tools

‎LICENSE

+1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
The MIT License (MIT)
22

3+
Copyright (c) 2016 Filip Gospodinov
34
Copyright (c) 2015 constabulary
45

56
Permission is hereby granted, free of charge, to any person obtaining a copy

‎README.md

+168
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,168 @@
1+
# gsv
2+
3+
## Table of Contents
4+
5+
* [Introduction](#introduction)
6+
* [Dependencies](#dependencies)
7+
* [Installation](#installation)
8+
* [Usage](#usage)
9+
* [FAQ](#faq)
10+
* [TODO](#todo)
11+
* [Credits](#credits)
12+
13+
## Introduction
14+
15+
gsv is the **G**o **S**ubmodule **V**endoring tool. It does native
16+
Go vendoring using Git submodules. This approach makes configuration
17+
files redundant and doesn't require additional tooling to build a
18+
gsv-vendored Go project because Git (which you have installed anyway)
19+
is used to track the revisions of your vendored dependencies.
20+
21+
Therefore, in order to fetch and install a Go package that has used
22+
`gsv` to vendor its dependencies a simple
23+
24+
```sh
25+
go get $PACKAGE_URL
26+
```
27+
28+
will do the job. Go 1.5.X needs some additional steps (see FAQ).
29+
30+
Compared to a copy-based vendoring approach gsv preserves the dependencies'
31+
histories and links them to the main project's history which facilitates
32+
the usage of the Git tool suite. Therefore, it's possible to go through
33+
each dependency's `git log` and analyze what changes have been done since
34+
the last update. Also, `git bisect` can be used to find a commit in a
35+
dependency's repository that for example caused the main project to break.
36+
And so on.
37+
38+
## Dependencies
39+
40+
[libgit2](https://github.com/libgit2/libgit2) needs to be installed.
41+
Packages exist for
42+
43+
* [Ubuntu](http://packages.ubuntu.com/source/wily/libgit2)
44+
* [Fedora](https://admin.fedoraproject.org/pkgdb/package/rpms/libgit2/)
45+
* [Arch Linux](https://www.archlinux.org/packages/extra/x86_64/libgit2/)
46+
* [Homebrew](https://github.com/Homebrew/homebrew/blob/master/Library/Formula/libgit2.rb)
47+
48+
If your distro does not package it then you need to install it from
49+
[source](https://github.com/libgit2/libgit2#building-libgit2---using-cmake).
50+
51+
## Installation
52+
53+
After installing the [dependencies](#dependencies) execute
54+
55+
```sh
56+
go get github.com/toxeus/gsv
57+
```
58+
59+
and make sure that `$GOPATH/bin` is in your `$PATH`.
60+
61+
## Usage
62+
63+
Let's assume we want to vendor `go-etcd` and its recursive dependencies
64+
in our project
65+
66+
```sh
67+
cd $GOPATH/$OUR_PROJECT
68+
gsv github.com/coreos/go-etcd/etcd
69+
git commit -m "vendor: added go-etcd and its dependencies"
70+
```
71+
72+
Done.
73+
74+
## FAQ
75+
76+
### I have a problem. What next?
77+
78+
Please open an [issue](https://github.com/toxeus/gsv/issues/new)
79+
and try to give reproducible examples ;)
80+
81+
### I want to contribute. What next?
82+
83+
If you want to contribute a bigger change then
84+
please open an [issue](https://github.com/toxeus/gsv/issues/new)
85+
such that we can discuss what the best way is to proceed.
86+
87+
If you want to fix something minor then feel free to open
88+
a pull request straightaway.
89+
90+
### How do I update all my dependencies?
91+
92+
As of now using
93+
94+
```sh
95+
git submodule foreach 'git fetch && git rebase master@{u}'
96+
```
97+
98+
will do the trick. Some day it'll be integrated into `gsv`.
99+
100+
### Do you know about git2go?
101+
102+
Yes! Last time I checked the support for submodules in
103+
[`git2go`](https://github.com/libgit2/git2go) was not sufficient
104+
for this project's requirements. That's why the Git code
105+
ended up being written in Cgo.
106+
107+
### How do I get and build a gsv project using Go 1.5?
108+
109+
Like this
110+
111+
```sh
112+
export GO15VENDOREXPERIMENT=1
113+
go get -d $PROJECT_URL
114+
cd $GOPATH/$PROJECT_PATH
115+
git submodule update --init
116+
go install ./...
117+
```
118+
119+
### How do I build gsv using Go 1.5?
120+
121+
See [here](#how-do-i-get-and-build-a-gsv-project-using-go-15)
122+
123+
### `go build ./...` or `go test ./...` fails
124+
125+
`gsv` only vendors dependencies which are needed to
126+
satisfy the building and testing of **your** project.
127+
Dependencies that are needed to build and test
128+
the vendored dependencies are **not** pulled in.
129+
130+
As a consequence `go build ./...` and/or `go test ./...`
131+
*might* break when run from the project's *root folder*.
132+
To fix this the following commands should be used
133+
134+
```sh
135+
go build $(go list ./... | grep -v vendor)
136+
go test $(go list ./... | grep -v vendor)
137+
```
138+
139+
This is not a joke but the
140+
[official recommendation](https://github.com/golang/go/issues/11659#issuecomment-171678025)
141+
by the golang team.
142+
143+
Note that pulling in the build- and test-dependencies of your
144+
dependencies is unlikely to fix `go build ./...` and `go test ./...`
145+
because the pulled in packages will then have unsatisfied
146+
dependencies. And going all the way down in the recursion doesn't
147+
seem to be the right solution for managing your build- and
148+
test-dependencies.
149+
150+
## TODO
151+
152+
In order of priority. Might not be implemented exactly as
153+
suggested here.
154+
155+
1. Add a `-purge` flag such that unused vendored submodules will
156+
be detected and removed.
157+
1. Use code from Go's stdlib instead from gb-vendor
158+
1. Add a `-updateall` flag such that all dependencies are
159+
updated to their current `origin/master`.
160+
1. Take a look at `git2go` and see if there is progress in
161+
submodules support.
162+
1. Look for alternatives for `libgit2`. This dependency reduces
163+
portability and makes installation a bit harder.
164+
165+
## Credits
166+
167+
This project could bootstrap on the work done for
168+
[gb-vendor](https://github.com/constabulary/gb/tree/master/cmd/gb-vendor).

‎git.go

+96
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
package main
2+
3+
/*
4+
#cgo pkg-config: libgit2
5+
6+
#include<git2.h>
7+
8+
// C macros seem not to be accessible in cgo
9+
git_clone_options clone_opts_init = GIT_CLONE_OPTIONS_INIT;
10+
11+
int just_return_origin(git_remote **out, git_repository *repo, const char *name, const char *url, void *payload)
12+
{
13+
return git_remote_lookup(out, repo, name);
14+
}
15+
16+
int just_return_repo(git_repository **out, const char *path, int bare, void *payload)
17+
{
18+
return git_submodule_open(out, (git_submodule*)payload);
19+
}
20+
*/
21+
import "C"
22+
import (
23+
"errors"
24+
"path/filepath"
25+
"unsafe"
26+
)
27+
28+
func init() {
29+
C.git_libgit2_init()
30+
}
31+
32+
func addSubmodule(url, repoDir, importPath string) error {
33+
var repo, r *C.git_repository
34+
var module *C.git_submodule
35+
36+
curDir := C.CString(".")
37+
defer C.free(unsafe.Pointer(curDir))
38+
defer C.git_repository_free(repo)
39+
if C.git_repository_open(&repo, curDir) < 0 {
40+
return convertErr(C.giterr_last())
41+
}
42+
43+
cURL := C.CString(url)
44+
defer C.free(unsafe.Pointer(cURL))
45+
submoduleSubpath := filepath.Join("vendor", importPath)
46+
cSubmoduleSubpath := C.CString(submoduleSubpath)
47+
defer C.free(unsafe.Pointer(cSubmoduleSubpath))
48+
defer C.git_submodule_free(module)
49+
if rc := C.git_submodule_add_setup(&module, repo, cURL, cSubmoduleSubpath, 1); rc < 0 {
50+
if rc == -4 {
51+
// submodule exists; job is done
52+
return nil
53+
}
54+
return convertErr(C.giterr_last())
55+
}
56+
57+
cloneOpts := C.clone_opts_init
58+
cloneOpts.repository_cb = (C.git_repository_create_cb)(unsafe.Pointer(C.just_return_repo))
59+
cloneOpts.remote_cb = (C.git_remote_create_cb)(unsafe.Pointer(C.just_return_origin))
60+
cloneOpts.repository_cb_payload = unsafe.Pointer(module)
61+
//cloneOpts.remote_cb_payload = unsafe.Pointer(module);
62+
63+
cSubmoduleRepoPath := C.CString(filepath.Join(repoDir, submoduleSubpath))
64+
defer C.free(unsafe.Pointer(cSubmoduleRepoPath))
65+
if C.git_clone(&r, cURL, cSubmoduleRepoPath, &cloneOpts) < 0 {
66+
return convertErr(C.giterr_last())
67+
}
68+
C.git_repository_free(r)
69+
70+
if C.git_submodule_add_finalize(module) < 0 {
71+
return convertErr(C.giterr_last())
72+
}
73+
return nil
74+
}
75+
76+
func findGitRoot(path string) (string, error) {
77+
buf := &C.git_buf{}
78+
p := C.CString(path)
79+
defer C.free(unsafe.Pointer(p))
80+
if C.git_repository_discover(buf, p, 0, nil) < 0 {
81+
return "", convertErr(C.giterr_last())
82+
}
83+
var repo *C.git_repository
84+
defer C.git_repository_free(repo)
85+
if C.git_repository_open(&repo, buf.ptr) < 0 {
86+
return "", convertErr(C.giterr_last())
87+
}
88+
return C.GoString(C.git_repository_workdir(repo)), nil
89+
}
90+
91+
func convertErr(err *C.git_error) error {
92+
if err != nil {
93+
return errors.New(C.GoString(err.message))
94+
}
95+
return errors.New("unknown error")
96+
}

‎main.go

+72
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
package main
2+
3+
import (
4+
"flag"
5+
"fmt"
6+
"log"
7+
"os"
8+
"sort"
9+
"strings"
10+
11+
// TODO: replace this dependency by stdlib
12+
"github.com/constabulary/gb/vendor"
13+
)
14+
15+
func main() {
16+
flag.Usage = func() {
17+
fmt.Fprintf(os.Stderr, "Usage: %s url|import path\n", os.Args[0])
18+
flag.PrintDefaults()
19+
}
20+
flag.Parse()
21+
log.SetFlags(0)
22+
23+
switch len(flag.Args()) {
24+
case 0:
25+
log.Fatal("import path missing")
26+
case 1:
27+
if err := run(flag.Arg(0)); err != nil {
28+
log.Fatal(err.Error())
29+
}
30+
default:
31+
log.Fatal("more than one import path supplied")
32+
}
33+
}
34+
35+
func run(path string) error {
36+
repo, err := NewRepo()
37+
if err != nil {
38+
return err
39+
}
40+
fullPath, err := repo.vendorDep(strings.TrimSuffix(stripscheme(path), ".git"))
41+
if err != nil {
42+
return err
43+
}
44+
45+
for {
46+
dsm, err := vendor.LoadPaths(repo.DependencyPaths...)
47+
if err != nil {
48+
return err
49+
}
50+
51+
is, ok := dsm[fullPath]
52+
if !ok {
53+
return fmt.Errorf("unable to locate depset for %q", path)
54+
}
55+
56+
missing := findMissing(pkgs(is.Pkgs), dsm)
57+
if len(missing) == 0 {
58+
break
59+
}
60+
// sort keys in ascending order, so the shortest missing import path
61+
// with be fetched first.
62+
keys := keys(missing)
63+
sort.Strings(keys)
64+
pkg := keys[0]
65+
fmt.Fprintf(os.Stderr, "analyzing recursive dependency %s\n", pkg)
66+
if _, err := repo.vendorDep(pkg); err != nil {
67+
return err
68+
}
69+
}
70+
71+
return nil
72+
}

‎repo.go

+45
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
package main
2+
3+
import (
4+
"os"
5+
"path/filepath"
6+
"runtime"
7+
8+
"golang.org/x/tools/go/vcs"
9+
)
10+
11+
type repo struct {
12+
Path string
13+
DependencyPaths []struct{ Root, Prefix string }
14+
}
15+
16+
func NewRepo() (*repo, error) {
17+
pwd, err := os.Getwd()
18+
if err != nil {
19+
return nil, err
20+
}
21+
repoPath, err := findGitRoot(pwd)
22+
if err != nil {
23+
return nil, err
24+
}
25+
if err := os.Chdir(repoPath); err != nil {
26+
return nil, err
27+
}
28+
paths := []struct{ Root, Prefix string }{{filepath.Join(runtime.GOROOT(), "src"), ""}}
29+
return &repo{Path: repoPath, DependencyPaths: paths}, nil
30+
}
31+
32+
func (r *repo) vendorDep(path string) (string, error) {
33+
rr, err := vcs.RepoRootForImportPath(path, false)
34+
if err != nil {
35+
return "", err
36+
}
37+
if err := addSubmodule(rr.Repo, filepath.Base(r.Path), rr.Root); err != nil {
38+
return "", err
39+
}
40+
41+
importPath := filepath.FromSlash(path)
42+
fullPath := filepath.Join(r.Path, "vendor", importPath)
43+
r.DependencyPaths = append(r.DependencyPaths, struct{ Root, Prefix string }{fullPath, importPath})
44+
return fullPath, nil
45+
}

0 commit comments

Comments
 (0)