cmd/go: 'go build' error from 'git ls-remote' when downloading from private repo

[lihaic]

What version of Go are you using (go version)?

$ go version
go version go1.13rc1 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/lhc/.cache/go-build"
GOENV="/home/lhc/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY="*.bmi"
GONOSUMDB="*.bmi"
GOOS="linux"
GOPATH="/home/lhc/go"
GOPRIVATE="*.bmi"
GOPROXY="https://goproxy.io,direct"
GOROOT="/home/lhc/sdk/go1.13rc1"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/lhc/sdk/go1.13rc1/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/lhc/gitlab.bmi/tss/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build750408901=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I set two enviroment variable first.
$ go1.13rc1 env -w GOPROXY=https://goproxy.io,direct
$ go1.13rc1 env -w GOPRIVATE=*gitlab.bmi
Then i tried to build my project.
$ go1.13rc1 build -v
And then it failed, because some packages my project rely on is in a private repo.
The error message is:

# cd .; git ls-remote -- https://gitlab.bmi/aaa/logv
usage: git ls-remote [--heads] [--tags]  [--upload-pack=]
                     [-q | --quiet] [--exit-code] [--get-url] [ [...]]
# cd .; git ls-remote -- git+ssh://gitlab.bmi/aaa/logv
usage: git ls-remote [--heads] [--tags]  [--upload-pack=]
                     [-q | --quiet] [--exit-code] [--get-url] [ [...]]
# cd .; git ls-remote -- ssh://gitlab.bmi/aaa/logv
usage: git ls-remote [--heads] [--tags]  [--upload-pack=]
                     [-q | --quiet] [--exit-code] [--get-url] [ [...]]
go: gitlab.bmi/aaa/[email protected]: reading gitlab.bmi/aaa/logv.git/go.mod at revision v1.2.2: unknown revision v1.2.2

I thought this git ls-remote is trying to get tag info, is this supposed to be git ls-remote --tags or git ls remote

Then i modified the source code /home/lhc/sdk/go1.13rc1/src/cmd/go/internal/get/vcs.go line186
pingCmd: "ls-remote -- {scheme}://{repo}", ---> to pingCmd: "ls-remote {scheme}://{repo}",

And then i tried to run go1.13rc1 build -v, and it succeed.

What did you expect to see?

I hope after i set GOPROXY and GOPRIVATE, i could download package from both public repo and private repo.

What did you see instead?

But i can't run go build correctly.

Question is that is that line wrong?

Thank you for your help.

[aofei]

This seems to work as designed. What you need to know is that go isn't smart enough to handle VCS authentications on its own.

See: Why does "go get" use HTTPS when cloning a repository?

[bcmills]

Thanks for the report. This may be related to https://golang.org/cl/181237.

Two questions to help us reproduce the problem:

1. What happens if you try running those ls-remote commands individually on the command line (in an empty temporary directory)?
2. What version of the git command are you using (as reported by git version)?
   • See also doc/install: define minimum supported VCS versions #26746.

[lihaic]

$git version
git version 2.7.4

Try running those git ls-remote command in an empty directory.

$mkdir emptydir; cd emptydir
$git ls-remote -- https://gitlab.bmi/aaa/logv
usage: git ls-remote [--heads] [--tags]  [--upload-pack=]
                     [-q | --quiet] [--exit-code] [--get-url] [ [...]]
$git ls-remote -- git+ssh://gitlab.bmi/aaa/logv
usage: git ls-remote [--heads] [--tags]  [--upload-pack=]
                     [-q | --quiet] [--exit-code] [--get-url] [ [...]]
#git ls-remote -- ssh://gitlab.bmi/aaa/logv
usage: git ls-remote [--heads] [--tags]  [--upload-pack=]
                     [-q | --quiet] [--exit-code] [--get-url] [ [...]]

Running git ls-remote in an empty directory produces the same result.

$git ls-remote https://gitlab.bmi/aaa/logv
46c721ee6bab43c94dd8b3f3dc8a357cb4884627	HEAD
46c721ee6bab43c94dd8b3f3dc8a357cb4884627	refs/heads/master
1dccbdb4d84588a52a147e9f0e98ce222ba7027a	refs/heads/v1
2741972f4356fd428439411eba51e19639fc4df1	refs/tags/v1.0.0
46c721ee6bab43c94dd8b3f3dc8a357cb4884627	refs/tags/v1.1.0
2c05092bd904db3c5d0f3dfa81059bb0808c317e   refs/tags/v1.2.0
1aeae1413f640f643de9d864f09b36c85d5409ca	refs/tags/v1.2.1
1dccbdb4d84588a52a147e9f0e98ce222ba7027a	refs/tags/v1.2.2
$git ls-remote --tags https://gitlab.bmi/aaa/logv
2741972f4356fd428439411eba51e19639fc4df1	refs/tags/v1.0.0
46c721ee6bab43c94dd8b3f3dc8a357cb4884627	refs/tags/v1.1.0
2c05092bd904db3c5d0f3dfa81059bb0808c317e   refs/tags/v1.2.0
1aeae1413f640f643de9d864f09b36c85d5409ca	refs/tags/v1.2.1
1dccbdb4d84588a52a147e9f0e98ce222ba7027a	refs/tags/v1.2.2

But running git ls-remote without the -- or with --tags get all tag info.

By the way, i order to use private repo, i added ssh git config like below.

$ cat ~/.gitconfig
[user]
	name = lhc
	email = [email protected]
[url "ssh://[email protected]/"]
	insteadOf = https://gitlab.bmi/

I think this sholud only avoid go tools using https since my private repo don't have https.
@bcmills Appreciate all you help, thanks.

[lihaic]

I am currently using ubuntu 16.04, default git version is 2.7.4.

$sudo add-apt-repository -y ppa:git-core/ppa
$sudo apt-get update
$sudo apt-get install git -y
$git version
git version 2.23.0

I did this to upgrade git to v2.23.0, and then git ls-remote -- worked! And with the origin go1.13rc1 binary, go build works fine with my private repo.
So seems like this problem related to git version.

[gopherbot]

Change https://golang.org/cl/191978 mentions this issue: cmd/go/internal/get: remove '--' separator from 'git ls-remote' command

[bcmills]

@gopherbot, please backport to Go 1.13.

[gopherbot]

Backport issue(s) opened: #33880 (for 1.13).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

[gopherbot]

Change https://golang.org/cl/191973 mentions this issue: [release-branch.go1.13] cmd/go/internal/get: remove '--' separator from 'git ls-remote' command