Cannot pull from an insecure registry when referred to by hostname

[mwcd]

Summary

pack create-builder and pack build fail with similar messages if the builder references images within an insecure registry referred to by hostname:

$ pack create-builder test-builder --config test-builder-config.toml
ERROR: failed to create builder: fetch build image: connect to repo store ?mdavis-l:5000/stack-img?: Get ?https://mdavis-l:5000/v2/?: http: server gave HTTP response to HTTPS client

---

Reproduction

Steps

1. Create an insecure local registry

$ docker run -d -p 5000:5000 --restart=always --name registry registry:2

2. Add your new registry to your insecure-registries list

    "insecure-registries" : ["mdavis-l:5000"]

3. Restart your docker daemon

$ sudo systemctl restart docker

4. Create a builder-config.toml file referencing a build-image or run-image from your insecure registry

# Stack that will be used by the builder
[stack]
  id = "test-cnb-stack"
	build-image = "mdavis-l:5000/stack-img"
	run-image = "mdavis-l:5000/stack-img"

5. Create your builder

$ pack create-builder builder-img --config test-builder.toml

Current behavior

$ pack create-builder test-builder --config test-builder-config.toml
ERROR: invalid run image config: failed to fetch image: connect to repo store 'mdavis-l:5000/stack-img': Get "https://mdavis-l:5000/v2/": http: server gave HTTP response to HTTPS client

Expected behavior

I expected the builder to be created successfully, as happens when I refer to my registry by IP rather than by hostname

$ pack create-builder builder-img --config test-builder.toml
latest: Pulling from stack-img
Digest: sha256:41144072024fed8d3ea3bb1b95f78e077d4d504dd3a756ae5f5d06a2e3af8ec5
Status: Image is up to date for 10.49.42.30:5000/stack-img:latest
Successfully created builder image builder-img
Tip: Run pack build <image-name> --builder builder-img to use this builder

---

Environment

pack info

$ pack report
Pack:
  Version:  0.12.0+git-a3ff87a.build-823
  OS/Arch:  linux/amd64

Default Lifecycle Version:  0.8.0

Config:
  default-builder-image = "mdavis-l:5000/builder-img"

docker info

$ docker info
Client:
 Debug Mode: false

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 24
 Server Version: 18.09.7
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: bb71b10fd8f58240ca47fbb579b9d1028eea7c84
 runc version: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 3.10.0-862.el7.x86_64
 Operating System: Red Hat Enterprise Linux Server 7.5 (Maipo)
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 31.29GiB
 Name: mdavis-l
 ID: GA4B:72L4:YQPP:MSUV:BMB4:MWLI:3HWP:35RU:R5LP:RHMP:OKA5:YMKN
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  mdavis-l:5000
  10.49.42.30:5000
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

[dfreilich]

Interesting. I tried verifying it, but I get another error:

$ curl -H GET "http://dfreilich.local:5000/v2/_catalog"
{"repositories":["stack-img"]}

$ pack create-builder builder-img --config builder.toml
Warning: builder configuration: empty order definition
ERROR: failed to create builder: fetch build image: connect to repo store 'dfreilich.local:5000/stack-img': Get "http://dfreilich.local:5000/v2/": dial tcp: lookup dfreilich.local on <IP>: no such host

[jromero]

@dfreilich I believe your particular issue is due to it running on a MacOS which uses a VM that has different networking properties.

It looks like the OP is using Linux.

[aemengo]

@mwcd I'm interested in this issue. I'm taking a look if you don't mind 👍🏾

I'm definitely curious though, what's the doing the resolution of mdavis-l -> 10.49.42.30? (ex. /etc/hosts)?

Thanks for such a detailed write-up.

---

EDIT:

I was able to reproduce. I performed the steps with:

1. An entry into /etc/hosts
2. A Global DNS entry

In both cases, the error following message showed up, even though DNS resolution succeeded inside a build container in case 2:

ERROR:
  invalid run image config:
    failed to fetch image:
      connect to repo store 'mdavis-l:5000/sample-stack-run:bionic':
        Get "https://mdavis-l:5000/v2/": http: server gave HTTP response to HTTPS client

---

EDIT:

My thoughts are that this issue should be closed.

I was able to reproduce without using buildpacks/pack code, but with the google/go-containerregistry library directly. My guess is that the gogcr library doesn't assume the insecure-registries option by default and I didn't see an option to pass it in. Here the repro code: https://gist.github.com/aemengo/f10d12de932c0ea3fcd43c4f773a5260

I tested and reproduced on 2 Linux Machines:

1. Operating System: Red Hat Enterprise Linux Server 7.9 (Maipo)
2. Operating System: Ubuntu 18.04.5 LTS

Output:

[Success] Accessed 'library/hello-world'
[Error] Get "https://mdavis-l:5000/v2/": http: server gave HTTP response to HTTPS client

Looks like a related issue: buildpacks/lifecycle#524

[jromero]

The issue buildpacks/lifecycle#524 is regarding the non-daemon case (aka --publish). It doesn't appear to be applicable to this issue based on the initial description.

@aemengo, in your reproduction it looks like you are using the remote package. Can you try it with daemon?

[aemengo]

I synced up with @jromero out-of-band for this.

Looks like the problem ultimately is that we're not passing "insecure-registries" : ["mdavis-l:5000"] options to the underlying google/go-containerregistry. We could move in that direction but it's a little more pragmatic to let the docker daemon do the heavy lifting here (instead of terminating early) because the daemon can already handle the insecure-registry options as you've configured.

I created #1145 to get around this issue.

Again, many thanks from bringing it up! 👍🏾