Can not pull dev image on OpenShift Origin v3.6.0-rc.0

[slaskawi]

The HOSA agent template in master branch causes Failed to pull image "hawkular/hawkular-openshift-agent:dev": rpc error: code = 2 desc = manifest unknown: manifest unknown errors.

Steps to reproduce:

1. wget https://raw.githubusercontent.com/jpkrohling/hawkular-openshift-agent/695fe99d3deb77b497518a8065b36db641ad6401/deploy/openshift/hawkular-openshift-agent-configmap.yaml
2. wget https://raw.githubusercontent.com/hawkular/hawkular-openshift-agent/master/deploy/openshift/hawkular-openshift-agent.yaml
3. oc create -f hawkular-openshift-agent-configmap.yaml -n default
4. oc process -f hawkular-openshift-agent.yaml | ./oc/oc create -n default -f -
5. oc adm policy add-cluster-role-to-user hawkular-openshift-agent system:serviceaccount:default:hawkular-openshift-agent
6. oc describe pod hawkular-openshift-agent-2hd10
7. Observe the events part of the output.

This issue might be connected to openshift/origin#9834 however the template does not use ImageStreams (so the suggested fix won't work).

Environment:

• OpenShift v3.6.0-rc.0:

./oc/oc version
oc v3.6.0-rc.0+66d9125-131-dirty
kubernetes v1.7.0+695f48a16f
features: Basic-Auth

Server https://127.0.0.1:8443
kubernetes v1.6.1+5115d708d7

• Fedora F25:

uname -r
4.11.10-200.fc25.x86_64

[jmazzitelli]

This is strange because master branch hasn't changed (that I remember) since the last tagged release. So the dev tag in your local docker daemon should be the same as the last release up in docker hub.

To eliminate the chance that your local dev build is bad, try to pull the latest tagged release - 1.4.1.Final:

https://hub.docker.com/r/hawkular/hawkular-openshift-agent/tags/

You might also want to try "latest" tagged release as well to see what that does.

[update: I stand corrected, there are two additional commits after the last release - but they don't seem relevant to this kind of issue. You should still verify the behavior of 1.4.1.Final - or at least the "latest" release (that one will have those two extra commits as well).]

[slaskawi]

Ha! It seems this is a Docker thing:

$ docker pull hawkular/hawkular-openshift-agent:dev                                                                                                   
Trying to pull repository docker.io/hawkular/hawkular-openshift-agent ... 
manifest unknown: manifest unknown

But the latest works fine:

$ docker pull hawkular/hawkular-openshift-agent:latest                                                                                                
Trying to pull repository docker.io/hawkular/hawkular-openshift-agent ... 
sha256:be795fcce57f7c708af6adc1affedb7bb512e3aa1556d04eec2949313559227c: Pulling from docker.io/hawkular/hawkular-openshift-agent
Digest: sha256:be795fcce57f7c708af6adc1affedb7bb512e3aa1556d04eec2949313559227c
Status: Image is up to date for docker.io/hawkular/hawkular-openshift-agent:latest

My Docker version is the following:

$ rpm -qa docker                                                                                                                                 
docker-1.12.6-6.gitae7d637.fc25.x86_64

So it seems this is connected to my local environment. just to confirm this, could you please tell me what is the Docker version on the machine used to push dev image? My lucky guess is that it is > 1.12.6.

For those who look for out-of-the-box solution, you may try to switch --signature-verification=false flag in Docker Daemon (see BZ1430266).

[jmazzitelli]

Running travis and adding "docker --version" in the build, I see this:

$ docker --version
Docker version 17.03.1-ce, build c6d412e

[jmazzitelli]

Just to be clear, that's the docker version being used by travis - see travis build log here: https://travis-ci.org/hawkular/hawkular-openshift-agent#L691

[slaskawi]

Thanks a lot for checking this @jmazzitelli!

It seems my theory was correct. According to Docker version history, 17.03.1-ce is one release ahead of 1.12.6 which is the default for Fedora F25. Fedora F26 uses 1.13.1 (link here) and CentOS 7 1.12.6 (link here). So either the client needs to upgrade Docker to >= 17.03 (unfortunately this is not possible with standard repos but using Docker repos might do the trick) or you might consider downgrading Docker in Travis to <= 1.12.X (which would probably make some of the users quite happy, like myself 😃).

[xiaods]

fix it by #177