I'm getting a "Failed to pull image" error on my deployment in minikube with a local registry, despite other deployments using the same image registry working as expected.

My workflow is to build the image from a Dockerfile called docker.collection as below.

$ minikube start --insecure-registry localhost:5000 --memory 4096 $ docker build . -f docker.collection -t localhost:5000/collection:latest -t localhost:5000/collection:dev $ docker push localhost:5000/collection $ cat deployment.yaml | kubectl apply -f - 

The deployment is created as desired, but the pods don't start, and the minikube dashboard shows:

Failed to pull image "localhost:5000/collection:dev": rpc error: code = Unknown desc = Error response from daemon: manifest for localhost:5000/collection:dev not found

I have no reason to believe there was any problem in building or pushing the image, and on the host machine, I can pull successfully...

$ docker pull localhost:5000/collection:dev dev: Pulling from collection Digest: sha256:8becfdd412f0b86ece8335aa5ee1aede75992d17f88739a28da5939eab28fde5 Status: Image is up to date for localhost:5000/collection:dev 

What should else I be checking? How can I get more detail on the failure?

Further debugging

I've also checked:

• the minikube node can reach localhost:5000:

$ minikube ssh $ ping localhost:5000 PING localhost:5000 (127.0.0.1): 56 data bytes 64 bytes from 127.0.0.1: seq=0 ttl=64 time=0.126 ms 64 bytes from 127.0.0.1: seq=1 ttl=64 time=0.041 ms 64 bytes from 127.0.0.1: seq=2 ttl=64 time=0.058 ms ^C --- localhost:5000 ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 0.041/0.075/0.126 ms $ 

• a minimal run of the image fails in the same way: kubectl run coll-test --image=localhost:5000/collection:dev yields the same events:

Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 24s default-scheduler Successfully assigned dev/coll-test-cf4467b8d-8vpdk to minikube Normal BackOff 23s kubelet, minikube Back-off pulling image "localhost:5000/collection:dev" Warning Failed 23s kubelet, minikube Error: ImagePullBackOff Normal Pulling 7s (x2 over 23s) kubelet, minikube Pulling image "localhost:5000/collection:dev" Warning Failed 7s (x2 over 23s) kubelet, minikube Failed to pull image "localhost:5000/collection:dev": rpc error: code = Unknown desc = Error response from daemon: manifest for localhost:5000/collection:dev not found Warning Failed 7s (x2 over 23s) kubelet, minikube Error: ErrImagePull 

• the entries for the image exist in the repo:

$ curl http://localhost:5000/v2/_catalog {"repositories":["collection","foo","bar"]} 

Deployment and Dockerfile

# deployment.yaml apiVersion: apps/v1beta2 kind: Deployment metadata: namespace: dev name: collection labels: app: collection spec: replicas: 1 selector: matchLabels: app: collection template: metadata: labels: app: collection spec: containers: - name: collection image: "localhost:5000/collection:dev" command: ["bash", "./collection_entry.sh"] imagePullPolicy: Always 

# docker.collection FROM python:3 WORKDIR /code COPY . /code/ RUN adduser --system --group --no-create-home celery RUN mkdir /var/run/celery RUN mkdir /var/log/celery RUN chown celery:celery /var/run/celery RUN chown celery:celery /var/log/celery RUN chmod 755 /var/run/celery RUN chmod 755 /var/log/celery RUN pip3 -q install -r requirements.txt USER celery ENTRYPOINT ./collection_entry.sh