Skip to content

Minikube docker driver doesn't work if Docker daemon is configured with the userns-remap option #9607

@bleggett

Description

@bleggett

Steps to reproduce the issue:

  1. Enable userns-remap option in your Docker daemon (defining appropriate entries in /etc/subuid/subguid as documented)
  2. Run minikube start
  3. Timeout due to inability to write to bind mounts or SSH.

I'm relatively certain this is because the Docker docs note that

The following standard Docker features are incompatible with running a Docker daemon with user namespaces enabled: - sharing PID or NET namespaces with the host (--pid=host or --network=host). - external (volume or storage) drivers which are unaware or incapable of using daemon user mappings. - **Using the --privileged mode flag on docker run without also specifying --userns=host.** 

(emphasis mine)

and the init logic is doing stuff like

docker exec --privileged minikube chown docker:docker /home/docker/.ssh/authorized_keys

Full output of failed command:

I1105 21:28:42.464515 70799 cli_runner.go:110] Run: docker network inspect bridge --format "{{(index .IPAM.Config 0).Subnet}},{{(index .IPAM.Config 0).Gateway}},{{(index .Options "com.docker.network.driver.mtu")}}" I1105 21:28:42.503322 70799 network_create.go:96] attempt to create network 192.168.49.0/24 with subnet: minikube and gateway 192.168.49.1 and MTU of 1500 ... I1105 21:28:42.503437 70799 cli_runner.go:110] Run: docker network create --driver=bridge --subnet=192.168.49.0/24 --gateway=192.168.49.1 -o --ip-masq -o --icc --label=created_by.minikube.sigs.k8s.io=true minikube -o com.docker.network.driver.mtu=1500 I1105 21:28:42.584167 70799 kic.go:93] calculated static IP "192.168.49.2" for the "minikube" container I1105 21:28:42.584261 70799 cli_runner.go:110] Run: docker ps -a --format {{.Names}} I1105 21:28:42.622476 70799 cli_runner.go:110] Run: docker volume create minikube --label name.minikube.sigs.k8s.io=minikube --label created_by.minikube.sigs.k8s.io=true I1105 21:28:42.663114 70799 oci.go:102] Successfully created a docker volume minikube I1105 21:28:42.663216 70799 cli_runner.go:110] Run: docker run --rm --entrypoint /usr/bin/test -v minikube:/var gcr.io/k8s-minikube/kicbase:v0.0.13@sha256:4d43acbd0050148d4bc399931f1b15253b5e73815b63a67b8ab4a5c9e523403f -d /var/lib I1105 21:28:43.333061 70799 oci.go:106] Successfully prepared a docker volume minikube W1105 21:28:43.333120 70799 oci.go:153] Your kernel does not support swap limit capabilities or the cgroup is not mounted. I1105 21:28:43.333126 70799 preload.go:97] Checking if preload exists for k8s version v1.19.2 and runtime docker I1105 21:28:43.333167 70799 preload.go:105] Found local preload: /home/ubuntu/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v6-v1.19.2-docker-overlay2-amd64.tar.lz4 I1105 21:28:43.333187 70799 kic.go:148] Starting extracting preloaded images to volume ... I1105 21:28:43.333193 70799 cli_runner.go:110] Run: docker info --format "'{{json .SecurityOptions}}'" I1105 21:28:43.333241 70799 cli_runner.go:110] Run: docker run --rm --entrypoint /usr/bin/tar -v /home/ubuntu/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v6-v1.19.2-docker-overlay2-amd64.tar.lz4:/preloaded.tar:ro -v minikube:/extractDir gcr.io/k8s-minikube/kicbase:v0.0.13@sha256:4d43acbd0050148d4bc399931f1b15253b5e73815b63a67b8ab4a5c9e523403f -I lz4 -xvf /preloaded.tar -C /extractDir I1105 21:28:43.403788 70799 cli_runner.go:110] Run: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var --security-opt apparmor=unconfined --memory=2200mb --memory-swap=2200mb --cpus=2 -e container=docker --expose 8443 --userns=host --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 gcr.io/k8s-minikube/kicbase:v0.0.13@sha256:4d43acbd0050148d4bc399931f1b15253b5e73815b63a67b8ab4a5c9e523403f I1105 21:28:43.983107 70799 cli_runner.go:110] Run: docker container inspect minikube --format={{.State.Running}} I1105 21:28:44.033416 70799 cli_runner.go:110] Run: docker container inspect minikube --format={{.State.Status}} I1105 21:28:44.097916 70799 cli_runner.go:110] Run: docker exec minikube stat /var/lib/dpkg/alternatives/iptables I1105 21:28:44.291324 70799 oci.go:245] the created container "minikube" has a running status. I1105 21:28:44.291352 70799 kic.go:179] Creating ssh key for kic: /home/ubuntu/.minikube/machines/minikube/id_rsa... I1105 21:28:44.501567 70799 vm_assets.go:96] NewFileAsset: /home/ubuntu/.minikube/machines/minikube/id_rsa.pub -> /home/docker/.ssh/authorized_keys I1105 21:28:44.501604 70799 kic_runner.go:179] docker (temp): /home/ubuntu/.minikube/machines/minikube/id_rsa.pub --> /home/docker/.ssh/authorized_keys (381 bytes) I1105 21:28:44.998841 70799 cli_runner.go:110] Run: docker container inspect minikube --format={{.State.Status}} I1105 21:28:45.096769 70799 kic_runner.go:93] Run: chown docker:docker /home/docker/.ssh/authorized_keys I1105 21:28:45.096792 70799 kic_runner.go:114] Args: [docker exec --privileged minikube chown docker:docker /home/docker/.ssh/authorized_keys] W1105 21:28:49.346897 70799 cli_runner.go:148] docker run --rm --entrypoint /usr/bin/tar -v /home/ubuntu/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v6-v1.19.2-docker-overlay2-amd64.tar.lz4:/preloaded.tar:ro -v minikube:/extractDir gcr.io/k8s-minikube/kicbase:v0.0.13@sha256:4d43acbd0050148d4bc399931f1b15253b5e73815b63a67b8ab4a5c9e523403f -I lz4 -xvf /preloaded.tar -C /extractDir returned with exit code 2 I1105 21:28:49.346944 70799 cli_runner.go:154] Completed: docker run --rm --entrypoint /usr/bin/tar -v /home/ubuntu/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v6-v1.19.2-docker-overlay2-amd64.tar.lz4:/preloaded.tar:ro -v minikube:/extractDir gcr.io/k8s-minikube/kicbase:v0.0.13@sha256:4d43acbd0050148d4bc399931f1b15253b5e73815b63a67b8ab4a5c9e523403f -I lz4 -xvf /preloaded.tar -C /extractDir: (6.013611315s) I1105 21:28:49.351754 70799 kic.go:155] Unable to extract preloaded tarball to volume: docker run --rm --entrypoint /usr/bin/tar -v /home/ubuntu/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v6-v1.19.2-docker-overlay2-amd64.tar.lz4:/preloaded.tar:ro -v minikube:/extractDir gcr.io/k8s-minikube/kicbase:v0.0.13@sha256:4d43acbd0050148d4bc399931f1b15253b5e73815b63a67b8ab4a5c9e523403f -I lz4 -xvf /preloaded.tar -C /extractDir: exit status 2 <snip many tar write errors> /usr/bin/tar: ./lib/docker/image/overlay2/distribution/v2metadata-by-diffid/sha256/57757cd7bb95e58d8e5c4b59bd07f73aa9fa446e52eb87aab87258357cdc1667: Cannot open: Permission denied /usr/bin/tar: ./lib/docker/image: Cannot utime: Operation not permitted /usr/bin/tar: ./lib/docker/image: Cannot change ownership to uid 0, gid 0: Operation not permitted /usr/bin/tar: Exiting with failure status due to previous errors I1105 21:28:49.362457 70799 cli_runner.go:110] Run: docker container inspect minikube --format={{.State.Status}} I1105 21:28:49.405721 70799 machine.go:88] provisioning docker machine ... I1105 21:28:49.405769 70799 ubuntu.go:166] provisioning hostname "minikube" I1105 21:28:49.405857 70799 cli_runner.go:110] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube I1105 21:28:49.451055 70799 main.go:119] libmachine: Using SSH client type: native I1105 21:28:49.451328 70799 main.go:119] libmachine: &{{{<nil> 0 [] [] []} docker [0x7e4fa0] 0x7e4f60 <nil> [] 0s} 127.0.0.1 32791 <nil> <nil>} I1105 21:28:49.451355 70799 main.go:119] libmachine: About to run SSH command: sudo hostname minikube && echo "minikube" | sudo tee /etc/hostname I1105 21:28:49.487335 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain I1105 21:28:52.520639 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain I1105 21:28:55.554354 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain I1105 21:28:58.587504 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain I1105 21:29:01.620960 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain I1105 21:29:04.653185 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain I1105 21:29:07.685572 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain I1105 21:29:10.718072 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain I1105 21:29:13.750733 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain I1105 21:29:16.782903 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain I1105 21:29:19.815134 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain I1105 21:29:22.847417 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain I1105 21:29:26.029242 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain I1105 21:29:29.061373 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain I1105 21:29:32.093713 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [publickey none], no supported methods remain I1105 21:29:35.125985 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain I1105 21:29:38.158449 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain I1105 21:29:41.191264 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [publickey none], no supported methods remain I1105 21:29:44.223366 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [publickey none], no supported methods remain I1105 21:29:47.255464 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain I1105 21:29:50.287963 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain I1105 21:29:53.320302 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [publickey none], no supported methods remain I1105 21:29:56.352435 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain I1105 21:29:59.384982 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain I1105 21:30:02.417702 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain I1105 21:30:05.450069 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain I1105 21:30:08.482266 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain I1105 21:30:11.514381 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain I1105 21:30:14.546839 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain I1105 21:30:17.579085 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain I1105 21:30:20.611148 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain I1105 21:30:23.643610 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain I1105 21:30:26.675962 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain I1105 21:30:29.708327 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain I1105 21:30:32.740870 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [publickey none], no supported methods remain I1105 21:30:35.773123 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain I1105 21:30:38.805161 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain I1105 21:30:41.837644 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain I1105 21:30:44.869864 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain I1105 21:30:47.902393 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain I1105 21:30:50.937813 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain I1105 21:30:53.970106 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain I1105 21:30:57.002628 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain I1105 21:31:00.035071 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain I1105 21:31:03.067227 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain I1105 21:31:06.099500 70799 main.go:119] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [publickey none], no supported methods remain 

Full output of minikube start command used, if not already included:

out.txt

Optional: Full output of minikube logs command:

docker info

ubuntu@ip:~$ docker info Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 1 Server Version: 18.09.7 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: runc version: N/A init version: v0.18.0 (expected: fec3683b971d9c3ef73f284f176672c44b448662) Security Options: apparmor seccomp Profile: default userns Kernel Version: 4.4.0-1117-aws Operating System: Ubuntu 16.04.7 LTS OSType: linux Architecture: x86_64 CPUs: 2 Total Memory: 7.795GiB Name: ID: Docker Root Dir: /var/lib/docker/165536.165536 Debug Mode (client): false Debug Mode (server): false Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false WARNING: No swap limit support 

Metadata

Metadata

Assignees

No one assigned

    Labels

    co/docker-driverIssues related to kubernetes in containerkind/documentationCategorizes issue or PR as related to documentation.kind/featureCategorizes issue or PR as related to a new feature.priority/awaiting-more-evidenceLowest priority. Possibly useful, but not yet enough support to actually get it done.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions