DEV Community

Cover image for How to use the Pulumi Operator to build Docker Images
Martin PAUCOT
Martin PAUCOT

Posted on • Originally published at martin-paucot.fr

How to use the Pulumi Operator to build Docker Images

#pulumi #devops #kubernetes #docker

I recently faced a problem when trying to build a Docker image using the @pulumi/docker provider.

In this Blog Post I explain my journey on how I use the Pulumi Operator to build Docker Images and what are the problems I encountered.

If you simply want the solution, click here to scroll down

import * as docker from '@pulumi/docker' import * as kubernetes from '@pulumi/kubernetes' const image = new docker.Image("app", { build: { context: "../", args: { BUILDKIT_INLINE_CACHE: "1", }, cacheFrom: { images: ["ghcr.io/kerwanp/pulumi-preview-test:latest"] } }, imageName: "ghcr.io/kerwanp/pulumi-preview-test:latest", }) const appLabels = { app: "preview-test" } const deployment = new kubernetes.apps.v1.Deployment("app", { spec: { replicas: 1, selector: { matchLabels: appLabels }, template: { metadata: { labels: appLabels }, spec: { imagePullSecrets: [ { name: 'regcred' } ], containers: [ { name: "app", image: image.imageName, imagePullPolicy: 'Always', } ] } } } })
Enter fullscreen mode Exit fullscreen mode

And then deploy it by creating a stack using the Pulumi Operator:

--- apiVersion: pulumi.com/v1 kind: Stack metadata: name: app-stack namespace: pulumi spec: envRefs: PULUMI_ACCESS_TOKEN: type: Secret secret: name: pulumi-secret key: pulumiAccessToken gitAuth: basicAuth: password: type: Secret secret: name: pulumi-secret key: githubToken userName: type: Literal literal: value: kerwanp stack: kerwanp/preview-test/test projectRepo: https://github.com/kerwanp/pulumi-preview-test repoDir: deploy branch: "refs/heads/main" destroyOnFinalize: true
Enter fullscreen mode Exit fullscreen mode

The problem

When the Pulumi Operator tries to deploy the stack it throws the following error:

failed to run update: exit status 255code: 255 stdout: Updating (test) View Live: https://app.pulumi.com/kerwanp/preview-test/test/updates/1 + pulumi:pulumi:Stack preview-test-test creating (0s) @ Updating...... + pulumi:pulumi:Stack preview-test-test creating (2s) panic: runtime error: invalid memory address or nil pointer dereference + pulumi:pulumi:Stack preview-test-test creating (2s) [signal SIGSEGV: segmentation violation code=0x1 addr=0x29 pc=0x124befc] + pulumi:pulumi:Stack preview-test-test creating (2s) goroutine 27 [running]: + pulumi:pulumi:Stack preview-test-test creating (2s) github.com/pulumi/pulumi-docker/provider/v4.dockerHybridProvider.Configure({{}, {0x21b7ca0, 0x26157, 0x26157}, {0x17dd3fc, 0x6}, {0x17fe470, 0xc0000d1400}, {0x17fe3c0, 0xc0000b0690}}, ...) + pulumi:pulumi:Stack preview-test-test creating (2s) /home/runner/work/pulumi-docker/pulumi-docker/provider/hybrid.go:93 +0x17c + pulumi:pulumi:Stack preview-test-test creating (2s) github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Configure_Handler.func1({0x17f1f08, 0xc00042e6f0}, {0x14c4e00?, 0xc0007d7900}) + pulumi:pulumi:Stack preview-test-test creating (2s) /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.64.0/proto/go/provider_grpc.pb.go:462 +0x78 + pulumi:pulumi:Stack preview-test-test creating (2s) github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1({0x17f1f08, 0xc0006ffe90}, {0x14c4e00, 0xc0007d7900}, 0xc000136ac0, 0xc00043eca8) + pulumi:pulumi:Stack preview-test-test creating (2s) /home/runner/go/pkg/mod/github.com/grpc-ecosystem/grpc-opentracing@v0.0.0-20180507213350-8e809c8a8645/go/otgrpc/server.go:57 +0x3e8 + pulumi:pulumi:Stack preview-test-test creating (2s) github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Configure_Handler({0x153bb40?, 0xc0007644b0}, {0x17f1f08, 0xc0006ffe90}, 0xc0005da8c0, 0xc000653960) + pulumi:pulumi:Stack preview-test-test creating (2s) /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.64.0/proto/go/provider_grpc.pb.go:464 +0x138 + pulumi:pulumi:Stack preview-test-test creating (2s) google.golang.org/grpc.(*Server).processUnaryRPC(0xc000234000, {0x17fa100, 0xc000702b60}, 0xc0003c5e60, 0xc00075cba0, 0x21f98e8, 0x0) + pulumi:pulumi:Stack preview-test-test creating (2s) /home/runner/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1345 +0xdf3 + pulumi:pulumi:Stack preview-test-test creating (2s) google.golang.org/grpc.(*Server).handleStream(0xc000234000, {0x17fa100, 0xc000702b60}, 0xc0003c5e60, 0x0) + pulumi:pulumi:Stack preview-test-test creating (2s) /home/runner/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1722 +0xa36 + pulumi:pulumi:Stack preview-test-test creating (2s) google.golang.org/grpc.(*Server).serveStreams.func1.2() + pulumi:pulumi:Stack preview-test-test creating (2s) /home/runner/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:966 +0x98 + pulumi:pulumi:Stack preview-test-test creating (2s) created by google.golang.org/grpc.(*Server).serveStreams.func1 + pulumi:pulumi:Stack preview-test-test creating (2s) /home/runner/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:964 +0x28a docker:index:Image app error: error reading from server: EOF docker:index:Image app **failed** 1 error + pulumi:pulumi:Stack preview-test-test created (2s) 19 messages Diagnostics: docker:index:Image (app): error: error reading from server: EOF pulumi:pulumi:Stack (preview-test-test): panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x29 pc=0x124befc] goroutine 27 [running]: github.com/pulumi/pulumi-docker/provider/v4.dockerHybridProvider.Configure({{}, {0x21b7ca0, 0x26157, 0x26157}, {0x17dd3fc, 0x6}, {0x17fe470, 0xc0000d1400}, {0x17fe3c0, 0xc0000b0690}}, ...) /home/runner/work/pulumi-docker/pulumi-docker/provider/hybrid.go:93 +0x17c github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvide r_Configure_Handler.func1({0x17f1f08, 0xc00042e6f0}, {0x14c4e00?, 0xc0007d7900}) /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.64.0/proto/go/provider_grpc.pb.go:462 +0x78 github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1({0x17f1f08, 0xc0006ffe90}, {0x14c4e00, 0xc0007d7900}, 0xc000136ac0, 0xc00043eca8) /home/runner/go/pkg/mod/github.com/grpc-ecosystem/grpc-opentracing@v0.0.0-20180507213350-8e809c8a8645/go/otgrpc/server.go:57 +0x3e8 github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Configure_Handler({0x153bb40?, 0xc0007644b0}, {0x17f1f08, 0xc0006ffe90}, 0xc0005da8c0, 0xc000653960) /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.64.0/proto/go/provider_grpc.pb.go:464 +0x138 google.golang.org/grpc.(*Server).processUnaryRPC(0xc000234000, {0x17fa100, 0xc000702b60}, 0xc0003c5e60, 0xc00075cba0, 0x21f98e8, 0x0) /home/runner/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1345 +0xdf3 google.golang.org/grpc.(*Server).handleStream(0xc000234000, {0x17fa100, 0xc000702b60}, 0xc0003c5e60, 0x0) /home/runner/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1722 +0xa36 google.golang.org/grpc.(*Server).serveStreams.func1.2() /home/runner/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:966 +0x98 created by google.golang.org/grpc.(*Server).serveStreams.func1 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:964 +0x28a Resources: + 1 created Duration: 4s stderr: warning: A new version of Pulumi is available. To upgrade from version '3.66.0' to '3.68.0', visit https://pulumi.com/docs/reference/install/ for manual instructions and release notes.
Enter fullscreen mode Exit fullscreen mode

The Debuging Process

Replicating the issue

I opened a shell on the Pulumi Operator Pod to see if I can replicate the pulumi up inside the container.

I found that the Operator cloned my repository inside /tmp/pulumi-working/<stack-name> I jumped into it and simply ran pupumi up.

Problem...

getcwd() failed: No such file or directory
Enter fullscreen mode Exit fullscreen mode

This error usually appears when the folder you are in does not exist anymore, so I decided to duplicate it and re-run pulumi up.

And voilà, error replicated!

Docker is simply not there..

If I try to build the Docker image by myself directly from the container I receive the following error:

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Enter fullscreen mode Exit fullscreen mode

It makes completely sense! If you do not have a Docker Daemon running, you cannot build a Docker image.

The solution

The idea is to add a docker:dind container inside the Pulumi Operator pod to be used as a Docker Daemon.

Starting in 18.09+ the Docker Dind automatically generate TLS certificates. We have to mount a directory accross the containers to share thoses.

--- apiVersion: apps/v1 kind: Deployment metadata: name: pulumi-operator-fc266171 namespace: pulumi spec: replicas: 1 selector: matchLabels: name: pulumi-kubernetes-operator template: spec: volumes: - name: dind-storage emptyDir: {} - name: docker-certs-client emptyDir: {} containers: - name: dind image: docker:24-dind resources: {} volumeMounts: - name: dind-storage mountPath: /var/lib/docker - name: docker-certs-client mountPath: /certs/client # ^^^^^^ # Mount volume to share certificates to the operator securityContext: privileged: true # ^^^^ # Must be defined to run Docker - name: pulumi-kubernetes-operator image: pulumi/pulumi-kubernetes-operator:v1.12.0 args: - "--zap-level=error" - "--zap-time-encoding=iso8601" env: - name: DOCKER_HOST value: 'tcp://localhost:2376' # ^^^^ # Define address for remote Docker Daemon  - name: DOCKER_CERT_PATH value: '/certs' # ^^^^ # Define custom certificate directory (mounted volume)  - name: DOCKER_TLS_VERIFY value: 1 # ^^^^ # Force Docker client to use TLS  - name: WATCH_NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace - name: POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name - name: OPERATOR_NAME value: pulumi-kubernetes-operator - name: GRACEFUL_SHUTDOWN_TIMEOUT_DURATION value: 5m - name: MAX_CONCURRENT_RECONCILES value: "10" resources: {} volumeMounts: - name: docker-certs-client mountPath: /certs # ^^^^^^ # Mount volume to get Docker certificates terminationMessagePath: /dev/termination-log terminationMessagePolicy: File imagePullPolicy: Always
Enter fullscreen mode Exit fullscreen mode

You can actually check that it works by running docker info inside the Operator Pod.

Client: Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.10.4 Path: /usr/libexec/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.17.3 Path: /usr/libexec/docker/cli-plugins/docker-compose Server: Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 0 Server Version: 24.0.1 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 1 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 runc Default Runtime: runc Init Binary: docker-init containerd version: 1677a17964311325ed1c31e2c0a3589ce6d5c30d runc version: v1.1.7-0-g860f061 init version: de40ad0 Security Options: seccomp Profile: builtin Kernel Version: 5.15.90.1-microsoft-standard-WSL2 Operating System: Alpine Linux v3.18 (containerized) OSType: linux Architecture: x86_64 CPUs: 16 Total Memory: 31.32GiB Name: pulumi-operator-anaxago-fc266171-6f896556dc-lxkvp ID: 32511430-499e-4fa7-bbce-9b801ffb77ec Docker Root Dir: /var/lib/docker Debug Mode: false Registry: https://index.docker.io/v1/ Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false Product License: Community Engine
Enter fullscreen mode Exit fullscreen mode

Tada! 🎉 You can now build images using Pulumi and the Pulumi Operator!

I hope that this Blog Post helped you! If you have any questions, feel free to use the comment section! 💬

Oh and if you want more content like this, follow me:

Top comments (0)