How to use GitLab Development Kit
Container Registry

Container Registry

Depending on your needs, you can set up Container Registry locally in the following ways:

  • Display the Container Registry in the UI only (not push or pull images).
  • Use the Container Registry as an insecure registry (can push and pull images).
  • Use the Container Registry with a self-signed certificate (can push and pull images).

Set up Container Registry to display in UI only

To set up Container Registry to display in the UI only (but not be able to push or pull images) add the following to your gdk.yml:

registry:  enabled: true

Then run the following commands:

  1. gdk reconfigure.
  2. gdk restart.

Set up pushing and pulling of images over HTTP

To set up Container Registry to allow pushing and pulling of images over HTTP, you must have a Docker-compatible client installed. For example:

In these instructions, we assume you set up registry.test.

  1. Update gdk.yml as follows:

    hostname: gdk.test registry:  enabled: true  host: registry.test  self_signed: false  auth_enabled: true  listen_address: 0.0.0.0

  2. Locate the Docker daemon configuration file and set the insecure-registries directive to point to the local registry registry.test:5100:

    • For Rancher Desktop, create a provisioning script:

      • Create ~/Library/Application\ Support/rancher-desktop/lima/_config/override.yaml. Enter the following:

        provision:  - mode: system  script: |  #!/bin/sh  cat <<'EOF' > /etc/docker/daemon.json  {  "insecure-registries": ["registry.test:5100"]  }  EOF

      • Restart Rancher Desktop

    • For Colima, see How to customize Docker config e.g. add insecure registries?.

      • You also need to redirect the registry hostname in Colima itself. To do so, run the following:

        colima ssh sudo sh -c 'echo "172.16.123.1 registry.test" >> /etc/hosts' sudo sh -c 'echo "172.16.123.1 gdk.test" >> /etc/hosts' exit colima restart

    • For general information, see the CNCF documentation.

  3. Restart the Docker engine.

  4. Run gdk reconfigure.

  5. Run gdk restart.

After completing these instructions, you should be ready to work with the registry locally. See the Interacting with the local container registry section for examples of how to query the registry manually using curl.

Set up pushing and pulling of images over HTTPS

This section is relevant if you set self_signed: true in your gdk.yml.

Since the registry is self-signed, Docker treats it as insecure. The certificate must be in your GDK root, called registry_host.crt, and must be copied as ca.crt to the appropriate configuration location.

If you are using Docker Desktop for Mac, GDK includes the shorthand:

rm -f registry_host.{key,crt} && make trust-docker-registry

This places the certificate under ~/.docker/certs.d/$REGISTRY_HOST:$REGISRY_PORT/ca.crt, overwriting any existing certificate at that path.

Afterwards, you must restart Docker to apply the changes.

Observe the registry

Run gdk tail registry.

Example:

registry : level=warning msg="No HTTP secret provided - generated random secret ... registry : level=info msg="redis not configured" go.version=go1.11.2 ... registry : level=info msg="Starting upload purge in 13m0s" go.version=go1.11.2 ... registry : level=info msg="using inmemory blob descriptor cache" go.version=go1.11.2 ... registry : level=info msg="listening on [::]:5100" go.version=go1.11.2 ...

Visit $REGISTRY_HOST:$REGISTRY_PORT (such as registry.test:5100) in your browser. Any response, even a blank page, means that the registry is probably running. If the registry is running, the output of gdk tail changes.

Configure an insecure registry for GitLab CI/CD

If you’re not using a self-signed certificate, you can instruct Docker to consider the registry as insecure. For example, Docker-in-Docker builds require an additional flag, --insecure-registry:

# .gitlab-ci.yml  services:  - name: docker:stable-dind  command: ["--insecure-registry=registry.test:5100"]

Configure a local Docker-based runner

For Docker-in-Docker builds to work in a local runner, you must also make the nested Docker service trust the certificates by editing volumes under [[runners.docker]] in your runner’s .toml configuration to include:

$HOME/.docker/certs.d:/etc/docker/certs.d

replacing $HOME with the expanded path. For example

volumes = ["/Users/hfyngvason/.docker/certs.d:/etc/docker/certs.d", "/certs/client", "/cache"]

Interacting with the local container registry

In this section, we assume you have obtained a Personal Access Token with all permissions, and exported it as GITLAB_TOKEN in your environment:

export GITLAB_TOKEN=...

Using the Docker Client

  • If you have authentication enabled, logging in is required.
  • If you have a self-signed local registry, trusting the registry’s certificates is required.
Log in to the registry 
docker login gdk.test:5100 -u gitlab-token -p "$GITLAB_TOKEN"
Build and tag an image 
docker build -t gdk.test:5100/custom-docker-image .
Push the image to the local registry 
docker push gdk.test:5100/custom-docker-image

Using HTTP

  • If you have a self-signed certificate, you can add --cacert registry_host.crt or -k to the curl commands.

  • If you have authentication enabled, you must obtain a bearer token for your requests:

    export GITLAB_REGISTRY_JWT=`curl "http://gitlab-token:$GITLAB_TOKEN@gdk.test:3000/jwt/auth?service=container_registry&scope=$SCOPE" | jq -r .token`

    where $SCOPE should be

    • registry:catalog:* to interact with the catalog
    • repository:your/project/path:* to interact with the images associated with a particular project

    Alternatively, you can obtain the token via the Rails console:

    ::Auth::ContainerRegistryAuthenticationService.pull_access_token('your/project/path')

    To use the token, append it as a header flag to the curl command:

    -H "Authorization: Bearer $GITLAB_REGISTRY_JWT"

The commands below assume a self-signed registry with authentication enabled, as this is the most complicated use case.

Retrieve a list of images available in the repository 
curl --cacert registry_host.crt -H "Authorization: Bearer $GITLAB_REGISTRY_JWT" \  gdk.test:5100/v2/_catalog
{  "repositories": [  "secure-group/docker-image-test",  "secure-group/klar",  "secure-group/tests/ruby-bundler/master",  "testing",  "ubuntu"  ] }
List tags for a specific image 
curl --cacert registry_host.crt -H "Authorization: Bearer $GITLAB_REGISTRY_JWT" \  gdk.test:5100/v2/secure-group/tests/ruby-bundler/master/tags/list
{  "tags": [  "3bf5c8efcd276bf6133ccb787e54b7020a00b99c",  "ca928571c661c42dbdadc090f4ef78c8f2854dd9",  "f7182b792a58d282ef3c69c2c6b7a22f78b2e950"  ], "name": "secure-group/tests/ruby-bundler/master" }
Get image manifest 
curl --cacert registry_host.crt -H "Authorization: Bearer $GITLAB_REGISTRY_JWT" \  gdk.test:5100/v2/secure-group/tests/ruby-bundler/master/manifests/3bf5c8efcd276bf6133ccb787e54b7020a00b99c
{  "schemaVersion": 1,  "name": "secure-group/tests/ruby-bundler/master",  "tag": "3bf5c8efcd276bf6133ccb787e54b7020a00b99c",  "architecture": "amd64",  "fsLayers": [  {  "blobSum": "sha256:f9b473be28291374820c40f9359f7f1aa014babf44aadb6b3565c84ef70c6bca"  },  "..."
Get image layers 
curl --cacert registry_host.crt \  -H "Authorization: Bearer $GITLAB_REGISTRY_JWT" \  -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' \  gdk.test:5100/v2/secure-group/tests/ruby-bundler/master/manifests/3bf5c8efcd276bf6133ccb787e54b7020a00b99c
{  "schemaVersion": 2,  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",  "config": {  "mediaType": "application/vnd.docker.container.image.v1+json",  "size": 7682,  "digest": "sha256:b5c7d3594559132203ca916d26e969f7bf6492d2e80d753db046dff06a5303e6"  },  "layers": [  {  "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",  "size": 45342599,  "digest": "sha256:e79bb959ec00faf01da52437df4fad4537ec669f60455a38ad583ec2b8f00498"  },  "..."
Get content of image layer 
curl --cacert registry_host.crt -H "Authorization: Bearer $GITLAB_REGISTRY_JWT" \  gdk.test:5100/v2/secure-group/tests/ruby-bundler/master/blobs/sha256:e79bb959ec00faf01da52437df4fad4537ec669f60455a38ad583ec2b8f00498

Using a custom Docker image as the main pipeline build image

It’s possible to use the local GitLab container registry as the source of the build image in pipelines.

  1. Create a new project called custom-docker-image with the following Dockerfile:

    FROM alpine RUN apk add --no-cache --update curl

  2. Build and tag an image from within the same directory as the Dockerfile for the project.

    docker build -t gdk.test:5100/custom-docker-image .

  3. Push the image to the registry. (See Configuring the GitLab Docker runner to automatically pull images for the preferred method which doesn’t require you to constantly push the image after each change.)

    docker push gdk.test:5100/custom-docker-image

    You should follow the directions given in the Configuring the GitLab Docker runner to automatically pull images section to avoid pushing images altogether.

  4. Create a .gitlab-ci.yml and add it to the Git repository for the project. Configure the image directive in the .gitlab-ci.yml file to reference the custom-docker-image which was tagged and pushed in previous steps:

    image: gdk.test:5100/custom-docker-image  stages:  - test  custom_docker_image_job:  allow_failure: false  script:  - curl -I httpstat.us/201

  5. The CI job should now pass and execute the curl command which we previously added to our base image:

    # CI job log output curl -I httpstat.us/201  HTTP/1.1 201 Created

Configuring the GitLab Docker runner to automatically pull images

In order to avoid having to push the Docker image after every change, it’s possible to configure the GitLab Runner to automatically pull the image if it isn’t present. This can be done by setting pull_policy = "if-not-present" in the Runner’s config.

# ~/.gitlab-runner/config.toml  [[runners]]  name = "docker-executor"  url = "http://gdk.test:3000/"  token = "<my-token>"  executor = "docker"  [runners.custom_build_dir]  [runners.docker]  image = "ruby:2.6.3"  privileged = true  # When the if-not-present pull policy is used, the Runner will first check if the image is present locally.  # If it is, then the local version of image will be used. Otherwise, the Runner will try to pull the image.  pull_policy = "if-not-present"

Building and pushing images to your local GitLab container registry in a build step

It’s sometimes necessary to use the local GitLab container registry in a pipeline. For example, the container scanning feature requires a build step that builds and pushes a Docker image to the registry before it can analyze the image.

To add a custom build step as part of a pipeline for use in later jobs such as container scanning, add the following to your .gitlab-yml.ci:

image: docker:stable  services:  - name: docker:stable-dind  command: ["--insecure-registry=gdk.test:5100"] # Only required if the registry is insecure  stages:  - build  build:  stage: build  variables:  DOCKER_TLS_CERTDIR: ""  script:  - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"  - docker pull $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG:$CI_COMMIT_SHA || true  - docker build -t $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG:$CI_COMMIT_SHA .  - docker push $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG:$CI_COMMIT_SHA

To verify that the build stage has successfully pushed an image to your local GitLab container registry, follow the instructions in the section List tags for a specific image.

Some notes about the above .gitlab-yml.ci configuration file:

  • The variable DOCKER_TLS_CERTDIR: "" is required in the build stage because of a breaking change introduced by Docker 19.03, described in this blog post.
  • It’s only necessary to set --insecure-registry=gdk.test:5100 for the docker:stable-dind if you have not set up a trusted self-signed registry.

Pushing multi-arch images to local GitLab container registry

  1. Install buildx for Docker.

  2. Optional. If you are using Colima, link the Colima socket to the default socket path:

    sudo ln -sf $HOME/.colima/default/docker.sock /var/run/docker.sock

    Note

    Run colima start and create this symlink whenever you restart your computer.

  3. If your local registry does not use HTTPS, add the following to ~/.docker/buildx/buildkitd.default.toml. Create the file if it doesn’t exist:

    [registry."registry.test:5100"]  http = true  insecure = true

  4. Create a new builder for Docker that uses the ~/.docker/buildx/buildkitd.default.toml configuration file from step 3 above:

    docker buildx create --name multi-arch-builder --config ~/.docker/buildx/buildkitd.default.toml

  5. Run the following commands:

    1. Pull the multi-arch image:

      docker pull alpine:latest

    2. Tag the multi-arch image so you can push it to your local registry:

      docker tag alpine:latest registry.test:5100/path/to/project:platform-specific

    3. Push a single-architecture image from the multi-arch image that matches your host machine architecture:

      docker push registry.test:5100/path/to/project:platform-specific

    4. Push a multi-arch image to the local registry:

      docker buildx imagetools create --builder=multi-arch-builder --tag registry.test:5100/path/to/project:multi-arch alpine:latest

Running container scanning on a local Docker image created by a build step in your pipeline

It’s possible to use a build step to create a custom Docker image and then execute a container scan against this newly built Docker image. This can be achieved by using the following .gitlab-ci.yml:

include:  template: Container-Scanning.gitlab-ci.yml  image: docker:stable  services:  - name: docker:stable-dind  command: ["--insecure-registry=gdk.test:5100"] # Only required if the registry is insecure  stages:  - build  - test  build:  stage: build  variables:  DOCKER_TLS_CERTDIR: ""  script:  - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"  - docker pull $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG:$CI_COMMIT_SHA || true  - docker build -t $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG:$CI_COMMIT_SHA .  - docker push $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG:$CI_COMMIT_SHA  container_scanning:  variables:  CS_REGISTRY_INSECURE: "true" # see note below for discussion

Note

The contents of the above .gitlab-ci.yml file differs depending on how the container registry has been configured:

  1. When the local container registry is insecure because registry.self_signed: false has been configured, the above .gitlab-ci.yml file can be used.

    It’s necessary to set CS_REGISTRY_INSECURE: "true" in the container_scanning job for the GitLab Container Scanning tool ( gcs) to fetch the image from our registry using HTTPS, meanwhile our registry is running insecurely over HTTP. Setting the CS_REGISTRY_INSECURE as documented in the Available CI/CD variables doc, forces gcs to use HTTP when fetching the container image from our insecure registry.

  2. When the registry is secure because registry.self_signed: true has been configured, but we haven’t referenced the self-signed certificate, then the following services and container_scanning sections of the above .gitlab-ci.yml must be used (the rest of the file has been omitted for brevity):

    services:  - docker:stable-dind  container_scanning:  variables:  CS_DOCKER_INSECURE: "true"

    Since the local container registry is now running securely over an HTTPS connection, we no longer need to use CS_REGISTRY_INSECURE: "true". However, we need to set the CS_DOCKER_INSECURE: "true" option to instruct gcs to accept a self-signed certificate.

  3. When the registry is secure because registry.self_signed: true has been configured, and we reference the self-signed certificate, then the following services and container_scanning sections of the above .gitlab-ci.yml must be used (the rest of the file has been omitted for brevity):

    services:  - docker:stable-dind  container_scanning:  variables:  ADDITIONAL_CA_CERT_BUNDLE: "-----BEGIN CERTIFICATE----- certificate-goes-here -----END CERTIFICATE-----"

    By configuring the ADDITIONAL_CA_CERT_BUNDLE, this instructs gcs to use the provided certificate when accessing the local container registry. Normally, the ADDITIONAL_CA_CERT_BUNDLE would be configured in the UI, but it’s displayed here in the .gitlab-ci.yml for demonstration purposes.

Switching Between docker-desktop-on-mac and docker-machine

To determine if you’re using docker-machine, execute the following command:

export | grep -i docker  DOCKER_CERT_PATH=~/.docker/machine/machines/default DOCKER_HOST=tcp://192.168.99.100:2376 DOCKER_MACHINE_NAME=default DOCKER_TLS_VERIFY=1

If a list of environment variables are returned as above, this means that you’re currently using docker-machine and any docker commands are routed to the virtual machine controlled by docker-machine.

To switch from docker-machine to docker-desktop-for-mac, simply unset the above environment variables:

unset DOCKER_CERT_PATH DOCKER_HOST DOCKER_MACHINE_NAME DOCKER_TLS_VERIFY

Using a Development Version of the Container Registry

To test development versions of the container registry against GDK:

  1. Within the container registry project root, create a branch with your changes, for example a branch called registry_dev.

  2. Under the registry section in your gdk.yml file, make sure that version is set to the branch name (e.g. registry_dev). You can do this by running gdk config set registry.version registry_dev.

  3. Reconfigure and Restart GDK:

    gdk reconfigure gdk restart

  4. Inspect the logs to confirm that the development version of the registry is running locally:

    gdk tail registry

Verify the logs have a field: "version":"registry-dev":

2024-09-26_21:36:36.51052 registry : {"go_version":"go1.22.0","instance_id":"2b3c8aba-4214-4f97-988b-21b6043e08be","level":"info","msg":"listening on [::]:5100","time":"2024-09-26T15:36:36.510-06:00","version":"registry-dev"}

Use local GitLab container registry with AutoDevops pipelines

When testing AutoDevops pipelines with a local registry, you can receive errors in the build step:

  • If a registry with self-signed certificate is used:

    $ /build/build.sh Logging to GitLab Container Registry with CI credentials... Error response from daemon: Get https://gdk.test:5100/v2/: x509: certificate signed by unknown authority ERROR: Job failed: command terminated with exit code 1

  • If a registry with insecure registry is used:

    $ /build/build.sh Logging to GitLab Container Registry with CI credentials... Error response from daemon: Get https://gdk.test:5100/v2/: http: server gave HTTP response to HTTPS client ERROR: Job failed: command terminated with exit code 1

To fix such issues, you can customize your build job as a part of an AutoDevOps pipeline, by adding the following to your .gitlab-ci.yml:

include:  - template: Auto-DevOps.gitlab-ci.yml  build:  services:  - name: docker:stable-dind  # Only required if the registry is insecure or used self signed certificate  command: ["--insecure-registry=gdk.test:5100"]

And for example, if you have minikube as a Kubernetes runner and you configured a self-signed registry, you can add a generated certificate to Docker inside of minikube:

  1. Run the following on your GDK instance:

    $ cat ~/.docker/certs.d/gdk.test\:5100/ca.crt -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----

  2. Copy this certificate to minikube:

    $ minikube ssh $ sudo mkdir -p /etc/docker/certs.d/gdk.test\:5100 $ sudo tee /etc/docker/certs.d/gdk.test\:5100/ca.crt > /dev/null <<EOT -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- EOT $ sudo systemctl restart docker $ logout

Or if you are using insecure registry, you can run minikube with command like:

minikube start --insecure-registry="gdk.test:5100"

Then the AutoDevOps pipeline should be able to build images and run them inside of Kubernetes.

Notifications

Some GitLab features, such as calculating the storage usage of the Container Registry images, requires the Container Registry Notifications to be enabled. When enabled, upon different events, such as pushing a container image, the Container Registry sends a notification to the Rails backend.

To enable Container Registry notifications, update the gdk.yml file as follows:

registry:  # ... other options  notifications_enabled: true

Then run gdk reconfigure.

Now, pushing an image to the Container Registry triggers a request to the /api/v4/container_registry_event/events route in the Rails backend.

Metadata Database

The Container Registry uses a PostgreSQL database to enable features like online garbage collection.

To use the Container Registry metadata database, you can either:

  • Import your existing container repositories from the older registry.
  • Use a new registry with no prior repositories.

Before you begin

  • After you enable the database, you must continue to use it. The database becomes the source of the registry metadata. If you disable the database, the registry loses visibility of all images written to it when the database was active.

  • Do not manually run offline garbage collection after the import step is complete. This action deletes data because it is not compatible with the registries using the metadata database.

  • Before you import data to the new registry, back up important data in your current registry.

Use a new registry

To use the Container Registry metadata database with a new registry:

  1. In gdk.yml, add:

    registry:  # ... other options  database:  enabled: true

  2. Reconfigure GDK by using the gdk reconfigure command.

  3. Restart the registry using the gdk restart registry command.

Use existing registries

To import existing repositories into the new metadata database registry:

  1. Stop the existing registry if it is running by using the gdk stop registry command.

  2. Update the gdk.yml file:

    registry:  # ... other options  read_only_maintenance_enabled: true  database:  enabled: false

  3. Run the following commands:

    gdk reconfigure gdk import-registry-data

    Wait until the registry import complete log appears.

  4. Update gdk.yml again:

    registry:  # ... other options  # read_only_maintenance_enabled: false (remove or set to false )  database:  enabled: true

  5. Reconfigure GDK by using the gdk reconfigure command.

  6. Restart the registry using the gdk restart registry command.

Reset registry database

To clean up existing Registry PostgreSQL data and reinstate a fresh registry database:

  1. Run the following command:

    gdk reset-registry-data

Troubleshooting

Missing container repositories in the UI

The container registry UI may only show one repository even after pushing two or more repositories. This may happen if authentication between the registry and GitLab is disabled (auth_enabled: false). To enable authentication follow these steps:

  1. Under the registry section in your gdk.yml file, make sure that auth_enabled is set to true:

    registry:  auth_enabled: true

  2. Run gdk reconfigure.

  3. Run gdk restart.

  4. Navigate to your project’s Container Registry page and verify more images show up in the UI.

denied: access forbidden when trying to access registry

Try running docker login registry.test:5100 -u gitlab-token -p "$GITLAB_TOKEN" where GITLAB_TOKEN is an access token for the GDK instance.

Last updated on
Configure Siphon to run in GDKContinuous Vulnerability Scanning