GitLab Agent Server (KAS) –GitLab Development Kit

GitLab Agent Server (KAS)

Run KAS from within GDK

If you wish to clone and keep an updated GitLab Agent for Kubernetes as part of your GDK, do the following:

If using MacOS, install XCode from App Store. Once installed, open it and agree to the terms and conditions. Please note, this is not the same as the XCode Command Line Tools.
Install build dependencies
The recommended way to install Bazel is to use mise. Just run mise install in the GitLab agent root directory.
Add the following settings in your gdk.yml:
```
gitlab_k8s_agent:  enabled: true
```

(Optional) To use the CI tunnel functionality, you must:

Enable NGINX in HTTPS mode. kubectl never sends credentials over a plain text connection.

Specify concrete IP addresses for kas to listen on. gdk.yml looks like this (assuming loopback alias IP was set up):

gitlab_k8s_agent:  enabled: true  agent_listen_address: 172.16.123.1:8150  k8s_api_listen_address: 172.16.123.1:8154 hostname: gdk.test port: 3443 https:  enabled: true nginx:  enabled: true  ssl:  certificate: gdk.test.pem  key: gdk.test-key.pem

Run gdk update to get kas installed as part of GDK.
Run gdk reconfigure to update various configuration files.

You can start GDK with gdk start. It prints the URL for agentk to use:

=> GitLab available at http://127.0.0.1:3000. => GitLab Agent Server (KAS) available at grpc://127.0.0.1:8150.

If you are using NGINX+HTTPS, the URL would show something like:

=> GitLab available at https://gdk.test:3443. => GitLab Agent Server (KAS) available at wss://gdk.test:3443/-/kubernetes-agent.

To verify that kas is running you can:
- Run gdk tail gitlab-k8s-agent to check the logs. You should see no errors in the logs. Empty logs are normal too.
- Run curl 127.0.0.1:8150. It should print
  Warning: Binary output can mess up your terminal. Use "--output -" to tell Warning: curl to output it to your terminal anyway, or consider "--output Warning: <FILE>" to save to a file.
  This is normal because gRPC is a binary protocol.
- If running with NGINX enabled, run using the loopback address: curl 172.16.123.1:8150. It should print
  WebSocket protocol violation: Connection header "close" does not contain Upgrade
  This is a normal response from kas for such a request because it’s expecting a WebSocket connection upgrade.
Once your GitLab Agent Server is running, you can connect to a Kubernetes cluster by installing agentk to the cluster. The kasAddress should be the GitLab Agent Server URL outputted when you ran gdk start or from the listed URLs when you run gdk status.
- To connect to a Kubernetes cluster on k3d, read the instructions below for deploying agentk with k3d

(Optional) Connecting your project to `agentk` using CI Tunnel

The GitLab Agent Server communicates with agentk through a Kubernetes proxy. You can check the proxy address by running gdk status, which will output it as one of the URLs:

=> GitLab available at https://gdk.test:3443. => GitLab Agent Server (KAS) available at wss://gdk.test:3443/-/kubernetes-agent. => Kubernetes proxy (via KAS) available at https://gdk.test:3443/-/k8s-proxy/.

The GitLab Runner must be authorized to access the https address. This is done by adding the NGINX SSL certificate to the relevant places.

Runner configuration

If your runner is configured with a docker executor, you must add your certificate to the volumes in your runner’s config.toml:

[[runners]]  name = "GDK local runner"  url = "https://gdk.test:3443"  # other config here  [runners.docker]  volumes = [  "path/to/gdk/directory/gdk.test.crt:/etc/ssl/certs/gdk.test.crt",  # other volumes here  "/certs/client",  "/cache"  ]

Alternatively, you can set the certificate-authority of agentk’s Kubernetes cluster:

Add the gdk.test certificate into your project that is using CI Tunnel. You can also use the root certificate if there is one. For example, the rootCA.pem file generated by mkcert.
1. Copy the contents of the certificate to the clipboard
2. Add a File type CI/CD variable in your project. For example: key is CA_CRT, value is content of .pem or .crt file, and type is File. Unset all flags.

In the project’s .gitlab-ci.yml, for steps that need to connect to agentk (e.g.: the deploy step), add a command to set the certificate authority of the associated Kubernetes cluster:

kubectl config set clusters.gitlab.certificate-authority $CA_CRT

This reads the certificate that you added to your project.

An example .gitlab-ci.yml, extended from this guide, looks like this:

deploy:  image:  name: bitnami/kubectl:latest  entrypoint: ['']  script:  - kubectl config get-contexts  - kubectl config use-context path/to/project:agentk-name  - kubectl config set clusters.gitlab.certificate-authority $CA_CRT  - kubectl get pods --namespace gitlab-agent

(Optional) Deploy the GitLab Agent (agentk) with k3d

Install k3d.
Create a k3d cluster:
```
k3d cluster create
```
Set up a loopback alias IP. We can use it as the listen address so that agentk can reach your local GitLab and KAS. Let’s assume this is 172.16.123.1. We recommend you also bind your hostname to this address in /etc/hosts, by adding the line 172.16.123.1 gdk.test to /etc/hosts
Then update your gdk.yml to include these global keys:
```
hostname: gdk.test listen_address: "172.16.123.1"
```
This sets the default hostname and listen address for all GDK services, including GitLab. For example, with the default ports:
- GitLab would now be available on http://gdk.test:3000.
- The registry would now be available on https://gdk.test:5100.
Run gdk reconfigure to apply the above change.
Deploy agentk:
1. Register the agent as you normally would to deploy it to any cluster. Take note of the token.
2. Install the Agent to the cluster.
  At this step, be sure to use the loopback alias as the KAS address instead of the gdk.test URL. This is necessary because k3d is running on Docker, which will not be able to resolve gdk.test.
  You can do this by changing the config.kasAddress in the Helm installation command:
  helm upgrade --install agentk-test gitlab/gitlab-agent \ --namespace gitlab-agent \ --create-namespace \ --set image.tag=v15.4.0 \ --set config.token=<token generated from the previous step> \ --set config.kasAddress=grpc://172.16.123.1:8150

(Optional) Run `agentk` from source using `support/agentk`

To increase development velocity, GDK includes a script support/agentk to run agentk with go run, skipping the build and deploy steps associated with running agentk in a cluster. The script takes a single mandatory argument, the agent token, and runs the agent in the foreground:

support/agentk TOKEN [EXTRA ARGS FOR AGENTK]

When run in this way, agentk implicitly uses the currently selected context in your kubeconfig, similar to how kubectl works.

To pick up new changes to the code, simply restart the script. If you are also making changes to KAS, consider running KAS from source as well.

When running the script, all arguments following the token get forwarded directly to agentk. For example, to use a different kubeconfig context, you can use the --context flag:

support/agentk TOKEN --context YOUR_CONTEXT

To see the list of available flags, you can run

support/agentk "" --help

(Optional) Run using Bazel instead of GDK

If you want to run GitLab Agent Server and Agent locally with Bazel instead of GDK, see the GitLab Agent documentation.

(Optional) Developing KAS observability together with GitLab Observability Stack’s development environment

To develop observability features of KAS, you can configure GDK and KAS so that traces are sent to a development instance of GitLab Observability Stack (GOS).

To configure GDK and KAS:

Launch a devvm - integrated development environment for GOS.
Once all the steps required to launch devvm are completed, note the GitLab groupID that was provisioned. You can find it in the output of the booter script - please refer to the devvm documentation. This is referred to as <gitlab groupID> in this document.
Create a GitLab Observability Stack API token.
1. In the web browser, enter https://gob.devvm/-/<gitlab groupID>/, for example https://gob.devvm/-/22/.
2. On the left sidebar, select Configuration > API Keys.
3. Select New API Key.
4. Enter a title for the key, select the User role, and then select Add.
5. Copy the created API key. You will use it in a later step.
Log into the devvm. All the following steps must be executed from the dev user inside devvm.
Store the API token in a file:
```
echo <token> > /home/dev/otel-token.txt
```
Perform steps required to run KAS from within GDK.

Configure the kubectl plugin for mise:

mise list kubectl | sort -n | head -n 1 | xargs -I{} mise use -g kubectl@{}

Fetch the CA certificate that GOB (GitLab Observability Backend) uses:

KUBECONFIG=/home/dev/kubeconfig kubectl get secret self-signed-ca-secret -o jsonpath='{ .data.ca\.crt }' | base64 -d > /home/dev/gitlab-development-kit/gob-ca.crt

Add OTel configuration to gdk.yml:

gitlab_k8s_agent:  <...>  otlp_endpoint: https://gob.devvm/v1/traces/<gitlab groupID>  otlp_token_secret_file: /home/dev/otel-token.txt  otlp_ca_certificate_file: /home/dev/gitlab-development-kit/gob-ca.crt  <...>

Uninstall the mise installation of Go to avoid issues:
```
mise uninstall golang 1.19.3
```
Sign out and sign back in to the dev account in order for changes to take effect. You should be using now the system version of Go:
```
$ which go /usr/bin/go
```
Reconfigure GDK:
```
gdk reconfigure
```

Configure GDK to use the CA certificate when deploying agentk.

export KUBECONFIG=/home/dev/kubeconfig helm repo add gitlab https://charts.gitlab.io helm repo update helm upgrade --install mynamespace gitlab/gitlab-agent \  --namespace gitlab-agent-mynamespace \  --create-namespace \  --set image.tag=v15.7.0-rc1 \  --set config.token=<...> \  --set config.kasAddress=wss://gdk.devvm:3443/-/kubernetes-agent \  --set config.caCert="$(cat /home/dev/gitlab-development-kit/gdk.devvm.pem)"

Deploy the agent by following the steps in Installing the agent for Kubernetes. To keep things simple, you should deploy the agent on the same Kubernetes cluster that GOB is running on.
Test if traces are properly received.
1. In the web browser, enter https://gob.devvm/-/<gitlab groupID>.
2. On the left sidebar, expand Explore, and select Explore > Traces. If you can see a section named gitlab-kas, then the traces are being ingested by GOB.

(Optional) Run KAS directly from source (with `go run`)

When working on KAS, it can be convenient to run directly from source instead of having to rebuild every time.

gdk config set gitlab_k8s_agent.run_from_source true gdk reconfigure

That way, changes to the code get picked up immediately every time you restart KAS:

gdk restart gitlab-k8s-agent

(Optional) Run KAS manually

When working on KAS, it can be convenient to run KAS manually instead of running it through the GDK Procfile. For example, this method can be ideal if you want to run KAS with a debugger, directly from the IDE, or run multiple KAS replicas.

gdk config set gitlab_k8s_agent.configure_only true gdk reconfigure

(Optional) Disable automatic repository updates

When working on the agent, you probably want to manage the repository on your own, to avoid having gdk update stash your changes:

gdk config set gitlab_k8s_agent.auto_update false gdk reconfigure

(Optional) Deploying a FluxCD Project to the cluster / installing the Agent with Flux

See the dedicated flux docs.

Troubleshooting

Bazel crashes

Because of a known issue, Bazel can crash if your version of Xcode is too old. To resolve this:

Update the version of Xcode you run.
Run bazel clean --expunge from the root of the gitlab-k8s-agent repository.

Bad CPU type in executable Target

See Bad CPU type in executable Target.

Last updated on June 23, 2025

Gitaly and Praefect GitLab AI Gateway