Troubleshooting GDK

Many problems caused by switching branches can be resolved by updating Node.js modules, Ruby gems, and performing database migrations.

Before attempting the specific troubleshooting steps documented below, try running the following command first. It performs a gdk update but for your current branch.

gdk switch HEAD

For more information about how to update GDK, see Update GDK.

Get help when troubleshooting

In case you run into an issue which turns difficult to resolve by yourself, you can use gdk report. This will gather data from your system including installed dependencies and their versions, your OS, your architecture, and telemetry data. After gathering this data your browser should automatically open a new tab where you can paste the report and create a new issue. Please provide a meaningful title, a meaningful description of the issue you are facing, and review the information you are about to report by creating the issue.
Each report will have its unique identifier like 5e3ba684-af38-410c-8c07-f4978cc8bed7.

```
gdk report
```
Wait for the command to complete, which will open your browser for you
Describe the problem you are facing in the issue description
Paste the report inside the issue description (it should already be in your clipboard)
Replace ENTER A TITLE FOR YOUR REPORT in the issue title
Review the contents of your report
Make sure the issue is marked as confidential
Submit the issue
Reach out for help using the unique identifier or the link to the issue

Specific problems

GDK troubleshooting information is available for the following:

If you can’t solve your problem, or if you have a problem in another area, open an issue on the GDK issue tracker.

Installing the GDK

`icuc4` setup failed during `make .gitlab-translations` in GDK installation

If you encounter an icuc4 setup error during the make .gitlab-translations step of GDK installation, try the following steps:

Upgrade Xcode to the latest version.
Run brew cleanup to clean up any unnecessary files.
Run brew upgrade to upgrade outdated packages.
Open a new terminal window and continue with the installation of GDK.

PostgreSQL socket creation failed due to GDK directory path being too long

If you encounter a PostgreSQL socket creation failure, it might be because Gitaly is unable to create a Unix socket due to your GDK directory path exceeding the supported character limit.

To fix this issue, try moving the GDK directory to a location with a shorter directory path, then continue with your GDK setup.

Error: Ruby gem extensions fail to compile `unknown type name bool`

You might get an error that states ruby/internal/special_consts.h:137:15: error: unknown type name ‘bool’.

This issue occurs when the GCC compiler fails to compile some native Ruby gem extensions (for example, gem io-event). The failure occurs because GCC recently changed the default standard from gnu17 to c23. Ruby built from source by using a GCC version after this change (for example, Fedora 42) is compiled without the HAVE_STDBOOLH config option which is required to compile some extensions.

The workaround is to build Ruby with gnu17 standard before running GDK: CC="gcc -std=gnu17" mise install ruby@3.4.2

Git

The following are possible solutions to problems you might encounter with Git and GDK.

‘Invalid reference name’ when creating a new tag

Make sure that git is configured correctly on your development machine (where GDK runs).

git checkout -b can-I-commit git commit --allow-empty -m 'I can commit'

`fatal: not a git repository`

If any gdk command gives you the following error:

fatal: not a git repository (or any of the parent directories): .git

Make sure you don’t have gdk aliased in your shell. For example the Git module in prezto has an alias for gdk that lists killed files.

Blobless clones

After you’ve installed GDK, it might have made a blobless partial clone of the gitlab repository. This can cause git blame to be slow or even fail:

$ git blame -- db/structure.sql  remote: Enumerating objects: 1, done. remote: Counting objects: 100% (1/1), done. remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0) Receiving objects: 100% (1/1), 219.47 KiB | 1.37 MiB/s, done. remote: Enumerating objects: 1, done. remote: Counting objects: 100% (1/1), done. remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0) Receiving objects: 100% (1/1), 219.32 KiB | 1.39 MiB/s, done. remote: Enumerating objects: 1, done. remote: Counting objects: 100% (1/1), done. remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0) Receiving objects: 100% (1/1), 219.38 KiB | 1.36 MiB/s, done. remote: Enumerating objects: 1, done. remote: Counting objects: 100% (1/1), done. ...

Here you can see that Git downloads objects one by one.

GDK created a partial clone because it’s a lot faster, but it comes with the downside of not having all objects through history of the repository.

git blame <file> fatal: You are attempting to fetch XYZ, which is in the commit graph file but not in the object database

To check if your clone is blobless, run:

$ git config remote.origin.partialclonefilter blob:none

If your clone is a blobless partial clone, this returns blob:none. If you have a blobless clone, you can backfill your repo by running:

# This requires Git version 2.49 or higher. $ git backfill

Puma

The following are possible solutions to problems you might encounter with Puma and GDK.

An error has occurred and reported in the system’s low-level handler

If you receive this error message:

An error has occurred and reported in the system's low-level error handler.

This is Puma catching an error that has slipped through from Rails. Here are some suggestions that may assist:

Completely stop your GDK to rule out an out-of-date process running:
```
gdk stop && gdk kill && gdk start
```
Update your GDK as the problem may have already been fixed:
```
gdk update && gdk restart
```
If the problem continues to persist, please raise a GDK Issue ensuring all detail requested in the Issue template is provided.

Puma timeout

Browser shows EOF. Logs show a timeout:

error: GET "/users/sign_in": badgateway: failed after 62s: EOF

Depending on the performance of your development environment, Puma may time out. Increase the timeout as a workaround.

Use environment variables to override the default timeout:

Variable	Type	Description
`GITLAB_RAILS_RACK_TIMEOUT`	integer	Sets `service_timeout`
`GITLAB_RAILS_WAIT_TIMEOUT`	integer	Sets `wait_timeout`

Sidekiq Cluster

GDK uses Sidekiq Cluster (running a single Sidekiq process) by default instead bundle exec sidekiq directly, which is a step towards making development a bit more like production.

Technically, running Sidekiq Cluster with a single Sidekiq process matches the same behavior of running Sidekiq directly, but eventually problems may arise.

If you’re experiencing performance issues or jobs not being picked up, try disabling Sidekiq Cluster by:

Stopping all running processes with gdk stop.
Opening the $GDKROOT/Procfile file.
Removing the SIDEKIQ_WORKERS environment variable from rails-background-jobs.
Starting GDK again with gdk start.

When doing so, please create an issue describing what happened.

Jaeger

If you’re seeing errors such as:

ERROR -- : Failure while sending a batch of spans: Failed to open TCP connection to localhost:14268 (Connection refused - connect(2) for "localhost" port 14268)

This is most likely because Jaeger is not configured in your $GDKROOT/Procfile. The easiest way to fix this is by re-creating your Procfile and then running a gdk reconfigure:

mv Procfile Procfile.old; make Procfile
gdk reconfigure

For more information about Jaeger, visit the distributed tracing GitLab developer documentation.

Gitaly

The following are possible solutions to problems you might encounter with Gitaly and GDK.

`config.toml: no such file or directory`

If you see errors such as:

07:23:16 gitaly.1 | time="2019-05-17T07:23:16-05:00" level=fatal msg="load config" config_path=<path-to-gdk>/gitaly/gitaly.config.toml error="open <path-to-gdk>/gitaly/gitaly.config.toml: no such file or directory"

Somehow, gitaly/gitaly.config.toml is missing. You can re-create this file by running the following in your GDK directory:

make gitaly-setup

Git fails to compile within Gitaly project

If you see the following error when running a gdk update:

ld: library not found for -lgit2

A known fix is to clean your Go cache by running the following from the GDK’s root directory:

go clean -cache rm -rf gitaly

Now rerun gdk update.

`libegit2.a Error 129`

If you see the following error when running a gdk install or a gdk update:

make[1]: Entering directory '/home/user/gitlab-development-kit/gitaly; error: unknown option `initial-branch=master' make[1]: *** [Makefile:424: /home/user/gitlab-development-kit/gitaly/_build/deps/libgit2/install/lib/libgit2.a] Error 129

Check which version of Git you’re running with git --version, and compare it against GitLab requirements. You might be running an unsupported version.

If the supported version is not available for you from pre-compiled packages, try following the instructions for:

If that doesn’t give you the supported version, you might have to compile Git from source.

Elasticsearch

Running a spec locally may give you something like the following:

rake aborted! Gitlab::TaskFailedError: # pkg-config --cflags -- icu-i18n icu-i18n Package icu-i18n was not found in the pkg-config search path. Perhaps you should add the directory containing `icu-i18n.pc' to the PKG_CONFIG_PATH environment variable No package 'icu-i18n' found Package icu-i18n was not found in the pkg-config search path. Perhaps you should add the directory containing `icu-i18n.pc' to the PKG_CONFIG_PATH environment variable No package 'icu-i18n' found pkg-config: exit status 1 make: *** [build] Error 2

This indicates that Go is trying to link (unsuccessfully) to brew’s icu4c.

Find the directory where icu-i18n.pc resides:

On macOS, using Homebrew, it is generally in /usr/local/opt/icu4c/lib/pkgconfig or /opt/homebrew/opt/icu4c/lib/pkgconfig
On Ubuntu/Debian it might be in /usr/lib/x86_64-linux-gnu/pkgconfig
On Fedora it is expected to be in /usr/lib64/pkgconfig

You need to add that directory to the PKG_CONFIG_PATH environment variable.

To fix this now, run the following on the command line:

export PKG_CONFIG_PATH="/usr/local/opt/icu4c/lib/pkgconfig:$PKG_CONFIG_PATH"

To fix this for the future, add the line above to ~/.bash_profile or ~/.zshrc.

Elasticsearch indexer looks for the wrong version of icu4c

You might get the following error when updating the application:

# gitlab.com/gitlab-org/gitlab-elasticsearch-indexer /usr/local/Cellar/go/1.14.2_1/libexec/pkg/tool/darwin_amd64/link: running clang failed: exit status 1 ld: warning: directory not found for option '-L/usr/local/Cellar/icu4c/64.2/lib' ld: library not found for -licui18n clang: error: linker command failed with exit code 1 (use -v to see invocation)  make[1]: *** [build] Error 2 make: *** [gitlab-elasticsearch-indexer/bin/gitlab-elasticsearch-indexer] Error 2

This means Go is trying to link to brew’s version of icu4c (64.2 in the example), and failing. This can happen when icu4c is not pinned and got updated. Verify the version with:

$ ls /usr/local/Cellar/icu4c 66.1

Clean Go’s cache to fix this error. From the GDK’s root directory:

cd gitlab-elasticsearch-indexer/ go clean -cache

Homebrew

Most brew problems can be figured out by running:

brew doctor

However, older installations may have significant cruft leftover from previous installations and updates. To manually remove outdated downloads for all formulae, casks, and stale lock files, run:

brew cleanup

For more information on uninstalling old versions of a formula, see the Homebrew FAQ. For additional troubleshooting information, see the Homebrew Common Issues page.

Live reloading

If you previously compiled production assets with bundle exec rake gitlab:assets:compile, the GDK serves the assets from the public/assets/ directory, which means that changing SCSS files doesn’t have any effect in development until you recompile the assets manually.

To re-enable live reloading of CSS in development, remove the public/assets/ directory and restart GDK.

Praefect

get shard for “default”: primary is not healthy

From the GDK’s root directory:

cd gitaly/ruby bundle install

You may need to run a gdk restart for the changes to take effect.

`/home/user/gitlab-development-kit/gitaly/_build/bin/praefect`: No such file or directory

You might encounter the following error while running Gitaly database migrations:

support/migrate-praefect: line 4: /home/user/gitlab-development-kit/gitaly/_build/bin/praefect: No such file or directory migrate failed make: *** [_postgresql-seed-praefect] Error 1

This means /gitaly/_build/bin/praefect is missing. To re-create this executable file, run the following in your GDK directory:

make gitaly-update

Updating the GDK

Run `gdk doctor`

As a general rule, if you encounter errors when you run gdk update, you should run gdk doctor and follow the suggestions it returns. They might resolve your issue.

You may use gdk doctor --correct to autocorrect trivial issues.

Blank page after update

If you encounter a blank page after performing an update, this is because of a hanging process. To solve the issue, follow these steps:

Identify the hanging process ID (PID) bound to port 3000:
```
lsof -i tcp:3000
```
Using the returned PID, kill the hanging process:
```
kill -9 <process id>
```
Restart GDK services:
```
gdk restart
```

If you’ve changed your default port, use the following command to find your custom port number: gdk config get port

Error due to `Net::OpenTimeout: Failed to open TCP connection to rubygems.org:443`

When you run gdk update you might get an error like the following:

ERROR: While executing gem ... (Gem::RemoteFetcher::FetchError)

This indicates that bundle failed to connect to the rubygems.org server.

If you are connected to the network and other network activities are working (i.e. ping gitlab.com), then this normally indicates an outage of rubygems.org. You can try manually running bundle update in the GDK root folder, and if it fails with a similar network error, you know this is the cause.

However, in some cases, even though bundle update is otherwise working successfully, you might get an error like the following:

INFO: Installing gitlab-development-kit Ruby gem.. ERROR: While executing gem ... (Gem::RemoteFetcher::FetchError)  Net::OpenTimeout: Failed to open TCP connection to rubygems.org:443 (execution expired) (https://rubygems.org/specs.4.8.gz)

This can happen if the IPv6 access to rubygems.org is having an outage, but IPv4 access is still working.

Using this comment as an example, you can add :ipv4_fallback_enabled: true to your ~/.gemrc to work around this until this rubygems pull request gets merged.

If that doens’t work for some reason, you can alternately go into your operating system network settings and disable IPv6 for your network adapter. Refer to your operating system documentation for detailed instructions.

Starting the GDK

Unable to start as CE (FOSS_ONLY)

If your GDK enters an infinite loop when running FOSS_ONLY=1 gdk start, try to remove - { name: 'group_saml' } option from your config/gitlab.yml. The favicon should be blue when in CE and green when in EE.

Error due to `ActionController::InvalidAuthenticityToken`

If you encounter an ActionController::InvalidAuthenticityToken error when starting GDK, try the following steps:

Hard refresh your browser to clear the cache. For more information, see How to hard refresh your browser.
Stop all GDK processes using gdk kill, then restart GDK with gdk start.
Delete all browser cache, cookies, local storage, and other related data for the relevant hostname.

Error starting `timeout: down: /Users/foo/gdk/services/rails-background-jobs: 0s, want up`

Check if a background job process is unexpectedly running:

Search for processes related to background jobs: ps aux | grep rails-background-jobs
If runit is currently trying to start rails-background-jobs then you may find: runsvdir -P /Users/foo/gdk/services log: atal: unable to lock supervise/lock: temporary failure\012runsv rails-background-jobs: fatal: unable to lock supervise/lock: temporary failure\012runsv rails-background-jobs: fatal: unable to lock supervise/lock: temporary failure\012runsv rails-background-jobs: fatal: unable to lock supervise/lock: temporary failure\012runsv rails-background-jobs: fatal: unable to lock supervise/lock: temporary failure\012
Stop GDK services: gdk stop
If ps aux | grep rails-background-jobs shows that runsv rails-background-jobs is still running, then it is preventing rails-background-jobs from starting
Kill the process by following Stopping and restarting the GDK

If the above is not the problem, then confirm you can run rails-background-jobs manually: RAILS_ENV=development ./bin/background_jobs start_foreground.

Error `Listen::Error::INotifyMaxWatchesExceeded`

Depending on your system configuration, you can receive errors about an excess or maximum number of inotify watchers. This error can appear when running gdk tail.

To solve this problem, increase the maximum allowed inotify watchers. For more information, see relevant documentation for the listen gem.

Page loading issue due to ad blocker

Some ad blockers, such as uBlock Origin, might prevent web-vitals.js from loading properly. This results in errors such as:

Failed to load 'pages.projects.jobs.index.js' entrypoint

To resolve this issue, disable your ad blocker and reload the application.

Stopping and restarting the GDK

Sometimes the GDK will fail to stop or restart. This is sometimes caused by processes not shutting down gracefully and can prevent subsequent attempts to stop/start the GDK.

You might see something like the following after running gdk stop:

kill: run: ./services/rails-background-jobs: (pid 89668) 98s, normally down, want down

To kill off the rogue processes, run gdk kill.

Unable to log in as root

If all the services are running after you run gdk install, but you cannot log in as root, you can reset the password through the Rails console.

For more information about the Ruby on Rails console in GitLab, see Rails console.

Open a new Rails console:

cd <gdk-dir>/gitlab bundle exec rails console

In the Rails console, update the password for the root user:

user = User.find_by_username('root') user.update!(password: 'newpassword')

To find the user IDs for all current users, run:

User.all

Account verification needed

Sometimes, after resetting data, or on initial setup, you may see a message like this when logging in for the first time:

Before you can run pipelines, we need to verify your account. We won't ask you for this information again. It will never be used for marketing purposes.

This is often caused by having GITLAB_SIMULATE_SAAS=1 in your env.runit file. You can set it to 0 if it’s not needed and then restart your GDK. But if you do need it to be set, such as for running the GitLab AI Gateway, you can try this from the Rails console:

Option 1: Create validation records

root = User.first Users::CreditCardValidation.create!(user: root, credit_card_validated_at: Time.zone.now) Users::PhoneNumberValidation.create!(user: root, validated_at: Time.zone.now, country: "US", phone_number: "7665554269", international_dial_code: "+1")

Option 2: Add identity verification exemption

root = User.first root.add_identity_verification_exemption("I'm local")

Orphaned `workerd` processes

Sometimes, workerd hangs and prevents GDK from working. You can try and kill all workerd processes:

killall -9 workerd

If that doesn’t help, you can try disabling gitlab-http-router.

Last updated on August 27, 2025