GitLab Geo
This document instructs you to set up GitLab Geo using GDK.
Geo allows you to replicate a whole GitLab instance. Customers use this for Disaster Recovery, as well as to offload read-only requests to secondary instances. For more information, see the Geo solutions page or the Geo documentation.
Easy installation
How to install 2 GDKs and configure Geo
The installation script:
- Clones the GDK project into a new
gdk
directory in the current working directory. - Installs the required software versions.
- Runs
gdk install
. - Runs
gdk start
. - Adds your license.
- Clones the GDK project into a new
gdk2
directory in the current working directory. - Runs
./support/geo-add-secondary
(see below for a description of this script).
Follow the dependency installation instructions.
Get a GitLab Premium or Ultimate license key or subscription. See Working on GitLab EE (developer licenses).
Set the
GITLAB_ACTIVATION_CODE
orGITLAB_LICENSE_FILE
environment variable in your shell. See Activate GitLab EE with a license file or keyIf you have a file on disk, then run:
export GITLAB_ACTIVATION_CODE=your_activation_code
Or, if you have plaintext, then run:
export GITLAB_LICENSE_FILE="/path/to/license/file"
Run the installation script:
curl "https://gitlab.com/gitlab-org/gitlab-development-kit/-/raw/main/support/geo-install" | bash
Or, if you want to name the GDK directories, then run:
curl "https://gitlab.com/gitlab-org/gitlab-development-kit/-/raw/main/support/geo-install" | bash -s name-of-primary name-of-secondary
To check if it is working, visit the unified URL at http://127.0.0.1:3001
and sign in. Your requests are always served by the secondary site (as if Geo-location based DNS is set up and you are located near the secondary site). It should behave no differently than the primary site. That is the goal anyway.
If needed, you can visit the primary directly at http://127.0.0.1:3000
but this would be considered a workaround, and you may notice quirks. For example, absolute URLs rendered on the page use the unified URL. You may also occasionally get redirected to the unified URL.
To see if you are able to run tests, you can run a simple spec like bin/rspec ee/spec/lib/gitlab/geo/logger_spec.rb
from the gitlab
directory in the primary gdk
directory.
Advanced Installation
Please visit GitLab Geo - Advanced Installation.
Running tests
On a primary
To run secondary-site tests on your primary GDK, you must configure the Geo tracking test database connection in the primary GDK by running:
gdk config set geo.experimental.allow_secondary_tests_in_primary true gdk reconfigure
GDK should then automatically create the test Geo tracking database, and you should be able to run tests which require this database.
If you want to manually run a Rails DB task on the test Geo tracking database in your primary GDK, then you must specify RAILS_ENV=test
, because a Geo tracking database connection is configured in database.yml
in test
but not in development
. For example, RAILS_ENV=test bin/rails db:test:prepare:geo
works, but bin/rails db:test:prepare:geo
fails with Don't know how to build task 'db:test:prepare:geo'
, since RAILS_ENV=development
is implied.
You can run the following to set up the Geo tracking database on your primary for testing:
RAILS_ENV=test bin/rails db:create:geo RAILS_ENV=test bin/rails db:migrate:geo
Confirm running secondary tests on the primary by running secondary specific tests, for example:
rspec ee/spec/requests/api/graphql/geo/registries_spec.rb
On a secondary
You should be able to run tests on GDKs that are configured as secondary sites, with no manual or special configuration.
For reference, when a GDK is configured as a Geo secondary site, the test environment database connections for the main
and ci
databases are automatically configured to use the tracking database in database.yml
. This is done so that tests can run on the secondary site, because otherwise the tests would fail while inserting or updating data into the read-only main
or ci
databases.
SSH cloning
If you used the Easy installation, then your primary site’s SSH service is disabled, and your secondary site’s SSH service is enabled. The listen port is unchanged. This simulates having a unified URL for SSH which happens to always route to the secondary site. With this setup, you can already observe Geo-specific behavior. For example, when you do a Git push, you will see This request to a Geo secondary node will be forwarded to the Geo primary node
.
You can enable the primary site’s SSH service, but you will need to specify non-default ports so they don’t conflict with the secondary site’s ports. For example, in your primary site’s gdk.yml
:
sshd: enabled: true listen_port: 2223 web_listen: localhost:9123 # the default is 9122
Or vice versa, you can specify non-default ports for your secondary site.
Note that the Git clone over SSH URL found in project show pages will always display the primary site’s Git SSH URL, even if the primary site’s SSH service is disabled. There is an issue to improve this behavior.
For more information, see SSH.
Updating GDKs
- Use
gdk update
on the primary GDK. - Use
gdk update
on the secondary GDK.
Upgrading PostgreSQL
Upgrading to a newer Postgres version is not automated in GDK with Geo.
It should be possible to manually accomplish an upgrade, but if you are not generally familiar with the process, it is recommended to set up your GDKs from scratch.
Troubleshooting
postgresql-geo/data
exists but is not empty
If you see this error during setup because you have already run make geo-secondary-setup
once:
initdb: directory "postgresql-geo/data" exists but is not empty If you want to create a new database system, either remove or empty the directory "postgresql-geo/data" or run initdb with an argument other than "postgresql-geo/data". make: *** [postgresql/geo] Error 1
Then you may delete or move that data in order to run make geo-secondary-setup
again.
mv postgresql-geo/data postgresql-geo/data.backup
GDK update command error on secondaries
You see the following error after running gdk update
on your local Geo secondary. It is ok to ignore. Your local Geo secondary does not have or need a test DB, and this error occurs on the very last step of gdk update
.
cd /Users/foo/Developer/gdk-geo/gitlab && \ bundle exec rake db:migrate db:test:prepare rake aborted! ActiveRecord::StatementInvalid: PG::ReadOnlySqlTransaction: ERROR: cannot execute DROP DATABASE in a read-only transaction : DROP DATABASE IF EXISTS "gitlabhq_test" /Users/foo/.rbenv/versions/2.6.3/bin/bundle:23:in `load' /Users/foo/.rbenv/versions/2.6.3/bin/bundle:23:in `<main>' Caused by: PG::ReadOnlySqlTransaction: ERROR: cannot execute DROP DATABASE in a read-only transaction /Users/foo/.rbenv/versions/2.6.3/bin/bundle:23:in `load' /Users/foo/.rbenv/versions/2.6.3/bin/bundle:23:in `<main>' Tasks: TOP => db:test:load => db:test:purge (See full trace by running task with --trace) make: *** [gitlab-update] Error 1
Enabling Docker Registry replication
For information on enabling Docker Registry replication in GDK, see Docker Registry replication.