DEV Community

Ruan Bekker
Ruan Bekker

Posted on

How to Setup a Concourse CI Server

#concourse #devops #linux #tutorial

concourse

Concourse is a Pipeline Based Continuous Integration System written in Go.

What is Concourse CI:

Concourse CI is a Continuous Integration Platform. Concourse enables you to construct pipelines with yaml configuration that can consist out of 3 core concepts that compose them:

  • tasks
  • resources
  • jobs

For more information about this have a look at their docs

What will we be doing today

We will setup a Concourse Server on Ubuntu 18.04 and run the traditional Hello, World pipeline

Setup the Server:

Concourse requires PostgresSQL for it's database (at least version 9.3)

$ apt update && apt upgrade -y $ apt install postgresql postgresql-contrib -y $ systemctl enable postgresql
Enter fullscreen mode Exit fullscreen mode

Create the Database and User for Concourse on your Postgres database:

$ sudo -u postgres createuser concourse $ sudo -u postgres createdb --owner=concourse atc
Enter fullscreen mode Exit fullscreen mode

Download the Concourse Server Archive:

$ export CONCOURSE_VERSION=5.7.2 $ wget https://github.com/concourse/concourse/releases/download/v${CONCOURSE_VERSION}/concourse-${CONCOURSE_VERSION}-linux-amd64.tgz
Enter fullscreen mode Exit fullscreen mode

Extract the concourse archive:

$ tar -zxf concourse-${CONCOURSE_VERSION}-linux-amd64.tgz -C /usr/local
Enter fullscreen mode Exit fullscreen mode

Create the encryption keys:

$ mkdir /etc/concourse $ ssh-keygen -t rsa -q -N '' -f /etc/concourse/tsa_host_key $ ssh-keygen -t rsa -q -N '' -f /etc/concourse/worker_key $ ssh-keygen -t rsa -q -N '' -f /etc/concourse/session_signing_key $ cp /etc/concourse/worker_key.pub /etc/concourse/authorized_worker_keys
Enter fullscreen mode Exit fullscreen mode

Concourse Web Process Configuration:

$ EXT_IP=$(curl -s https://ip.ruan.dev) $ cat > /etc/concourse/web_environment << EOF CONCOURSE_ADD_LOCAL_USER=admin:admin CONCOURSE_SESSION_SIGNING_KEY=/etc/concourse/session_signing_key CONCOURSE_TSA_HOST_KEY=/etc/concourse/tsa_host_key CONCOURSE_TSA_AUTHORIZED_KEYS=/etc/concourse/authorized_worker_keys CONCOURSE_POSTGRES_HOST=127.0.0.1 CONCOURSE_POSTGRES_USER=concourse CONCOURSE_POSTGRES_PASSWORD=concourse CONCOURSE_POSTGRES_DATABASE=atc CONCOURSE_MAIN_TEAM_LOCAL_USER=admin CONCOURSE_EXTERNAL_URL=http://${EXT_IP}:8080 EOF
Enter fullscreen mode Exit fullscreen mode

Concourse Worker Process Configuration:

$ cat > /etc/concourse/worker_environment << EOF CONCOURSE_WORK_DIR=/var/lib/concourse CONCOURSE_TSA_HOST=127.0.0.1:2222 CONCOURSE_TSA_PUBLIC_KEY=/etc/concourse/tsa_host_key.pub CONCOURSE_TSA_WORKER_PRIVATE_KEY=/etc/concourse/worker_key EOF
Enter fullscreen mode Exit fullscreen mode

Create a Concourse user:

$ mkdir /var/lib/concourse $ sudo adduser --system --group concourse $ sudo chown -R concourse:concourse /etc/concourse /var/lib/concourse $ sudo chmod 600 /etc/concourse/*_environment
Enter fullscreen mode Exit fullscreen mode

Create SystemD Unit Files, first for the Web Service:

$ cat > /etc/systemd/system/concourse-web.service << EOF [Unit] Description=Concourse CI web process (ATC and TSA) After=postgresql.service [Service] User=concourse Restart=on-failure EnvironmentFile=/etc/concourse/web_environment ExecStart=/usr/local/concourse/bin/concourse web [Install] WantedBy=multi-user.target EOF
Enter fullscreen mode Exit fullscreen mode

Then the SystemD Unit File for the Worker Service:

$ cat > /etc/systemd/system/concourse-worker.service << EOF [Unit] Description=Concourse CI worker process After=concourse-web.service [Service] User=root Restart=on-failure EnvironmentFile=/etc/concourse/worker_environment ExecStart=/usr/local/concourse/bin/concourse worker [Install] WantedBy=multi-user.target EOF
Enter fullscreen mode Exit fullscreen mode

Create a postgres password for the concourse user:

$ cd /home/concourse/ $ sudo -u concourse psql atc atc=> ALTER USER concourse WITH PASSWORD 'concourse'; atc=> \q
Enter fullscreen mode Exit fullscreen mode

Start and Enable the Services:

$ systemctl start concourse-web concourse-worker $ systemctl enable concourse-web concourse-worker postgresql $ systemctl status concourse-web concourse-worker $ systemctl is-active concourse-worker concourse-web active active
Enter fullscreen mode Exit fullscreen mode

The listening ports should more or less look like the following:

$ netstat -tulpn Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 127.0.0.1:5432 0.0.0.0:* LISTEN 11290/postgres tcp 0 0 127.0.0.1:7776 0.0.0.0:* LISTEN 13040/concourse tcp 0 0 127.0.0.1:7777 0.0.0.0:* LISTEN 13118/gdn tcp 0 0 127.0.0.1:7787 0.0.0.0:* LISTEN 13040/concourse tcp 0 0 127.0.0.1:7788 0.0.0.0:* LISTEN 13040/concourse tcp 0 0 127.0.0.1:8079 0.0.0.0:* LISTEN 13039/concourse tcp6 0 0 :::41299 :::* LISTEN 13039/concourse tcp6 0 0 :::8888 :::* LISTEN 13040/concourse tcp6 0 0 ::1:5432 :::* LISTEN 11290/postgres tcp6 0 0 :::35323 :::* LISTEN 13039/concourse tcp6 0 0 :::2222 :::* LISTEN 13039/concourse tcp6 0 0 :::8080 :::* LISTEN 13039/concourse udp 0 0 0.0.0.0:54582 0.0.0.0:* 13118/gdn
Enter fullscreen mode Exit fullscreen mode

Client Side:

When you access the UI on the external ip on port 8080:

image

After logging in, you should see a screen with no pipelines configured:

image

We require the fly cli client which is used to interact with concourse. On your local machine, download the fly archive, extract the archive and move the fly binary to your path.

I will by using the client for Mac, for alternate versions have a look at their releases page

$ export CONCOURSE_VERSION=5.7.2 $ wget https://github.com/concourse/concourse/releases/download/v${CONCOURSE_VERSION}/fly-${CONCOURSE_VERSION}-darwin-amd64.tgz $ tar -xf fly-${CONCOURSE_VERSION}-darwin-amd64.tgz $ sudo mv fly /usr/local/bin/fly
Enter fullscreen mode Exit fullscreen mode

Next, we need to setup our Concourse Target by Authenticating against our Concourse Endpoint, lets setup our target with the name ci:

$ fly -t ci login -c http://${EXT_IP}:8080 logging in to team 'main' navigate to the following URL in your browser: http://136.244.102.215:8080/login?fly_port=49722 or enter token manually:
Enter fullscreen mode Exit fullscreen mode

Follow the URL and your target should be saved. Lets list our targets:

$ fly targets name url team expiry ci http://136.244.102.215:8080 main Wed, 18 Dec 2019 20:48:02 UTC
Enter fullscreen mode Exit fullscreen mode

Listing Registered Workers:

$ fly -t ci workers name containers platform tags team state version age demo 0 linux none none running 2.2 16m1s
Enter fullscreen mode Exit fullscreen mode

Listing Active Containers:

$ fly -t ci containers handle worker pipeline job build # build id type name attempt
Enter fullscreen mode Exit fullscreen mode

Hello World Pipeline:

Let's create a basic pipeline, that will print out Hello, World!:

Our hello-world.yml

jobs: - name: my-job plan: - task: say-hello config: platform: linux image_resource: type: docker-image source: repository: alpine tag: edge run: path: /bin/sh args: - -c - | echo "=============" echo "Hello, World!" echo "============="
Enter fullscreen mode Exit fullscreen mode

Applying the configuration to our pipeline:

$ fly -t ci set-pipeline -p yeeehaa -c hello-world.yml jobs: job my-job has been added: name: my-job plan: - task: say-hello config: platform: linux image_resource: type: docker-image source: repository: alpine tag: edge run: path: /bin/sh args: - -c - | echo "=============" echo "Hello, World!" echo "=============" apply configuration? [yN]: y pipeline created! you can view your pipeline here: http:////136.244.102.215:8080/teams/main/pipelines/yeeehaa the pipeline is currently paused. to unpause, either: - run the unpause-pipeline command - click play next to the pipeline in the web ui
Enter fullscreen mode Exit fullscreen mode

In the Web-UI we will see the pipeline that we created:

image

We can unpause the pipeline in the Web-UI, but since I like to do everything on cli as far as possible, I will unpause the pipeline via cli:

$ fly -t ci unpause-pipeline -p yeeehaa unpaused 'yeeehaa'
Enter fullscreen mode Exit fullscreen mode

Now our Pipeline is unpaused, but since we did not specify any triggers, we need to manually trigger the pipeline to run, you can either via the WebUI, select your pipeline which in this case will be named yeeehaa and then select the job, which will be my-job then hit the + sign, which will trigger the pipeline.

I will be using the cli:

$ fly -t ci trigger-job --job yeeehaa/my-job started yeeehaa/my-job #1
Enter fullscreen mode Exit fullscreen mode

Via the WebUI on http://${EXT_IP}:8080/teams/main/pipelines/yeeehaa/jobs/my-job/builds/1 you should see the Hello, World! output.

If you are running Concourse on Ubuntu 18 and you ran into this issue:

image

It's a DNS fix that you can apply by doing this:

$ sudo apt install resolvconf $ echo -e "nameserver 8.8.4.4\nnameserver 8.8.8.8" > /etc/resolvconf/resolv.conf.d/head $ sudo service resolvconf restart
Enter fullscreen mode Exit fullscreen mode

We can trigger our pipeline via the cli, we also have the option to see the output, so let's trigger it again, but this time passing the --watch flag:

$ fly -t ci trigger-job --job yeeehaa/my-job --watch started yeeehaa/my-job #5 initializing running /bin/sh -c echo "=============" echo "Hello, World!" echo "=============" ============= Hello, World! ============= succeeded
Enter fullscreen mode Exit fullscreen mode

Listing our Workers and Containers again:

$ fly -t ci workers name containers platform tags team state version age demo 1 linux none none running 2.2 45m32s $ fly -t ci containers handle worker pipeline job handle worker pipeline job build # build id type name attempt e467775c-ab6e-413d-5823-45fc84e2a3fd demo yeeehaa my-job 5 8 task say-hello n/a e5908c49-0ca2-4482-558e-99da76c44a79 demo none none none none check none n/a
Enter fullscreen mode Exit fullscreen mode

Resources:

If you feel like exploring more, check out their website and my other blogs for more concourse content:

Thank You

Let me know what you think. If you liked my content, feel free to visit me at ruan.dev or follow me on twitter at @ruanbekker

Top comments (1)

Collapse
 
vjvijay868 profile image
vijay •

can u please help me how to setup concourse CI on windows like linux without docker and clusters