In this tutorial we will build a ci pipeline using concourse to build and push a image to dockerhub automatically, whenever a new git commit is made to the master branch.
Our Project Setup
Our Directory Tree:
$ find . ./Dockerfile ./ci ./ci/pipeline.yml ./README.md ./docker-tunnel The project used in this example is not important, but you can check it out at https://github.com/ruanbekker/docker-remote-tunnel
Our Pipeline
A visual to see how the pipeline will look like in concourse:
Our pipeline definition will consist of 3 resources, github repo, dockerhub image and a slack resource to inform use whether a build has completed.
Then we are specifying that the job should be triggered on a git commit for the master branch, build and push to our dockerhub repo.
Our pipeline definition ci/pipeline.yml:
resources: - name: git-repo type: git source: uri: git@github.com:ruanbekker/docker-remote-tunnel.git branch: master private_key: ((github_private_key)) - name: docker-remote-tunnel-image type: docker-image source: repository: ruanbekker/docker-remote-tunnel tag: test username: ((dockerhub_user)) password: ((dockerhub_password)) - name: slack-alert type: slack-notification source: url: ((slack_notification_url)) resource_types: - name: slack-notification type: docker-image source: repository: cfcommunity/slack-notification-resource tag: v1.3.0 jobs: - name: build-cached-image plan: - get: git-repo trigger: true - task: build-cached-image-workspace config: platform: linux image_resource: type: docker-image source: repository: rbekker87/build-tools outputs: - name: workspace inputs: - name: git-repo run: path: /bin/sh args: - -c - | output_dir=workspace cat << EOF > "${output_dir}/Dockerfile" FROM alpine ADD git-repo /tmp/git-repo RUN mv /tmp/git-repo/docker-tunnel /usr/bin/docker-tunnel RUN apk --no-cache add screen docker openssl openssh-client apache2-utils RUN /usr/bin/docker-tunnel -h RUN rm -rf /tmp/git-repo EOF cp -R ./git-repo "${output_dir}/git-repo" - put: docker-remote-tunnel-image params: build: workspace on_failure: put: slack-alert params: channel: '#system_events' username: 'concourse' icon_emoji: ':concourse:' silent: true text: | *$BUILD_PIPELINE_NAME/$BUILD_JOB_NAME* ($BUILD_NAME) FAILED to build image https://ci.domain.com/teams/$BUILD_TEAM_NAME/pipelines/$BUILD_PIPELINE_NAME/jobs/$BUILD_JOB_NAME/builds/$BUILD_NAME on_success: put: slack-alert params: channel: '#system_events' username: 'concourse' icon_emoji: ':concourse:' silent: true text: | *$BUILD_PIPELINE_NAME/$BUILD_JOB_NAME* ($BUILD_NAME) SUCCESS - Image has been published https://ci.domain.com/teams/$BUILD_TEAM_NAME/pipelines/$BUILD_PIPELINE_NAME/jobs/$BUILD_JOB_NAME/builds/$BUILD_NAME - name: test plan: - get: docker-remote-tunnel-image passed: [build-cached-image] trigger: true - get: git-repo passed: [build-cached-image] - task: run-tests image: docker-remote-tunnel-image config: platform: linux inputs: - name: git-repo run: dir: git-repo path: sh args: - /usr/bin/docker-tunnel - --help on_failure: put: slack-alert params: channel: '#system_events' username: 'concourse' icon_emoji: ':concourse:' silent: true text: | *$BUILD_PIPELINE_NAME/$BUILD_JOB_NAME* ($BUILD_NAME) FAILED - Testing image failure https://ci.domain.com/teams/$BUILD_TEAM_NAME/pipelines/$BUILD_PIPELINE_NAME/jobs/$BUILD_JOB_NAME/builds/$BUILD_NAME on_success: put: slack-alert params: channel: '#system_events' username: 'concourse' icon_emoji: ':concourse:' silent: true text: | *$BUILD_PIPELINE_NAME/$BUILD_JOB_NAME* ($BUILD_NAME) SUCCESS - Testing image Succeeded https://ci.domain.com/teams/$BUILD_TEAM_NAME/pipelines/$BUILD_PIPELINE_NAME/jobs/$BUILD_JOB_NAME/builds/$BUILD_NAME Note that our secret information is templatized and saved in our local credentials.yml which should never be stored in version control:
slack_notification_url: https://api.slack.com/aaa/bbb/ccc dockerhub_user: myuser dockerhub_password: mypasswd github_private_key: |- -----BEGIN RSA PRIVATE KEY----- some-secret-data -----END RSA PRIVATE KEY------ Set the Pipeline:
Now that we have our pipeline definition, credentials and application code (stored in version control), go ahead and set the pipeline, which will save the pipeline configuration in concourse:
# pipeline name: my-docker-app-pipeline $ fly -t scw sp -n main -c pipeline.yml -p my-docker-app-pipeline -l credentials.yml Now the pipeline is saved on concourse but in a paused state, go ahead and unpause the pipeline:
$ fly -t scw up -p my-docker-app-pipeline Test your Pipeline
Make a commit to master and head over to concourse and look at it go:
Thanks for reading, make sure to check out my other posts on #concourse
Top comments (2)
Nice post! I recently had to do this and having more resources like this available would have been helpful. 🙌
Have you played around with this new OCI registry image resource type/build task at all?
I haven't, but I think the goal of eliminating the Docker daemon (and eventually privileged containers as well) requirement is a noble one.
Running Docker within Concourse's Garden containers can be rough sometimes. 😔
Hey Tim,
I have not played around with them, but thanks for mentioning them, those look really useful. Think I will start to use them right away.
Totally agree about moving away from the docker daemon is a great one!
Thanks so much for the comment.