Implements a resource that passes sets of key-value pairs between jobs without using any external storage with resource like Git or S3.

Pulled by a get step, key-value pairs are provided to the build plan as an artifact directory with one file per key-value pair. The name of a file is the “key”, and its contents is the “value”. These key-value pairs can then be loaded as local build vars using a load_var step.

Pushed by a put step, key-value pairs are persisted in the Concourse SQL database. For this to be possible, the trick is that they are serialized as keys and values in version JSON objects. As such, they are designed to hold small, textual, non-secret data.

In case you're dealing with large text, binary data or secrets, we recommend you opt for other solutions. Indeed, secrets will be best stored in a vault like CredHub, large text files in Git, and binary data in some object storage or Git with Git-LFS, the “Large File Storage” addon.

This resource is a fork of the keyval resource by @moredhel, thanks for the work done!

Compared to the original keyval resource from SWCE by @regevbr and @ezraroi, writing key-value pairs as plain files in some resource folder is more consistent with usual conventions in Concourse, when it comes to storing anything in step artifacts. It is also compliant with the ConfigMap pattern from Kubernetes.

Writing/reading files is always easier in Bash scripts than parsing some Java Properties file, because much less boilerplate code is required.

Inital v1.1.0 release has fixed many defects, that made it not usable (non-working put steps).

That one implements Bloblang-generated contents, and only allow creating new key-values through the params of put steps, not through writing plain files in some artifact directory of tasksteps like we do here.

resource_types: - name: key-value type: registry-image source: repository: gstack/keyval-resource resources: - name: key-value type: key-value

• history_identifier: Optional. When the “global resources” feature is enabled on your Concourse installation, and you don't want a single resource history for all the keyval resources defined in your Concourse installation, then set this property to a relevant identifier, possibly unique or not. See the “global resources” section for a detailed discussion on use-cases and solutions.

This is a version-less resource so check behavior is no-op.

It will detect the latest store key/value pairs, if any, and won't provide any version history.

None.

Fetches the given key & values from the stored resource version JSON (in the Concourse SQL database) and write them in their respective files where the key is the file name and the value is the file contents.

"version": { "some_key": "some_value" }

would result in:

$ cat resource/some_key some_value

None.

Converts each file in the artifact directory designated by directory to a set of key-value pairs, where file names are the keys and file contents are the values. This set of key-value pairs is persisted in the version JSON object, to be stored in the Concourse SQL database.

A value from a file in directory can be overridden by a matching key with different value in the dictionary given as the overrides parameter. If you need to store some Concourse ((vars)) value in a key-value resource, then add it to the overrides parameter of some put step.

• directory: Required. The artifact directory to be scanned for files, in order to generate key-value pairs

• overrides: Optional. A dictionary of key-value pairs that will override any matching pair with same key found in directory.

This example make intentional ellipsis in order to focus on the main ideas behind the “keyval” resource. Seasoned Concourse practitioners can find an illustration here in one catch.

resource_types: - name: key-value type: registry-image source: repository: gstack/keyval-resource resources: - name: build-info type: key-value jobs: - name: build plan: - task: build file: tools/tasks/build/task.yml # <- must declare a 'build-info' output artifact - put: build-info params: directory: build-info - name: test-deploy plan: - in_parallel: - get: build-info passed: [ build ] - task: test-deploy file: tools/tasks/task.yml # <- must declare a 'build-info' input artifact

The build task writes all the key-value pairs it needs to pass along in files inside the build-info output artifact directory.

The test-deploy job then reads the files from the build-info resource, which produces a build-info artifact directory to be used by the test-deploy task.

This fully-working and detailed example goes deeper in showcasing what the resource can actually do and how. Concourse beginners are recommended to read this as it details very clearly the relation between resource, artifact directories, and tasks.

resource_types: - name: key-value type: registry-image source: { repository: gstack/keyval-resource } resources: - name: some-keyval-resource type: key-value - name: runner-image type: registry-image source: { repository: busybox } jobs: - name: step-1-job plan: - get: runner-image - task: write-keyval-aaa-1-task image: runner-image config: platform: linux outputs: [ { name: created-keyvals-artifact } ] run: path: sh args: - -exc - |  echo "1" > created-keyvals-artifact/aaa  - put: some-keyval-resource params: directory: created-keyvals-artifact - name: step-2-job plan: - in_parallel: - get: keyvals-artifact # here artifact directory is resource: some-keyval-resource # different from resource name trigger: true passed: [ step-1-job ] - get: runner-image - task: read-aaa-keyval-task image: runner-image config: platform: linux inputs: [ { name: keyvals-artifact } ] run: path: sh args: - -exc - |  cat keyvals-artifact/aaa # -> 1  - task: write-bbb-keyval-task image: runner-image config: platform: linux inputs: [ { name: keyvals-artifact } ] outputs: [ { name: keyvals-artifact } ] run: path: sh args: - -exc - |  echo "2" > build-info/bbb  - put: some-keyval-resource params: directory: keyvals-artifact overrides: aaa: "11" ccc: "3" - name: step-3-job plan: - in_parallel: - get: some-keyval-resource # artifact dir will have same name trigger: true passed: [ step-2-job ] - get: runner-image - task: read-aaa-bbb-ccc-keyvals-task image: runner-image config: platform: linux inputs: [ { name: some-keyval-resource } ] run: path: sh args: - -exc - |  cat build-info/aaa # -> 11  cat build-info/bbb # -> 2  cat build-info/ccc # -> 3

The write-keyval-aaa-1-task creates a file named aaa with content 1 to the created-keyvals-artifact output artifact directory. The some-keyval-resource resource will read files in the created-keyvals-artifact directory and store a key-value pair {"aaa": "1"}.

The read-aaa-keyval-task reads the value from the aaa file in keyvals-artifact input artifact directory provided from the some-keyval-resource resource. This outputs 1.

The write-bbb-keyval-task creates a file named bbb with content 2 to keyvals-artifact output artifact directory. Because this directory is same as keyvals-artifact input artifact directory which already contains aaa. The some-keyval-resource resource will read all files in the keyvals-artifact directory and store key-value pairs {"aaa": "1"} and {"bbb": "2"}.

The put: some-keyval-resource in step-2-job provides the overrides option, which changes the original key-value pair {"aaa": "1"} to {"aaa": "11"} and add a new pair {"ccc": "3"}.

The read-aaa-bbb-ccc-keyvals-task reads values from files in the some-keyval-resource input artifact directory, as provided by the some-keyval-resource resource.

When the “global resources” feature is enabled, all resources with same resource.type and resource.source configuration will share the same version history. If you leave the resource.source configuration blank in all your keyval resources, then they will all be considered the exact same resource by Concourse, sharing the exact same history.

For most keyval resource-related use-case though, this is not releavant and thus requires proper scoping.

In many scearios, the key-value resources is used to transmit pipeline-specific data between jobs of the same pipeline. In such case, sharing resource history is most probably irrelevant. In order to avoid this, you can set the history_identifier to some value that will be unique in your Concourse installation.

For best portability of your pipeline across different Concourse installations, we recommend that you use a UUID that can be generated with the uuidgen command-line tool like this:

$ uuidgen | tr [:upper:] [:lower:] fc4cb2ba-d0d4-44e2-8589-8fa89a8271fd

Then use it in the resource configuration, so that the resource history is scoped privately:

resources: - name: key-value type: key-value source: history_identifier: fc4cb2ba-d0d4-44e2-8589-8fa89a8271fd

In some scenarios though, it may be interesting to share the resources history between different pipelines. Then you can leverage key-value resources that share the same history_identifier value.

As a result, as soon as a new version is pushed on the shared key-value resources, all other pipelines will see it.

This is interesting in case some pipeline has to trigger other pipelines. A usual solution is to use a “dummy” semver resource, backed by Git or some object storage.

Using the keyval resource can bring an elegant alternative. A limitation is that this resource basically has no version history. At every point in time, only the last vesion exisits for the resource. This is not an issue for this use-case, though. Indeed with a “dummy” semver resource, experience shows that nobody actually pays attention to the version history anyway.

With the keyval resource, the triggering version only need to specify the date and relevant data showing the reason why the pipeline has been triggered. These will appear and properly stay in job build logs, for later inspection.

Key-value pairs are no more written as Java .properties file, but rather one file per key-value pair. The name of a file is a “key”, and its contents is the related “value”.

The required file paramerter for put steps is replaced by directory.

The required directory paramerter has been added to put steps.

The file parameter of put steps is renamed overrides.

Golang unit tests can be run from some shell command-line with Ginkgo, that has to be installed first.

make test

These unit test are embedded in the Dockerfile, ensuring they are consistently run in a determined Docker image providing proper test environment. Whenever the tests fail the Docker build will be stopped.

In order to build the image and run the unit tests, use docker build as follows:

docker build -t keyval-resource .

Please make all pull requests to the master branch and ensure tests pass locally.

When submitting a Pull Request or pushing new commits, the Concourse CI/CD pipeline provides feedback with building the Dockerfile, which implies running Ginkgo unit tests.

Copyright © 2021-present, Benjamin Gandon, Gstack

Like Concourse, the key-value resource is released under the terms of the Apache 2.0 license.