Skip to content

Support suffix for jobs with name collisions in pipeline execution policy pipelines

Release post

An enhancement to the 17.2 release of pipeline execution policies, policy creators may now configure pipeline execution policies to handle collisions in job names gracefully. With the policy.yml for the pipeline execution policy, you may now configure the following options:

  • suffix: on_conflict configures the policy to gracefully handle collisions by renaming policy jobs
  • suffix: never enforces all jobs names are unique and will fail pipelines if collisions occur, which has been the default behavior since 17.2

With this improvement, you can ensure security and compliance jobs executed within a pipeline execution policy always run, while also preventing unnecessary impacts to developers downstream.

Why are we doing this work

At a minimum we need to be able to support existing compliance pipeline configurations so they can migrate. Customers want to be able to follow DRY principles and use things like includes rather having to maintain the templates. So when using includes today they may end up enforcing jobs with the same name.

Currently, we enforce job name uniqueness which hard-fails the pipeline. This can cause issues with the built-in templates for security scanners which don't allow for job name customizations. If two policies include the same built-in template, the pipeline fails.

Acceptance criteria

  • Avoid hard-fail or disruption to project pipelines, as these jobs will need to be enforced globally across thousands of projects in some cases. Worst case scenario, we could hard-fail the policy jobs being enforced, but not the downstream pipelines. And in this scenario, we'd need error handling that can be consumed by users to identify failures.
  • Gracefully handle job name conflicts to avoid CI errors raised as concerns from the Verify team, while also ensuring policy jobs can still run.
  • Indicate to users which jobs are being enforced from outside of their project configuration.

Use Cases

  1. As a security or compliance team member, I want to enforce Dependency Scanning to run in all of my project pipelines. I want to be able to drop in a scan template using an include that references the GitLab security template: https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Dependency-Scanning.gitlab-ci.yml. This template has defined the job in stage test and it uses the name dependency_scanning. If I enforce this through a PEP on all projects, but some projects also have this include in their own configuration, this could result in conflicting stage names. This also results in the job being run in test. Ideally, with pipeline execution policies, I could use an include for GitLab templates without any modification and have those jobs enforced securely, with assurance of compliance, and with no impact to downstream development projects.
  2. Similar to above, I want to take the Dependency Scanning template and customize the rules to suite my business case. So I copy the contents and store it locally. This could be in a central shared location within my repo, or within my security policy project. I would then customize the template to add my own rules, ideally with minimal changes to what we use upstream from GitLab. As GitLab adds improvements, we can then merge those changes from upstream templates. Customized rules may be additional logic for which pipelines to run the jobs in, such as to run jobs in merge request pipelines or build pipelines. In this case, ideally the responsibility would not be on me to ensure uniqueness of the job names to be sure the jobs run or to avoid disruption in downstream projects.
  3. Enforce security scan templates and a reporting script - #473189 (comment 2051673952)

Proposal

  1. Add an option in the pipeline execution policy yaml to configure the behavior for how jobs are injected for the inject mode only.
  2. Users may choose an option for the suffix parameter, with the following options:
    1. suffix: on_conflict -- if a conflict occurs between the policy job and another job with the same name (from another policy or a downstream project pipeline), the conflict will be handled by the policy.
    2.  suffix: never -- if a conflict occurs between the policy job and another job with the same name (from another policy or a downstream project pipeline), there will be an assumed configuration error and the pipeline will fail with an error, requiring the issue to be resolved before a pipeline can run successfully. In YAML, this option will appear as below:
type: pipeline_execution_policy
name: PEP
description: ''
enabled: true
pipeline_config_strategy: inject_ci
suffix: on_conflict
content:
 include:
 - project: gitlab-org/pipeline-execution-policies/compliance-project
 file: override.yml
  1. When on_conflict is selected, any jobs enforced by this policy, if a collision occurs, will be modified to contain a unique suffix using the pattern policy-<id>-0. This pattern will include an ID providing a more deterministic way to identify policy enforced jobs. The index -0 at the end of the job will ensure job names are unique, non-conflicting, and cannot be circumvented.
  2. Within the policy editor, we'll add a new option to select the behavior for managing conflicts, with context for when to use either option.

Relevant links

Non-functional requirements

  • Documentation:
  • Feature flag:
  • Performance:
  • Testing:

Implementation plan

Verification steps

Edited by Grant Hickman