Compliance handling of `needs` statements in pipeline execution policies

Feature Flag

This feature is available behind a feature flag: ensure_pipeline_policy_pre_stage_complete. Rollout tracked in #500652 (closed).

Release notes

To strengthen pipeline execution control, we are updating the behavior of jobs in the reserved stage .pipeline-policy-pre for users enforcing Pipeline Execution Policies. Currently, jobs in .pipeline-policy-pre and jobs with needs: [] both start immediately. This update ensures that jobs in .pipeline-policy-pre complete before any other jobs without dependencies start, helping users enforce ordered execution as intended in compliance and security policies.

Customers rely on reserved stages to enforce compliance and security checks before developer jobs run. For example, a compliance check job may be required to run first and, if it fails, to fail the pipeline. Allowing jobs to run out of order can bypass this enforcement, weakening policy intent. This improvement supports consistent compliance enforcement.

As an exception, manual jobs and jobs running when: on_failure will not require jobs in the .pipeline-policy-pre stage to complete as this could result in pipelines that never complete or resolve.

Demo

📺 Compliance handling of needs statements in pipeline execution policies

Why are we doing this work

When there are jobs in the reserved stage .pipeline-policy-pre using Pipeline Execution Policies, we want to ensure that jobs in this stage finish before any other jobs with needs: [] start executing.

Currently, jobs with needs: [] start immediately, and jobs in .pipeline-policy-pre also start immediately. This behavior is consistent with the documented behavior with regards to the .pre, but we would like to change it for .pipeline-policy-pre.

As customers enforce pipeline execution policies, they want to ensure that jobs they've defined in reserved stages are enforced and cannot be circumvented, and that any jobs that need to run upfront prior to the development team jobs cannot be run out of order.

A common use cases is to create a compliance check job that runs first and if it fails, fail the pipeline. If a job runs out of order, it may circumvent the intent of such a job.

Relevant links

Non-functional requirements

• Documentation:
• Feature flag:
• Performance:
• Testing:

Implementation plan

Verification steps

1. Create a group
2. Create a project test in your group
3. Create a a .gitlab-ci.yml file with content:

   project job:
    stage: test
    script:
    - echo "Project CI script"
    needs: []

4. Create a another file group-policy-ci.yml on the project with content:

   group policy pre job:
    stage: .pipeline-policy-pre
    script:
    - echo "Enforce your policy here"
   group policy manual job:
    stage: .pipeline-policy-pre
    script:
    - echo "Enforce your policy here"
    when: manual
   group policy skipped job:
    stage: .pipeline-policy-pre
    script:
    - echo "Enforce your policy here"
    when: on_failure
   group policy post job:
    stage: .pipeline-policy-post
    script:
    - echo "Enforce your policy here"
    needs: []

5. Create a another file project-policy-ci.yml on the project with content:

   project policy pre job:
    stage: .pipeline-policy-pre
    script:
    - echo "Enforce your policy here"
   project policy post job:
    stage: .pipeline-policy-post
    script:
    - echo "Enforce your policy here"
    needs:
    - project policy pre job

6. Enable the feature flag:

   /chatops run feature set --project=path-to-your-group/test ensure_pipeline_policy_pre_stage_complete true

7. On the left sidebar, select Security & Compliance and Policies..
8. Select New Policy.
9. Select Pipeline execution policy.
10. Choose a name for the policy.
11. In the Actions section select the project and project-policy-ci.ym file you created before.
12. Select Update via Merge Request.
13. Merge the MR.
14. Go to the parent group of the project you created in step 1.
15. On the left sidebar, select Security & Compliance and Policies..
16. Select New Policy.
17. Select Pipeline execution policy.
18. Choose a name for the policy.
19. In the Actions section select the project and group-policy-ci.ym file you created before.
20. Select Update via Merge Request.
21. Merge the MR.
22. Go back to the project and start a new pipeline

All jobs in the pipeline should wait for the .pipeline-policy-pre to be completed before they start processing. Skipped and manual jobs don't have to be completed.