CLOUDP-333692: Re-design images building

[Julien-Ben]

Re-design images building

Note for review:

Since atomic_pipeline.py is largely a refactored version of pipeline.py, it’s much clearer to review their side-by-side diff than to wade through GitHub’s “all new lines” view.
Here's the diff:

https://gist.github.com/Julien-Ben/3698d532d17bafb380f2e4f05b5db153

You can also take a look at the related TD Section

Changes

The PR refactors our Docker image build system. Most notably by replacing pipeline.py along with other components, detailed below.

Usage of standalone Dockerfiles

Added in a previous PR, they eliminate the need for templating, and make it possible to retire Sonar once the Atomic Releases Epic is completed.

Building with docker buildx, multi-platform builds

In build_images.py we use docker buildx through a python API. It eliminates the need for building images separately for each platform (ARM/AMD), and then manually bundling them in a manifest.

Handle build environments explicitly

We’ve introduced a framework that centralizes build configuration by scenario (e.g local development, staging releases etc) so the pipeline automatically picks sensible defaults (registry, target platforms, signing flags, and more) based on where you’re running.

In pipeline_main.py (with support from build_configuration.py and build_context.py) we treat each execution context (local dev, merge to master, release etc...) as an explicit, top-level environment.
It infers defaults automatically but lets you override any value via CLI flags, ensuring all build parameters live in one single source of truth rather than scattered through pipeline scripts.

CLI usage

usage: pipeline_main.py [-h] [--parallel] [--debug] [--sign] [--scenario {BuildScenario.RELEASE,BuildScenario.PATCH,BuildScenario.STAGING,BuildScenario.DEVELOPMENT}] [--platform PLATFORM] [--version VERSION] [--registry REGISTRY] [--parallel-factor PARALLEL_FACTOR] image Build container images. positional arguments: image Image to build. options: -h, --help show this help message and exit --parallel Build images in parallel. --debug Enable debug logging. --sign Sign images. --scenario {BuildScenario.RELEASE,BuildScenario.PATCH,BuildScenario.STAGING,BuildScenario.DEVELOPMENT} Override the build scenario instead of inferring from environment. Options: release, patch, master, development --platform PLATFORM Target platforms for multi-arch builds (comma-separated). Example: linux/amd64,linux/arm64. Defaults to linux/amd64. --version VERSION Override the version/tag instead of resolving from build scenario --registry REGISTRY Override the base registry instead of resolving from build scenario --parallel-factor PARALLEL_FACTOR Number of builds to run in parallel, defaults to number of cores 

Proof of work

CI is building images with the new pipeline, and tests pass.

Note

For the duration of the Atomic Releases epic, both pipelines will be in the repository, until we are done with the staging and promotion process. This new pipeline will only be used for Evergreen patches.
This PR also heavily depends on changes that are introduced by the agent matrix removal, and the multi-platform support epic.

The existing Evergreen function, that uses pipeline.py has been renamed legacy_pipeline, and is used for release and periodic builds tasks.
A new one has been created, calling the new pipeline.

Once the Atomic Release Epic is complete, we'll be able to remove:

• Sonar
• Inventories
• Periodic builds
• pipeline.py

Follow up ticket to this PR: https://jira.mongodb.org/browse/CLOUDP-335471

Checklist

• Have you linked a jira ticket and/or is the ticket in the title?
• Have you checked whether your jira ticket required DOCSP changes?
• Have you added changelog file?
  • use skip-changelog label if not needed
  • refer to Changelog files and Release Notes section in CONTRIBUTING.md for more details

[github-actions]

⚠️ (this preview might not be accurate if the PR is not rebased on current master branch)

MCK 1.3.0 Release Notes

Other Changes

• Optional permissions for PersistentVolumeClaim moved to a separate role. When managing the operator with Helm it is possible to disable permissions for PersistentVolumeClaim resources by setting operator.enablePVCResize value to false (true by default). When enabled, previously these permissions were part of the primary operator role. With this change, permissions have a separate role.
• subresourceEnabled Helm value was removed. This setting used to be true by default and made it possible to exclude subresource permissions from the operator role by specifying false as the value. We are removing this configuration option, making the operator roles always have subresource permissions. This setting was introduced as a temporary solution for this OpenShift issue. The issue has since been resolved and the setting is no longer needed.

[Julien-Ben]

How will this be merged with Maciej's changes (the release_info file)?

Maciej answered above, but I also plan to sync with him as soon as this is merged, to properly rebase and wire everything togetger

[lucian-tosa]

So this is true only when triggering manual patches or PR commits? Will it be false on merges to master?

[Julien-Ben]

I made it more explicit, and we'll make proper use of staging registry in a follow up:
2ec7587

[lucian-tosa]

I was referring to the is_patch variable from evergreen. This is what I found from the devprod docs

${is_patch} is "true" if the running task is in a patch build and undefined if it is not.

Does this mean that is_patch will be false in a merge to master?

[Julien-Ben]

@nammn @lucian-tosa @MaciejKaras

I'd like your opinion on two solutions here
This function is sometimes failing when building multiple agents concurrently.
I tried two alternatives:

• Catching the exception
• Use ensure_buildx_builder at top leven, in the pipeline_main

None of them are ideal. Using exception is not the right way of handling concurrency. Making the main aware of buildx increase coupling.
My knowledge of python multiprocessing is limited. I know that for multi threading it is easy to make a lock with a global variable and threading.Lock() . From what I've seen online, for multi processing we would need to pass the lock from the agent building method all the way down to here.
I feel like it adds a lot of complexity and optional variables.

What solutions do you prefer ?
Do you see an alternative ?

Here is the alternative solution with ensuring builder in main:
1badff0

[nammn]

since we have the span already setup here - can we save the span attributes as we already log them anyway?

[Julien-Ben]

I think it should really be done as a follow up, it might look like one minute but I have no idea how to properly setup spans.
The scope of this PR is getting quite large

[Julien-Ben]

It's listed in the follow up ticket
https://jira.mongodb.org/browse/CLOUDP-335471

[nammn]

you don't have to setup anything. All you need to do is already there.
its
span.set_attribute("mck.platforms", build_configuration.platforms)
span.set_attribute("mck.registry", registry)

and the same everywhere...

[nammn]

or here - i think adding tracing costs 1 minute and saves us all the troubles later.

[nammn]

especially since you are logging the exact information we need as metrics anyway

[Julien-Ben]

I think it should really be done as a follow up, it might look like one minute but I have no idea how to properly setup spans.
The scope of this PR is getting quite large