Publish SBOMs for operator images

[dervoeti]

Fixes stackabletech/issues#398.

Successful test run with airflow-operator: https://github.com/stackabletech/airflow-operator/actions/runs/7652323720/job/20851912198

The verified SBOM for the airflow-operator can be fetched with:

cosign verify-attestation --type cyclonedx --certificate-identity-regexp '^https://github.com/stackabletech/.+/.github/workflows/.+@.+' --certificate-oidc-issuer https://token.actions.githubusercontent.com oci.stackable.tech/sdp/airflow-operator:0.0.0-dev | jq '.payload' -r | base64 -d | jq '.predicate' -r > verified_sbom.json 

Notes and possible next steps:

• I also removed an incorrect colon from the image references. Before, it was ${DOCKER_REPO}/${ORGANIZATION}/${OPERATOR_NAME}:@$$REPO_DIGEST_OF_IMAGE which results in something like oci.stackable.tech/sdp/airflow-operator:@sha256:b2707e4a282844d314aa4ee261b4680c91c0c5f484ed6d235282d7c4ff10bda2, which is an invalid reference (but cosign ignored it and treated it correctly). Now it's ...airflow-operator@sha256:b27... instead of ...airflow-operator:@sha256:b27... (first colon removed).
• I handcraft PURLs for the operator images in the Makefile. They look like this: pkg:docker/sdp/airflow-operator@sha256:50c2df6011949ded5e17cf1c85facf568ffdcca679664feb540dd300ea7c07f5?repository_url=oci.stackable.tech. I verified that one using packageurl.rs:

PackageUrl { ty: "docker", namespace: Some( "sdp", ), name: "airflow-operator", version: Some( "sha256:50c2df6011949ded5e17cf1c85facf568ffdcca679664feb540dd300ea7c07f5", ), qualifiers: { "repository_url": "oci.stackable.tech", }, subpath: None, } 

• Syft adds all the labels from the container image to the SBOM under .metadata.properties. This includes a description of the UBI minimal image, which we could remove from the image labels during the build process.
• Out operator images contain Python (/usr/libexec/platform-python3.6), we could check whether it's needed and where it comes from.
• We need to document to our customers how to get and verify the SBOMs. Customers can enforce the presence of a valid SBOM before starting a container via tools like Kyverno. Since SBOMs are published in Harbor this will first be relevant once we pull our images from Harbor by default.
• When I ran Syft version 0.101.1 on my Macbook, it reported all the crates for airflow-operator twice: once with our generated SBOM as source (the one generated by cargo cyclonedx), once with the operator binary as source. This did not happen on Linux (even though it was the same version), there it only reported the crates with the SBOM as source (which is what we want). I haven't debugged this further and just passed the --exclude "/usr/local/bin/stackable-${OPERATOR_NAME}" flag to Syft to make sure it does not try to analyze the binary, that way it produced the correct output for me on MacOS and Linux.

[lfrancke]

I haven't checked but I assume shellcheck would complain about missing double quotes everywhere.

Could you run it through shellcheck?

[dervoeti]

I haven't checked but I assume shellcheck would complain about missing double quotes everywhere.

Could you run it through shellcheck?

I think I can't shellcheck the Makefile, I added some double-quotes though 🙂

[lfrancke]

I just copied all of it into a shell file and ran shellcheck, no need for it to actually run the script :)

[lfrancke]

Only tangentially related to this but when I look in Harbor I see a few things not signed by cosign and I have a hard time figuring out what is what.
Is there any way to "see" the SBOMs there?

[dervoeti]

Yes, I was confused by this in the beginning as well. The thing that's stored in Harbor is not directly the SBOM itself, but an in-toto attestation wrapped in a DSSE envelope (that's the one carrying the signature metadata). These attestations are tagged with the digest of the artifacts it's attached to and a ".att" suffix ("sha256-20f210167078941709d63ec7b7d9b004a95932148ece9a05a9fe73c55d874f0a.att"). This is similar to signatures (there it's a .sig suffix). So the signature is an "attachment" (via this special tag) to the artifact, containing nothing but the data about the signing process and the signature itself. The SBOM attestation artifact contains the signature plus the base64 encoded SBOM. Harbor supports displaying signature artifacts that are attached to an OCI artifact (those OCI artifacts are then displayed as "signed"). But the SBOM attestation contains the signature in itself (there's no .sig artifact attached to it). That's why it's not displayed as "signed", even though it is (otherwise the cosign verification command would fail). I can try to explain more if needed 🙂 I traversed the whole "attestation verifiaction" flow with docker manifest inspect and oras blob fetch and decoded the single layers manually.
In the beginning I expected it's simply the SBOM stored as OCI artifact with a separate signature attached to it, which is not the case.

[lfrancke]

Pewww....that comment needs to be captured somewhere.

Okay, we can do this as a separate documentation issue, but: We somehow need (I think) an "easy" way for people to download the SBOMs. This can be a separate step where we upload them somewhere for example,

[dervoeti]

Yep. Maybe we can publish them as JSON in separate places (tbd where exactly). I think the current place (Harbor) is more suitable for a "machine-readable" use case: Customers wanting to enforce that all Stackable containers in their cluster have a valid SBOM (signed by one of Stackable's GH Action Workflows) attached. It is possible to get the verified SBOM out of Harbor (with the cosign command in the first comment), but this is rather complicated compared to "Just download this JSON file". But it solves the issue of "this JSON file can actually be trusted". If we publish links to JSON files in our release notes, these JSON files are neither signed nor can they in any way directly be related to the build process. The SBOM attestation in Harbor contains signed provenance information along with the SBOM itself:

Certificate subject: https://github.com/stackabletech/airflow-operator/.github/workflows/build.yml@refs/heads/spike/attest-sbom Certificate issuer URL: https://token.actions.githubusercontent.com GitHub Workflow Trigger: push GitHub Workflow SHA: 23fadc3b1fa5d9762794af82825b48f4594d8f02 GitHub Workflow Name: Stackable Build Pipeline GitHub Workflow Repository: stackabletech/airflow-operator GitHub Workflow Ref: refs/heads/spike/attest-sbom