Example project showing how to get started with Crossplane, connect it to multiple providers like AWS, Azure - and provision some resources like a S3Bucket or a StorageAccount through K8s CRDs

This repo is accompanied by this blog post https://blog.codecentric.de/en/2022/07/crossplane/

Also it is used as an example in these three iX Magazin issues (simply click on the magazine covers to access the full articles):

Crossplane https://crossplane.io/ claims to be the "The cloud native control plane framework". It introduces a new way how to manage any cloud resource (beeing it Kubernetes-native or not). It's an alternative Infrastructure-as-Code tooling to Terraform, AWS CDK/Bicep or Pulumi and introduces a higher level of abstraction - based on Kubernetes CRDs.

Literally the best intro post to Crossplane for me was https://blog.crossplane.io/crossplane-vs-cloud-infrastructure-addons/ - here the real key benefits especially compared to other tools are described. Without marketing blabla. If you love deep dives, I can also recommend Nate Reid's blog https://vrelevant.net/ who works as Staff Solutions Engineer at Upbound.

Here are the brief steps to spin up Crossplane and provision an S3 Bucket on AWS (Azure is also possible):

## DEMO AWS No.1 # Create a kind cluster kind create cluster --image kindest/node:v1.33.1 --wait 5m kubectl get crd # Install Crossplane helm dependency update crossplane-install helm upgrade --install crossplane --namespace crossplane-system crossplane-install --create-namespace kubectl wait --for=condition=ready pod -l app=crossplane --namespace crossplane-system --timeout=120s # Check Crossplane is there & get CRDs before the Provider installation kubectl get all -n crossplane-system kubectl get crd # Install AWS Provider (Official Upbound Provider Family-based) kubectl apply -f upbound/provider-aws-s3/config/provider-aws-s3.yaml kubectl wait --for=condition=healthy --timeout=180s provider/upbound-provider-aws-s3 kubectl get crd kubectl get crossplane # Configure AWS Provider  echo "[default] aws_access_key_id = $(aws configure get aws_access_key_id) aws_secret_access_key = $(aws configure get aws_secret_access_key) " > aws-creds.conf kubectl create secret generic aws-creds -n crossplane-system --from-file=creds=./aws-creds.conf kubectl apply -f upbound/provider-aws-s3/config/provider-config-aws.yaml # Create Simple S3 Bucket (based on Crossplane Managed Resources only) kubectl apply -f upbound/provider-aws-s3/resources/simple-bucket.yaml kubectl get crossplane kubectl delete -f upbound/provider-aws-s3/resources/simple-bucket.yaml ## DEMO AWS No.2:  # Create XRD kubectl apply -f upbound/provider-aws-s3/definition.yaml kubectl wait --for=condition=Established --timeout=120s xrd xobjectstorages.crossplane.jonashackt.io kubectl get xrd # Create Composition kubectl apply -f upbound/provider-aws-s3/composition.yaml kubectl get composition # Create Claim - which will provision the S3 Bucket kubectl apply -f upbound/provider-aws-s3/claim.yaml crossplane beta trace objectstorage.crossplane.jonashackt.io/managed-upbound-s3 -o wide kubectl get crossplane kubectl get claim kubectl get composite # Upload a static website (e.g. "App") aws s3 sync static s3://container-conf-bucket --acl public-read # Open Browser at http://crossplane-meetup-softwerkskammer.s3-website.eu-central-1.amazonaws.com aws s3 rm s3://container-conf-bucket/index.html # don't forget to delete the Claim kubectl delete -f upbound/provider-aws-s3/claim.yaml ```shell ## DEMO Azure No.1:   # Install Azure Provider (Official Upbound Provider Family-based) kubectl apply -f upbound/provider-azure-storage/config/provider-azure-storage.yaml kubectl wait --for=condition=healthy --timeout=120s provider/upbound-provider-azure-storage kubectl get crd  # Configure Azure Provider  az ad sp create-for-rbac --sdk-auth --role Owner --scopes /subscriptions/$SUBSCRIPTION_ID --name servicePrincipalCrossplaneGHActions > crossplane-azure-provider-key.json  kubectl create secret generic azure-account-creds -n crossplane-system --from-file=creds=./crossplane-azure-provider-key.json  kubectl apply -f upbound/provider-azure-storage/config/provider-config-azure.yaml  # Create XRD kubectl apply -f upbound/provider-azure-storage/definition.yaml  # Create Composition kubectl apply -f upbound/provider-azure-storage/composition.yaml  # Create Claim - which will provision the Azure Storage Account kubectl apply -f upbound/provider-azure-storage/claim.yaml crossplane beta trace storageazure.crossplane.jonashackt.io/managed-storage-account -o wide kubectl wait --for=condition=ready --timeout=180s resourcegroup rg-crossplane kubectl wait --for=condition=ready --timeout=360s storageazure.crossplane.jonashackt.io/managed-storage-account  kubectl delete -f upbound/provider-azure-storage/claim.yaml

https://docs.crossplane.io/latest/concepts/

• Managed Resourced (MR): Kubernetes custom resources (CRDs) that represent infrastructure primitives (mostly in cloud providers). All Crossplane Managed Resources could be found via https://doc.crds.dev/
• Composite Resources (XR): compose Managed Resources into higher level infrastructure units (especially interesting for platform teams). They are defined by:
  • a CompositeResourceDefinition (XRD) (which defines an OpenAPI schema the Composition needs to be conform to)
  • (optional) CompositeResourceClaims (XRC) (which is an abstraction of the XR for the application team to consume) - but is fantastic to hold the exact configuration parameters for the concrete resources you want to provision
  • a Composition that describes the actual infrastructure primitives aka Managed Resources used to build the Composite Resource. One XRD could have multiple Compositions - e.g. to one for every environment like development, stating and production
  • and configured by a Configuration
• Packages: OCI container images to handle distribution, version updates, dependency management & permissions for Providers & Configurations. Packages were formerly named Stacks.
  • Providers: are Packages that bundle a set of Managed Resources & a Controller to provision infrastructure resources - all providers can be found on GitHub, e.g. provider-aws or on docs.crds.dev. A list of all available Providers can also be found on GitHub.
  • Configuration: define your own Composite Resources (XRs) & package them via kubectl crossplane build configuration (now they are a Package) - and push them to an OCI registry via kubectl crossplane push configuration. With this Configurations can also be easily installed into other Crossplane clusters.

In order to use Crossplane we'll need any kind of Kubernetes cluster to let it operate in. This management cluster with Crossplane installed will then provision the defined infrastructure. Using any managed Kubernetes cluster like EKS, AKS and so on is possible - or even a local Minikube, kind or k3d.

https://crossplane.io/docs/v1.8/getting-started/install-configure.html

Be sure to have kind, the package manager Helm and kubectl installed:

brew install kind helm kubectl

https://docs.crossplane.io/latest/cli/

Also we should install the crossplane CLI

curl -sL "https://raw.githubusercontent.com/crossplane/crossplane/master/install.sh" | sh sudo mv crossplane /usr/local/bin

Now the kubectl crossplane --help command should be ready to use.

Now spin up a local kind cluster

kind create cluster --image kindest/node:v1.29.2 --wait 5m

The Crossplane docs tell us to use Helm for installation:

helm repo add crossplane-stable https://charts.crossplane.io/stable helm repo update helm upgrade --install crossplane --namespace crossplane-system --create-namespace crossplane-stable/crossplane

Using the appended --create-namespace, we don't need to explicitely create the namespace before running helm upgrade.

As an Renovate-powered alternative we can [create our own simple Chart.yaml to enable automatic updates](https://stackoverflow.com/a/71765472/4964553) of our installation if new crossplane versions get released:

apiVersion: v2 type: application name: crossplane version: 0.0.0 # unused appVersion: 0.0.0 # unused dependencies: - name: crossplane repository: https://charts.crossplane.io/stable version: 1.15.1

To install Crossplane using our own Chart.yaml simply run:

helm dependency update crossplane-install helm upgrade --install crossplane --namespace crossplane-system crossplane-install --create-namespace

Be sure to exclude charts and Chart.lock files via .gitignore.

# Exclude Helm charts lock and packages **/**/charts **/**/Chart.lock

Check Crossplane version installed with helm list -n crossplane-system :

$ helm list -n crossplane-system NAME	NAMESPACE	REVISION	UPDATED	STATUS	CHART	APP VERSION crossplane	crossplane-system	1	2022-06-21 09:28:21.178357 +0200 CEST	deployed	crossplane-1.8.1	1.8.1

Before we can actually apply a Provider we have to make sure that crossplane is actually healthy and running. Therefore we can use the kubectl wait command like this:

kubectl wait --for=condition=ready pod -l app=crossplane --namespace crossplane-system --timeout=120s

Otherwise we will run into errors like this when applying a Provider:

error: resource mapping not found for name: "provider-aws" namespace: "" from "provider-aws.yaml": no matches for kind "Provider" in version "pkg.crossplane.io/v1" ensure CRDs are installed first

Finally check crossplane status with kubectl get all -n crossplane-system:

$ kubectl get all -n crossplane-system NAME READY STATUS RESTARTS AGE pod/crossplane-7c88c45998-d26wl 1/1 Running 0 69s pod/crossplane-rbac-manager-8466dfb7fc-db9rb 1/1 Running 0 69s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/crossplane-webhooks ClusterIP 10.96.160.156 <none> 9443/TCP 4d NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/crossplane 1/1 1 1 69s deployment.apps/crossplane-rbac-manager 1/1 1 1 69s NAME DESIRED CURRENT READY AGE replicaset.apps/crossplane-7c88c45998 1 1 1 69s replicaset.apps/crossplane-rbac-manager-8466dfb7fc 1 1 1 69s

"The base Crossplane installation consists of two pods, the crossplane pod and the crossplane-rbac-manager pod. Before starting the core Crossplane container an init container runs. The init container installs the core Crossplane Custom Resource Definitions (CRDs), configures Crossplane webhooks and installs any supplied Providers or Configurations."" See https://docs.crossplane.io/latest/concepts/pods/

Now we should be able to find some new Kubernetes API objects:

$ kubectl api-resources | grep crossplane compositeresourcedefinitions xrd,xrds apiextensions.crossplane.io/v1 false CompositeResourceDefinition compositionrevisions comprev apiextensions.crossplane.io/v1 false CompositionRevision compositions comp apiextensions.crossplane.io/v1 false Composition environmentconfigs envcfg apiextensions.crossplane.io/v1alpha1 false EnvironmentConfig usages apiextensions.crossplane.io/v1alpha1 false Usage configurationrevisions pkg.crossplane.io/v1 false ConfigurationRevision configurations pkg.crossplane.io/v1 false Configuration controllerconfigs pkg.crossplane.io/v1alpha1 false ControllerConfig deploymentruntimeconfigs pkg.crossplane.io/v1beta1 false DeploymentRuntimeConfig functionrevisions pkg.crossplane.io/v1beta1 false FunctionRevision functions pkg.crossplane.io/v1beta1 false Function locks pkg.crossplane.io/v1beta1 false Lock providerrevisions pkg.crossplane.io/v1 false ProviderRevision providers pkg.crossplane.io/v1 false Provider storeconfigs secrets.crossplane.io/v1alpha1 false StoreConfig

https://docs.crossplane.io/latest/getting-started/provider-aws/

Currently there are two crossplane providers for AWS:

• https://github.com/crossplane-contrib/provider-aws (legacy now)
• https://github.com/upbound/provider-aws

The first is written "by hand" and supports around 200 Managed Resources and the other is generated using Upjet supporting the full AWS API with 924 Managed Resources.

The second provider, which is also called the Upbound "official" provider has been donated to the OpenSource community in September 2023 - so there are now 2 OpenSource providers for AWS.

Which one should you choose? This issue clarifies it:

"Upbound (the company behind crossplane) has moved to its own Terraform-based set of providers. This means that https://github.com/crossplane-contrib/provider-aws is now only maintained by community volunteers since there is a number of people out there who are still using it. It is kind of legacy but it will receive further updates as long as the community is willing to contribute to it.

Until June 2023 the Upbound providers had a problem. As Managed Resources are CRDs, the Upbound provider introduced a problem to most control planes, since Kubernetes wasn't designed for that amount of CRDs. I gave a talk about crossplane at our local DevOps Meetup and already struggled with the Upbound AWS provider.

Therefore in June 2023 the Upjet generated official Upbound provider has been split into Provider Families: https://blog.crossplane.io/crd-scaling-provider-families/ These split up Upbound providers are the way to go if you start using crossplane today:

The migration guide also states that the monolithic provider is already deprecated:

"Warning: The monolithic AWS provider (upbound/provider-aws) has been deprecated in favor of the AWS provider family. You can read more about the provider families in our blog post and the official documentation for the provider families is here. We will continue support for the monolithic AWS provider until June 12, 2024. https://marketplace.upbound.io/providers/upbound/provider-aws/"

So head over to the Upbound marketplace and search for your provider familiy. As we want to use AWS S3 for example, here we went with https://marketplace.upbound.io/providers/upbound/provider-aws-s3.

https://marketplace.upbound.io/providers/upbound/provider-aws-s3

Now we can install the Upbound AWS Provider. Therefore the upbound/provider-aws-s3/config/provider-aws-s3.yaml looks only slightly different compared to the classic AWS Provider - only the spec.package changed:

apiVersion: pkg.crossplane.io/v1 kind: Provider metadata: name: upbound-provider-aws-s3 spec: package: xpkg.upbound.io/upbound/provider-aws-s3:v0.46.0 packagePullPolicy: Always revisionActivationPolicy: Automatic revisionHistoryLimit: 1

Install it via kubectl:

kubectl apply -f upbound/provider-aws-s3/config/provider-aws-s3.yaml

The package version in combination with the packagePullPolicy configuration here is crucial, since we can configure an update strategy for the Provider here. I'am not sure, if the Crossplane team will provide an installation method where we can use tools like Renovate to keep our Crossplane providers up to date (now Renovate supports Crossplane, see paragraph Provider & Configuration Package Upgrades with Renovate). A full table of all possible fields can be found in the docs: https://crossplane.io/docs/v1.8/concepts/packages.html#specpackagepullpolicy We can also let crossplane itself manage new versions for us. But we also leave the GitOps way here!

Multiple Provider Package revisions can also be installed at the same time (especially when using packagePullPolicy: Always). If you installed multiple package versions, you'll see them as providerrevision.pkg.x when running kubectl get crossplane:

$ kubectl get crossplane ... NAME HEALTHY REVISION IMAGE STATE DEP-FOUND DEP-INSTALLED AGE providerrevision.pkg.crossplane.io/upbound-provider-aws-s3-4c95b368de88 True 1 xpkg.upbound.io/upbound/provider-aws-s3:v1.2.1 Active 1 1 3d23h providerrevision.pkg.crossplane.io/upbound-provider-aws-s3-d6a6663caff3 True 1 xpkg.upbound.io/upbound/provider-aws-s3:v1.1.0 Inactive 1 1 5d22h ...

As our first Crossplane Provider has been installed. We need to wait for the Provider to become healthy:

kubectl wait "providers.pkg.crossplane.io/upbound-provider-aws-s3" --for=condition=Healthy --timeout=180s

Obtain the status via kubectl get provider.pkg.crossplane.io:

$ kubectl get provider.pkg.crossplane.io NAME INSTALLED HEALTHY PACKAGE AGE upbound-provider-aws-s3 True True xpkg.upbound.io/upbound/provider-aws-s3:v0.46.0 113s upbound-provider-family-aws True True xpkg.upbound.io/upbound/provider-family-aws:v0.46.0 108s

Otherwise we may run into errors like this when applying the ProviderConfig right after the Provider.

I assume here that you have aws CLI installed and configured. So that the command aws configure should work on your system.

With this prepared we can create an aws-creds.conf file:

echo "[default] aws_access_key_id = $(aws configure get aws_access_key_id) aws_secret_access_key = $(aws configure get aws_secret_access_key) " > aws-creds.conf

Don't ever check this file into source control - it holds your AWS credentials! Add it to your .gitignore file right now:

# Exclude credential configuration files like aws-creds.conf *-creds.conf

If you're using a CI system like GitHub Actions (as this repository is based on), you need to have both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY configured as Repository Secrets:

Also make sure to have your Default region configured locally - or as a env: variable in your CI system. All three needed variables in GitHub Actions for example look like this:

env: AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} AWS_DEFAULT_REGION: 'eu-central-1'

Now we need to use the aws-creds.conf file to create the Crossplane AWS Provider secret:

kubectl create secret generic aws-creds -n crossplane-system --from-file=creds=./aws-creds.conf

If everything went well there should be a new aws-creds Secret ready:

To get our Provider finally working we also need to create a ProviderConfig accordingly that will tell the Provider where to find it's AWS credentials. Therefore we create a upbound/provider-aws-s3/config/provider-config-aws.yaml:

apiVersion: aws.upbound.io/v1beta1 kind: ProviderConfig metadata: name: default spec: credentials: source: Secret secretRef: namespace: crossplane-system name: aws-creds key: creds

Crossplane resources use the ProviderConfig named default if no specific ProviderConfig is specified, so this ProviderConfig will be the default for all AWS resources.

The secretRef.name and secretRef.key has to match the fields of the already created Secret.

Apply it via kubectl:

kubectl apply -f upbound/provider-aws-s3/config/provider-config-aws.yaml

Now we should have everything in place to use the Upbound AWS Provider! We can double check via kubectl get crossplane:

$ kubectl get crossplane NAME AGE providerconfig.aws.upbound.io/default 55s NAME HEALTHY REVISION IMAGE STATE DEP-FOUND DEP-INSTALLED AGE providerrevision.pkg.crossplane.io/upbound-provider-aws-s3-4c95b368de88 True 1 xpkg.upbound.io/upbound/provider-aws-s3:v1.2.1 Active 1 1 3d23h NAME INSTALLED HEALTHY PACKAGE AGE provider.pkg.crossplane.io/upbound-provider-aws-s3 True True xpkg.upbound.io/upbound/provider-aws-s3:v1.2.1 3d23h NAME AGE TYPE DEFAULT-SCOPE storeconfig.secrets.crossplane.io/default 4d Kubernetes crossplane-system

Now we should have some more Kubernetes API resources available:

$ kubectl api-resources | grep aws providerconfigs aws.upbound.io/v1beta1 false ProviderConfig providerconfigusages aws.upbound.io/v1beta1 false ProviderConfigUsage storeconfigs aws.upbound.io/v1alpha1 false StoreConfig bucketaccelerateconfigurations s3.aws.upbound.io/v1beta1 false BucketAccelerateConfiguration bucketacls s3.aws.upbound.io/v1beta1 false BucketACL bucketanalyticsconfigurations s3.aws.upbound.io/v1beta1 false BucketAnalyticsConfiguration bucketcorsconfigurations s3.aws.upbound.io/v1beta1 false BucketCorsConfiguration bucketintelligenttieringconfigurations s3.aws.upbound.io/v1beta1 false BucketIntelligentTieringConfiguration bucketinventories s3.aws.upbound.io/v1beta1 false BucketInventory bucketlifecycleconfigurations s3.aws.upbound.io/v1beta1 false BucketLifecycleConfiguration bucketloggings s3.aws.upbound.io/v1beta1 false BucketLogging bucketmetrics s3.aws.upbound.io/v1beta1 false BucketMetric bucketnotifications s3.aws.upbound.io/v1beta1 false BucketNotification bucketobjectlockconfigurations s3.aws.upbound.io/v1beta1 false BucketObjectLockConfiguration bucketobjects s3.aws.upbound.io/v1beta1 false BucketObject bucketownershipcontrols s3.aws.upbound.io/v1beta1 false BucketOwnershipControls bucketpolicies s3.aws.upbound.io/v1beta1 false BucketPolicy bucketpublicaccessblocks s3.aws.upbound.io/v1beta1 false BucketPublicAccessBlock bucketreplicationconfigurations s3.aws.upbound.io/v1beta1 false BucketReplicationConfiguration bucketrequestpaymentconfigurations s3.aws.upbound.io/v1beta1 false BucketRequestPaymentConfiguration buckets s3.aws.upbound.io/v1beta1 false Bucket bucketserversideencryptionconfigurations s3.aws.upbound.io/v1beta1 false BucketServerSideEncryptionConfiguration bucketversionings s3.aws.upbound.io/v1beta1 false BucketVersioning bucketwebsiteconfigurations s3.aws.upbound.io/v1beta1 false BucketWebsiteConfiguration objectcopies s3.aws.upbound.io/v1beta1 false ObjectCopy objects s3.aws.upbound.io/v1beta1 false Object

The crossplane core Controller and the Provider AWS Controller should now be ready to provision any infrastructure component in AWS!

You heard right: we don't create a Kubernetes based infrastructure component - but we start with a simple S3 Bucket.

Therefore we can have a look into the Crossplane AWS provider API docs:https://doc.crds.dev/github.com/upbound/provider-aws/s3.aws.upbound.io/Bucket/v1beta1@v0.45.0

https://docs.crossplane.io/latest/concepts/composite-resource-definitions/

A CompositeResourceDefinition (or XRD) defines the type and schema of your XR. It lets Crossplane know that you want a particular kind of XR to exist, and what fields that XR should have.

Since defining your own CompositeResourceDefinitions and Compositions is the main work todo with Crossplane, it's always good to know the full Reference documentation which can be found here https://docs.crossplane.io/latest/concepts/compositions/

One of the things to know is that Crossplane automatically injects some common 'machinery' into the manifests of the XRDs and Compositions: https://docs.crossplane.io/latest/concepts/composite-resources/

All possible fields an XRD can have are documented in the docs.

The field spec.versions.schema must contain a OpenAPI schema, which is similar to the ones used by any Kubernetes CRDs. They determine what fields the XR (and Claim) will have. The full CRD documentation and a guide on how to write OpenAPI schemas could be found in the Kubernetes docs.

Note that Crossplane will be automatically extended this section. Therefore the following fields are used by Crossplane and will be ignored if they're found in the schema:

spec.resourceRef spec.resourceRefs spec.claimRef spec.writeConnectionSecretToRef status.conditions status.connectionDetails 

So our Composite Resource Definition (XRD) for our S3 Bucket could look like crossplane-contrib/provider-aws/s3/definition.yaml:

apiVersion: apiextensions.crossplane.io/v1 kind: CompositeResourceDefinition metadata: # XRDs must be named 'x<plural>.<group>' name: xobjectstorages.crossplane.jonashackt.io spec: # This XRD defines an XR in the 'crossplane.jonashackt.io' API group. # The XR or Claim must use this group together with the spec.versions[0].name as it's apiVersion, like this: # 'crossplane.jonashackt.io/v1alpha1' group: crossplane.jonashackt.io # XR names should always be prefixed with an 'X' names: kind: XObjectStorage plural: xobjectstorages # This type of XR offers a claim, which should have the same name without the 'X' prefix claimNames: kind: ObjectStorage plural: objectstorages # default Composition when none is specified (must match metadata.name of a provided Composition) # e.g. in composition.yaml defaultCompositionRef: name: objectstorage-composition versions: - name: v1alpha1 served: true referenceable: true # OpenAPI schema (like the one used by Kubernetes CRDs). Determines what fields # the XR (and claim) will have. Will be automatically extended by crossplane. # See https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/ # for full CRD documentation and guide on how to write OpenAPI schemas schema: openAPIV3Schema: type: object properties: spec: type: object # We define 2 needed parameters here one has to provide as XR or Claim spec.parameters properties: parameters: type: object properties: bucketName: type: string region: type: string required: - bucketName - region

Install the XRD into our cluster with:

kubectl apply -f crossplane-contrib/provider-aws/s3/definition.yaml

We can double check the CRDs beeing created with kubectl get crds and filter them using grep to our group name crossplane.jonashackt.io:

$ kubectl get crds | grep crossplane.jonashackt.io objectstorages.crossplane.jonashackt.io 2022-06-27T09:54:18Z xobjectstorages.crossplane.jonashackt.io 2022-06-27T09:54:18Z

The main work in Crossplane has to be done crafting the Compositions. This is because they interact with the infrastructure primitives the cloud provider APIs provide.

Detailled docs to many of the possible manifest configurations can be found here https://docs.crossplane.io/latest/concepts/compositions/

aws/aws-cdk#25288 (comment)

https://doc.crds.dev/github.com/crossplane/provider-aws/s3.aws.crossplane.io/Bucket/v1beta1@v0.39.0

https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-policy-language-overview.html

https://stackoverflow.com/questions/76097031/aws-s3-bucket-cannot-have-acls-set-with-objectownerships-bucketownerenforced-s

https://marketplace.upbound.io/providers/upbound/provider-aws/v0.34.0/resources/s3.aws.upbound.io/BucketPublicAccessBlock/v1beta1

According to hashicorp/terraform-provider-aws#28353 and https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_acl we need to separate Bucket creation from BucketPublicAccessBlock, BucketOwnershipControls and BucketACL - which should now be available finally leveraging the Upbound AWS Provider:

apiVersion: apiextensions.crossplane.io/v1 kind: Composition metadata: name: objectstorage-composition labels: # An optional convention is to include a label of the XRD. This allows # easy discovery of compatible Compositions. crossplane.io/xrd: xobjectstorages.crossplane.jonashackt.io # The following label marks this Composition for AWS. This label can  # be used in 'compositionSelector' in an XR or Claim. provider: aws spec: # Each Composition must declare that it is compatible with a particular type # of Composite Resource using its 'compositeTypeRef' field. The referenced # version must be marked 'referenceable' in the XRD that defines the XR. compositeTypeRef: apiVersion: crossplane.jonashackt.io/v1alpha1 kind: XObjectStorage # When an XR is created in response to a claim Crossplane needs to know where # it should create the XR's connection secret. This is configured using the # 'writeConnectionSecretsToNamespace' field. writeConnectionSecretsToNamespace: crossplane-system # Each Composition must specify at least one composed resource template. resources: # Providing a unique name for each entry is good practice. # Only identifies the resources entry within the Composition. Required in future crossplane API versions. - name: bucket base: # see https://marketplace.upbound.io/providers/upbound/provider-aws/v0.34.0/resources/s3.aws.upbound.io/Bucket/v1beta1 apiVersion: s3.aws.upbound.io/v1beta1 kind: Bucket metadata: {} spec: deletionPolicy: Delete patches: - fromFieldPath: "spec.parameters.bucketName" toFieldPath: "metadata.name" - fromFieldPath: "spec.parameters.region" toFieldPath: "spec.forProvider.region" - name: bucketpublicaccessblock base: # see https://marketplace.upbound.io/providers/upbound/provider-aws/v0.34.0/resources/s3.aws.upbound.io/BucketPublicAccessBlock/v1beta1 apiVersion: s3.aws.upbound.io/v1beta1 kind: BucketPublicAccessBlock spec: forProvider: blockPublicAcls: false blockPublicPolicy: false ignorePublicAcls: false restrictPublicBuckets: false patches: - fromFieldPath: "spec.parameters.bucketPABName" toFieldPath: "metadata.name" - fromFieldPath: "spec.parameters.bucketName" toFieldPath: "spec.forProvider.bucketRef.name" - fromFieldPath: "spec.parameters.region" toFieldPath: "spec.forProvider.region" - name: bucketownershipcontrols base: # see https://marketplace.upbound.io/providers/upbound/provider-aws/v0.34.0/resources/s3.aws.upbound.io/BucketOwnershipControls/v1beta1#doc:spec-forProvider-rule-objectOwnership apiVersion: s3.aws.upbound.io/v1beta1 kind: BucketOwnershipControls spec: forProvider: rule: - objectOwnership: ObjectWriter patches: - fromFieldPath: "spec.parameters.bucketOSCName" toFieldPath: "metadata.name" - fromFieldPath: "spec.parameters.bucketName" toFieldPath: "spec.forProvider.bucketRef.name" - fromFieldPath: "spec.parameters.region" toFieldPath: "spec.forProvider.region" - name: bucketacl base: # see https://marketplace.upbound.io/providers/upbound/provider-aws/v0.34.0/resources/s3.aws.upbound.io/BucketACL/v1beta1 apiVersion: s3.aws.upbound.io/v1beta1 kind: BucketACL spec: forProvider: acl: "public-read" patches: - fromFieldPath: "spec.parameters.bucketAclName" toFieldPath: "metadata.name" - fromFieldPath: "spec.parameters.bucketName" toFieldPath: "spec.forProvider.bucketRef.name" - fromFieldPath: "spec.parameters.region" toFieldPath: "spec.forProvider.region" - name: bucketwebsiteconfiguration base: # see https://marketplace.upbound.io/providers/upbound/provider-aws/v0.34.0/resources/s3.aws.upbound.io/BucketWebsiteConfiguration/v1beta1 apiVersion: s3.aws.upbound.io/v1beta1 kind: BucketWebsiteConfiguration spec: forProvider: indexDocument: - suffix: index.html patches: - fromFieldPath: "spec.parameters.bucketWebConfName" toFieldPath: "metadata.name" - fromFieldPath: "spec.parameters.bucketName" toFieldPath: "spec.forProvider.bucketRef.name" - fromFieldPath: "spec.parameters.region" toFieldPath: "spec.forProvider.region" # If you find yourself repeating patches a lot you can group them as a named # 'patch set' then use a PatchSet type patch to reference them. #patchSets:

Let's create Composition via:

kubectl apply -f upbound/provider-aws-s3/composition.yaml

Crossplane could look quite intimidating when having a first look. There are few guides around to show how to approach a setup when using Crossplane the first time. You can choose between writing an XR OR XRC! You don't need both, since the XR will be generated from the XRC, if you choose to craft a XRC.

https://docs.crossplane.io/v1.14/concepts/composite-resources/

Since we want to create a S3 Bucket, here's an suggestion for an claim.yaml:

# Use the spec.group/spec.versions[0].name defined in the XRD apiVersion: crossplane.jonashackt.io/v1alpha1 kind: ObjectStorage metadata: # Only claims are namespaced, unlike XRs. namespace: default name: managed-s3 spec: # The compositionRef specifies which Composition this XR will use to compose # resources when it is created, updated, or deleted. compositionRef: name: objectstorage-composition # Parameters for the Composition to provide the Managed Resources (MR) with # to create the actual infrastructure components parameters: bucketName: microservice-ui-nuxt-js-static-bucket2 region: eu-central-1

Testdrive with:

kubectl apply -f crossplane-contrib/provider-aws/s3/claim.yaml

When somthing goes wrong with the validation, this could look like this:

$ kubectl apply -f claim.yaml error: error validating "claim.yaml": error validating data: [ValidationError(S3Bucket.metadata): unknown field "crossplane.io/external-name" in io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta_v2, ValidationError(S3Bucket.spec): unknown field "parameters" in io.jonashackt.crossplane.v1alpha1.S3Bucket.spec, ValidationError(S3Bucket.spec.writeConnectionSecretToRef): missing required field "namespace" in io.jonashackt.crossplane.v1alpha1.S3Bucket.spec.writeConnectionSecretToRef, ValidationError(S3Bucket.spec): missing required field "bucketName" in io.jonashackt.crossplane.v1alpha1.S3Bucket.spec, ValidationError(S3Bucket.spec): missing required field "region" in io.jonashackt.crossplane.v1alpha1.S3Bucket.spec]; if you choose to ignore these errors, turn validation off with --validate=false

The Crossplane validation is a great way to debug your yaml configuration - it hints you to the actual problems that are still present.

https://docs.crossplane.io/knowledge-base/guides/troubleshoot/

Per Kubernetes convention, Crossplane keeps errors close to the place they happen. This means that if your claim is not becoming ready due to an issue with your Composition or with a composed resource you’ll need to “follow the references” to find out why. Your claim will only tell you that the XR is not yet ready.

The docs also tell us what they mean by "follow the references":

• Find your XR by running kubectl describe <claim-kind> <claim-metadata.name> and look for its “Resource Ref” (aka spec.resourceRef).
• Run kubectl describe on your XR. This is where you’ll find out about issues with the Composition you’re using, if any.
• If there are no issues but your XR doesn’t seem to be becoming ready, take a look for the “Resource Refs” (or spec.resourceRefs) to find your composed resources.
• Run kubectl describe on each referenced composed resource to determine whether it is ready and what issues, if any, it is encountering.

There are some possible things to check while your resources (may) get deployed after running a kubectl apply -f claim.yaml (see https://crossplane.io/docs/v1.8/getting-started/provision-infrastructure.html#claim-your-infrastructure).

The best overview gives a kubectl get crossplane which will simply list all the crossplane resources:

$ kubectl get crossplane Warning: Please use v1beta1 version of this resource that has identical schema. NAME ESTABLISHED OFFERED AGE compositeresourcedefinition.apiextensions.crossplane.io/xs3buckets.crossplane.jonashackt.io True True 23m NAME AGE composition.apiextensions.crossplane.io/s3bucket 2d17h NAME INSTALLED HEALTHY PACKAGE AGE provider.pkg.crossplane.io/provider-aws True True crossplanecontrib/provider-aws:v0.34.0 4d21h NAME HEALTHY REVISION IMAGE STATE DEP-FOUND DEP-INSTALLED AGE providerrevision.pkg.crossplane.io/provider-aws-2189bc61e0bd True 1 crossplanecontrib/provider-aws:v0.34.0 Active 4d21h NAME AGE TYPE DEFAULT-SCOPE storeconfig.secrets.crossplane.io/default 5d23h Kubernetes crossplane-system

• kubectl get claim: get all resources of all claim kinds, like PostgreSQLInstance.
• kubectl get composite: get all resources that are of composite kind, like XPostgreSQLInstance.
• kubectl get managed: get all resources that represent a unit of external infrastructure.
• kubectl get <name-of-provider>: get all resources related to .
• kubectl get crossplane: get all resources related to Crossplane.

We can also check our claim with kubectl get <claim-kind> <claim-metadata.name> like this:

$ kubectl get ObjectStorage managed-s3 NAME READY CONNECTION-SECRET AGE managed-s3 managed-s3-connection-details 5s

To watch the provisioned resources become ready we can run kubectl get crossplane -l crossplane.io/claim-name=<claim-metadata.name>:

kubectl get crossplane -l crossplane.io/claim-name=managed-s3

Check if the S3 Bucket has been created successfully via aws CLI with aws s3 ls | grep microservice-ui-nuxt-js-static-bucket2.

$ aws s3 ls | grep microservice-ui-nuxt-js-static-bucket2 2022-06-27 11:56:26 microservice-ui-nuxt-js-static-bucket2

Our bucket should be there! We can also double check in the AWS console:

Let's deploy our app (a simple index.html) to our S3 Bucket using the aws CLI like this:

aws s3 sync static s3://devopsthde-bucket --acl public-read

Now we can open up http://microservice-ui-nuxt-js-static-bucket2.s3-website.eu-central-1.amazonaws.com/ in our Browser and should see our app beeing deployed:

Before removing the claim, we should remove our index.html - otherwise we'll run into errors like this:

 Warning CannotDeleteExternalResource 37s (x16 over 57s) managed/bucket.s3.aws.crossplane.io (combined from similar events): operation error S3: DeleteBucket, https response error StatusCode: 409, RequestID: 0WHR906YZRF0YDSH, HostID: x7cz2iYF/8Ag2wKtKRZUy1j3hPk67tBUOTFeR//+grrD7plqQ5Zo6EecO70KOOgHKbY7hUyp9vU=, api error BucketNotEmpty: The bucket you tried to delete is not empty

So first delete the index.html:

aws s3 rm s3://devopsthde-bucket/index.html

Finally remove our S3 Bucket again with

kubectl delete -f upbound/provider-aws-s3/claim.yaml

Now also the S3 Bucket should be removed by crossplane.

https://docs.crossplane.io/v1.14/getting-started/provider-azure/

https://marketplace.upbound.io/providers/upbound/provider-azure-storage/v0.39.0

https://docs.crossplane.io/latest/getting-started/provider-azure/#create-a-kubernetes-secret-for-azure

I assume here that you have azure CLI installed and you're logged in to your Azure subscription via az login. So that the command az account show should work on your system.

The current Crossplane docs on Azure propose a az ad sp create-for-rbac command that isn't working: ERROR: Usage error: To create role assignments, specify both --role and --scopes. When using --role we must use --scopes also. The documentation about the latter isn't easy to grasp https://docs.microsoft.com/en-us/cli/azure/ad/sp?view=azure-cli-latest#az-ad-sp-create-for-rbac at first sight. As there are examples like --scopes /subscriptions/{subscriptionId}/resourceGroups/{resourceGroup1} I thought I need to create a resourceGroup before even connecting to Azure to create a resourceGroup with Crossplane (that doesn't make any sense). BUT: It's possible to use --scopes /subscriptions/{subscriptionId} here without any resourceGroups, which will create a scope for your whole subscription.

So prepare the SUBSCRIPTION_ID variable like this:

export SUBSCRIPTION_ID=$(az account show --query id --output tsv)

With this prepared we can create a service principal in Azure AD using the following command:

az ad sp create-for-rbac --sdk-auth --role Owner --scopes /subscriptions/$SUBSCRIPTION_ID --name servicePrincipalCrossplaneGHActions > crossplane-azure-provider-key.json

Although this produces a warning message like WARNING: Option '--sdk-auth' has been deprecated and will be removed in a future release. we definitely need to use the --sdk-auth parameter. Otherwise our Managed Resources will run into errors like connect failed: cannot get authorizer from client credentials config: failed to get SPT from client credentials: parameter 'activeDirectoryEndpoint' cannot be empty because there are configuration entries missing in the file like "activeDirectoryEndpointUrl": "https://login.microsoftonline.com",.

This produces a crossplane-azure-provider-key.json file you should never ever check into version control! For this repository I added it to the .gitignore file.

If you're using a CI system like GitHub Actions (as this repository is based on), define 3 repository secrets from the output. Choose the appId as the ARM_CLIENT_ID, the password as the ARM_CLIENT_SECRET and the tenant as the ARM_TENANT_ID. Additionally we need to define the ARM_SUBSCRIPTION_ID secret. Therefore run a az account show (after you logged your CLI into your Azure subscription via azure login) and use the value of "id":.

On GitHub Actions we need to

echo "{  \"clientId\": \"$ARM_CLIENT_ID\",  \"clientSecret\": \"$ARM_CLIENT_SECRET\",  \"subscriptionId\": \"$ARM_SUBSCRIPTION_ID\",  \"tenantId\": \"$ARM_TENANT_ID\",  \"activeDirectoryEndpointUrl\": \"https://login.microsoftonline.com\",  \"resourceManagerEndpointUrl\": \"https://management.azure.com/\",  \"activeDirectoryGraphResourceId\": \"https://graph.windows.net/\",  \"sqlManagementEndpointUrl\": \"https://management.core.windows.net:8443/\",  \"galleryEndpointUrl\": \"https://gallery.azure.com/\",  \"managementEndpointUrl\": \"https://management.core.windows.net/\" }" > crossplane-azure-provider-key.json

All three needed variables in GitHub Actions for example look like this:

env: ARM_CLIENT_ID: ${{ secrets.ARM_CLIENT_ID }} ARM_CLIENT_SECRET: ${{ secrets.ARM_CLIENT_SECRET }} ARM_SUBSCRIPTION_ID: ${{ secrets.ARM_SUBSCRIPTION_ID }} ARM_TENANT_ID: ${{ secrets.ARM_TENANT_ID }}

Now we need to use the crossplane-azure-provider-key.json file to create the Crossplane Azure Provider secret:

kubectl create secret generic azure-account-creds -n crossplane-system --from-file=creds=./crossplane-azure-provider-key.json

If everything went well there should be a new azure-account-creds Secret ready.

https://docs.crossplane.io/latest/getting-started/provider-azure/#install-the-azure-provider

Let's create a provider-azure-storage.yaml file like this:

apiVersion: pkg.crossplane.io/v1 kind: Provider metadata: name: provider-azure-storage spec: package: xpkg.upbound.io/upbound/provider-azure-storage:v0.39.0 packagePullPolicy: Always revisionActivationPolicy: Automatic revisionHistoryLimit: 1

Install the Azure provider:

kubectl apply -f upbound/provider-azure-storage/config/provider-azure-storage.yaml

Now our first Crossplane Provider has been installed. You may check it with kubectl get provider:

$ kubectl get provider NAME INSTALLED HEALTHY PACKAGE AGE provider-azure True Unknown crossplane/provider-azure:v0.19.0 13s

Before we can actually apply a ProviderConfig to our Azure provider we have to make sure that it's actually healthy and running. Therefore we can use the kubectl wait command like this:

kubectl wait --for=condition=healthy --timeout=120s provider/provider-azure

Otherwise we may run into errors like this when applying the ProviderConfig right after the Provider.

https://docs.crossplane.io/latest/getting-started/provider-azure/#create-a-providerconfig

Now we need to create ProviderConfig object that will tell the AWS Provider where to find it's AWS credentials. Therefore we create a provider-config-azure.yaml:

apiVersion: azure.upbound.io/v1beta1 kind: ProviderConfig metadata: name: default spec: credentials: source: Secret secretRef: namespace: crossplane-system name: azure-account-creds key: creds

Crossplane resources use the ProviderConfig named default if no specific ProviderConfig is specified, so this ProviderConfig will be the default for all Azure resources.

The secretRef.name and secretRef.key has to match the fields of the already created Secret.

Apply it with:

kubectl apply -f upbound/provider-azure-storage/config/provider-config-azure.yaml

The crossplane core Controller and the Provider Azure Controller should now be ready to provision any infrastructure component in Azure!

https://doc.crds.dev/github.com/upbound/provider-azure/storage.azure.upbound.io/Account/v1beta1@v0.38.2

https://doc.crds.dev/github.com/upbound/provider-azure/azure.upbound.io/ResourceGroup/v1beta1@v0.38.2

So our Composite Resource Definition (XRD) for our Storage Account could look like upbound/provider-azure-storage/definition.yaml:

--- apiVersion: apiextensions.crossplane.io/v1 kind: CompositeResourceDefinition metadata: name: xstoragesazure.crossplane.jonashackt.io spec: group: crossplane.jonashackt.io names: kind: XStorageAzure plural: xstoragesazure claimNames: kind: StorageAzure plural: storagesazure defaultCompositionRef: name: storageazure-composition versions: - name: v1alpha1 served: true referenceable: true # See https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/ schema: openAPIV3Schema: type: object properties: spec: type: object properties: parameters: type: object properties: location: type: string resourceGroupName: type: string storageAccountName: type: string required: - location - resourceGroupName - storageAccountName

Install the XRD into our cluster with:

kubectl apply -f upbound/provider-azure-storage/definition.yaml

Let's wait for the XRD to become Offered:

kubectl wait --for=condition=Offered --timeout=120s xrd xstoragesazure.crossplane.jonashackt.io 

A Composite to manage an Storage Account in Azure with public access for static website hosting could for example look like this upbound/provider-azure-storage/composition.yaml:

--- apiVersion: apiextensions.crossplane.io/v1 kind: Composition metadata: name: storageazure-composition labels: crossplane.io/xrd: xstoragesazure.crossplane.jonashackt.io provider: azure spec: compositeTypeRef: apiVersion: crossplane.jonashackt.io/v1alpha1 kind: XStorageAzure writeConnectionSecretsToNamespace: crossplane-system resources: - name: storageaccount base: # see https://doc.crds.dev/github.com/crossplane/provider-azure/storage.azure.crossplane.io/Account/v1alpha3@v0.19.0 apiVersion: storage.azure.upbound.io/v1beta1 kind: Account metadata: {} spec: forProvider: accountKind: StorageV2 accountTier: Standard accountReplicationType: LRS patches: - fromFieldPath: spec.parameters.storageAccountName toFieldPath: metadata.name - fromFieldPath: spec.parameters.resourceGroupName toFieldPath: spec.forProvider.resourceGroupName - fromFieldPath: spec.parameters.location toFieldPath: spec.forProvider.location - name: resourcegroup base: # see https://doc.crds.dev/github.com/crossplane/provider-azure/azure.crossplane.io/ResourceGroup/v1alpha3@v0.19.0 apiVersion: azure.upbound.io/v1beta1 kind: ResourceGroup metadata: {} patches: - fromFieldPath: spec.parameters.resourceGroupName toFieldPath: metadata.name - fromFieldPath: spec.parameters.location toFieldPath: spec.forProvider.location

Install our Composition with

kubectl apply -f upbound/provider-azure-storage/composition.yaml

Since we want to create a Storage Account, here's an suggestion for an claim.yaml:

--- apiVersion: crossplane.jonashackt.io/v1alpha1 kind: StorageAzure metadata: namespace: default name: managed-storage-account spec: compositionRef: name: storageazure-composition parameters: location: West Europe resourceGroupName: rg-crossplane storageAccountName: account4c8672f

Testdrive with

kubectl apply -f upbound/provider-azure-storage/claim.yaml

We can use the new trace command of the crossplane CLI introduced in 1.14 to have a look, what's going on with our Azure resources:

$ crossplane beta trace storageazure.crossplane.jonashackt.io/managed-storage-account NAME SYNCED READY STATUS StorageAzure/account (default) True False Waiting: ...resource claim is waiting for composite resource to become Ready └─ XStorageAzure/account-g97s8 False - ReconcileError: ... "object": spec.forProvider.accountTier is a required parameter] ├─ Account/account4c8672f - - Error: accounts.storage.azure.upbound.io "account4c8672f" not found └─ ResourceGroup/rg-crossplane - - Error: resourcegroups.azure.upbound.io "rg-crossplane" not found 

Ahh, so our Composition isn't crafted correctly.

To fix this, we need to have a look into the API docs at https://doc.crds.dev/github.com/upbound/provider-azure/storage.azure.upbound.io/Account/v1beta1@v0.38.2. After fixing our Composition and reapplying it together with the Claim, we can have a look into the Azure Portal. Our Resource Group should show up:

And also our Storage account should be visible inside the group:

Let's package our Composite Resource definitions as a Configuration. Therefore we need to use the Crossplane CLI via kubectl crossplane build configuration (now they are a Package) - and push them to an OCI registry via kubectl crossplane push configuration. With this Configurations can also be easily installed into other Crossplane clusters.

Therefore we need a crossplane.yaml file as described in https://crossplane.io/docs/v1.8/getting-started/create-configuration.html#build-and-push-the-configuration

See also https://crossplane.io/docs/v1.8/concepts/packages.html#configuration-packages

Our crossplane-contrib/provider-aws/s3/crossplane.yaml is of kind: Configuration and defines the minimum crossplane version needed alongside the crossplane AWS provider:

apiVersion: meta.pkg.crossplane.io/v1 kind: Configuration metadata: name: s3-bucket-composition spec: crossplane: version: ">=v1.8" dependsOn: - provider: crossplanecontrib/provider-aws version: ">=v0.33.0"

Having this crossplane.yaml in place we can build our Configuration at last. On a command line go into the directory where the crossplane.yaml resides and run the kubectl crossplane build command:

$ cd crossplane-s3 $ kubectl crossplane build configuration

Really strange, getting

kubectl crossplane build configuration kubectl crossplane: error: failed to build package: failed to parse package: {path:/Users/jonashecht/dev/kubernetes/crossplane-awws-azure/crossplane-contrib/provider-aws/s3/composition.yaml position:0}: no kind "S3Bucket" is registered for version "crossplane.jonashackt.io/v1alpha1" in scheme "/home/runner/work/crossplane/crossplane/internal/xpkg/scheme.go:47"

The crossplane core Controller and the Provider AWS Controller should now be ready to provision any infrastructure component in AWS!

So in order to maximise the Inception let's provision an EKS based Kubernetes cluster in AWS with our crossplane Kubernetes cluster :)

A EKS cluster is a complex AWS construct leveraging different basic AWS services. That's exactly what a crossplane Composite Resource (XR) is all about. So let's create our own Composite Resource.

Looking only at the Crossplane docs and blog I thought something is missing: A curated library of higher level abstractions (like Pulumi Crosswalk). First initiatives from cloud vendors like AWS Blueprints for Crossplane: https://aws.amazon.com/blogs/opensource/introducing-aws-blueprints-for-crossplane/

But then I found the Upbound blog - and finally stumbled upon the Upbound Platform Reference Architectures https://blog.upbound.io/azure-reference-platform/ which are intended as

foundations to help accelerate understanding and application of Crossplane within your organization

What I didn't wanted to do is to duplicate some code into my Composition from all those sources I found (mainly this https://www.kloia.com/blog/production-ready-eks-cluster-with-crossplane, this https://aws.amazon.com/blogs/containers/gitops-model-for-provisioning-and-bootstrapping-amazon-eks-clusters-using-crossplane-and-argo-cd/ and https://aws.amazon.com/blogs/opensource/introducing-aws-blueprints-for-crossplane/) about provisioning a EKS cluster with crossplane.

https://github.com/upbound/platform-ref-aws#install-the-platform-configuration

See this blog post https://blog.upbound.io/first-official-providers/ and the paragraph Provider Coverage Problem:

The Crossplane community has been adding CustomResourceDefinitions (CRDs) to cover the API surface, but the pace is unable to catch up with users who are blocked from getting into production with Crossplane due to the lack of coverage. To give you an idea, AWS has ~1000 resources and the classic AWS provider has yet to cover ~200 as of writing.

Now with the classic AWS provider https://marketplace.upbound.io/providers/crossplane-contrib/provider-aws/v0.39.0/crds you have around 180 CRDs available. If you only need the CRDs that are provided by this classic provider, everything is okay. But I recently stumbled upon a new security default implementation of AWS S3: #12

This project stopped to work, simply since AWS doesn't support Access Control Lists (ACLs) for all new S3 Buckets anymore. See hashicorp/terraform-provider-aws#28353

Starting in April 2023, Amazon S3 will introduce two new default bucket security settings by automatically enabling S3 Block Public Access and disabling S3 access control lists (ACLs) for all new S3 buckets.

This in fact means, that we not only need the Bucket CRD, but we additionally need BucketACL, BucketPublicAccessBlock and BucketOwnershipControls, which are only available in the so called official Upbound providers with their 901 CRDs: https://marketplace.upbound.io/providers/upbound/provider-aws/v0.34.0/crds

Up until now I didn't really know the difference between the classic providers and the official providers. But the official providers are based on the tool Upjet https://github.com/upbound/upjet/ which is the successor of the former Terrajet https://github.com/crossplane/terrajet, where Terraform modules are used to generate Crossplane CRDs. With this approach, every API Terraform provides is also available in Crossplane. Upjet is currently used to generate all CRDs for the official AWS, Azure and GCP providers.

And with Upjet it's possible to generate really any Crossplane provider from Terraform, meaning one can rely on every Provider Terraform already has: https://registry.terraform.io/browse/providers

Why not use Terraform then you may ask. Theres a good post here https://blog.crossplane.io/crossplane-vs-terraform/ which makes the point, that Terraform can only be used from a bash script - but Crossplane delivers a control plane based approach, which continously reconciles all infrastructure configured inside of it.

So TLDR: To fix the error #12

S3 creation error: CannotCreateExternalResource managed/bucket.s3.aws.crossplane.io failed to create the Bucket: InvalidBucketAclWithObjectOwnership: Bucket cannot have ACLs set with ObjectOwnership's BucketOwnerEnforced setting

we need to switch over to the official AWS provider based on Upjet. Do we need to use the UXP (Universal Control Plane) flavour of Crossplane for that? No, the docs https://marketplace.upbound.io/providers/upbound/provider-aws/v0.34.0/docs/configuration state:

The Upbound AWS official provider may also be used with upstream Crossplane.

So let's use the provider inside our project!

https://blog.upbound.io/moving-crossplane-package-authoring-from-plain-yaml-to-ide-aided-development/

Upbound VS Code plugin https://blog.upbound.io/crossplane-vscode-plugin-announcement/

https://crossplane.io/

Intro post https://medium.com/@khelge/crossplane-7197ad200013

Crossplane is based on OAM (3 roles: Developer, application operator & infrastructure operator) https://github.com/oam-dev/spec

https://www.kloia.com/blog/production-ready-eks-cluster-with-crossplane

https://www.forbes.com/sites/janakirammsv/2021/09/15/how-crossplane-transforms-kubernetes-into-a-universal-control-plane/

Crossplane essentially transforms Kubernetes into a universal control plane that can orchestrate the lifecycle of public cloud-based services such as virtual machines, database instances, Big Data clusters, machine learning jobs, and other managed services offered by the hyperscalers.

Crossplane takes the concept of Infrastructure as Code (IaC) to the next level through its tight integration with Kubernetes.

Intro Slides: https://docs.google.com/presentation/d/1PxZweRpB6HElxd9qGK1McboGZ1kluCDCS5qxgYnX5f0/edit#slide=id.g8801599ecb_2_169

Promoted from sandbox to incubation by the CNCF: https://blog.crossplane.io/crossplane-cncf-incubation/

Crossplane vs. Terraform: https://blog.crossplane.io/crossplane-vs-terraform/

Crossplane 101 https://vrelevant.net/crossplane-bringing-the-basics-together/

https://www.techtarget.com/searchitoperations/news/252508176/Crossplane-project-could-disrupt-infrastructure-as-code

Argo + crossplane https://morningspace.medium.com/using-crossplane-in-gitops-what-to-check-in-git-76c08a5ff0c4

Infrastructure-as-Apps https://codefresh.io/blog/infrastructure-as-apps-the-gitops-future-of-infra-as-code/

New Charter from Sep 2023 on: crossplane/crossplane#4643 Developer Experience and Providers are added to the scope of crossplane

Even Upjet and the major Upbound developed (aka Upjet generated) Providers are donated to the OS project & CNCF: https://blog.crossplane.io/charter-expansion-upjet-donation/

See also crossplane-contrib/provider-aws#1954

https://aws.amazon.com/blogs/containers/gitops-model-for-provisioning-and-bootstrapping-amazon-eks-clusters-using-crossplane-and-argo-cd/

Crossplane’s infrastructure provider for AWS relies on code generated by the AWS Go Code Generator, which is also used by AWS Controllers for Kubernetes (ACK).

https://www.kloia.com/blog/production-ready-eks-cluster-with-crossplane

AWS Blueprints for Crossplane:

https://aws.amazon.com/blogs/opensource/introducing-aws-blueprints-for-crossplane/

Writing your first Composition and debugging can be intimidating.

And it's quite a doublicate effort IMHO compared to other tools like Pulumi that provide higher level abstractions including well-architectured best practices (like Pulumi https://www.pulumi.com/docs/guides/crosswalk/aws/)

https://github.com/aws-samples/crossplane-aws-blueprints

Thoughworks Tech Radar: Assess https://www.thoughtworks.com/de-de/radar/tools/crossplane

IONOS and crossplane https://docs.ionos.com/crossplane-provider/example

https://github.com/upbound/platform-ref-s3-website

k8s event-log! :events and then scroll down to the newest events and have a (constant) look

Composition Updates are applied automatically to all XRs/Claims by default. As the docs state:

If you have 10 PlatformDB XRs all using the big-platform-db Composition, all 10 of those XRs will be instantly updated in accordance with any updates you make to the big-platform-db Composition.

So if we change a Composition, all the XRs will be updated - which we can see in the Kubernetes Events:

This can be a wanted behavior - or not at all.

But be aware of this error:

cannot compose resources: cannot parse base template of composed resource "securitygrouprule-cluster-inbound": cannot change the kind or group of a composed resource from ec2.aws.upbound .io/v1beta1, Kind=SecurityGroupRule to ec2.aws.upbound.io/v1beta1, Kind=SecurityGroupIngressRule (possible composed resource template mismatch) 

See crossplane/crossplane#1909

In the first steps with Crossplane you might encounter the issue already:

cannot compose resources: cannot parse base template of composed resource "securitygrouprule-cluster-inbound": cannot change the kind or group of a composed resource from ec2.aws.upbound .io/v1beta1, Kind=SecurityGroupRule to ec2.aws.upbound.io/v1beta1, Kind=SecurityGroupIngressRule (possible composed resource template mismatch) 

Composition Revisions allow XRs to opt out of automatic updates.

If you don't want Composition to do automatic updates, you need to use Composition Revisions in your XRs/XRCs via the compositionUpdatePolicy: Manual keyword:

apiVersion: example.org/v1alpha1 kind: PlatformDB metadata: name: example spec: parameters: storageGB: 20 # The Manual policy specifies that you do not want this XR to update to the # current CompositionRevision automatically. compositionUpdatePolicy: Manual compositionRef: name: example writeConnectionSecretToRef: name: db-conn

https://docs.crossplane.io/knowledge-base/guides/troubleshoot/

https://github.com/crossplane/crossplane/releases/tag/v1.15.0

Enhancements to the Crossplane CLI: New subcommands like crossplane beta validate for schema validation, crossplane beta top for resource utilization views similar to kubectl top pods, and crossplane beta convert for converting resources to newer formats or configurations.

Install crossplane CLI:

curl -sL "https://raw.githubusercontent.com/crossplane/crossplane/master/install.sh" |sh

If that produces an error, try to manually craft the download link:

curl -sfLo crossplane "https://releases.crossplane.io/stable/v1.15.0/bin/linux_amd64/crank" sudo mv crossplane /usr/local/bin

Be sure to have the v1.15.0 version installed as a minimum, otherwise the crossplane beta validate command won't work:

crossplane --version v1.15.0

Before using the command have a look at the command reference: https://docs.crossplane.io/latest/cli/command-reference/#beta-validate:

The crossplane beta validate command validates compositions against provider or XRD schemas using the Kubernetes API server’s validation library.

So let's grab a definition.yaml and validate a claim.yaml against it:

crossplane beta validate --cache-dir ~/.crossplane definition.yaml claim.yaml [✓] crossplane.jonashackt.io/v1alpha1, Kind=ObjectStorage, managed-upbound-s3 validated successfully Total 1 resources: 0 missing schemas, 1 success cases, 0 failure cases

To prevent the command from polluting our projects with .crossplane directories, we should also provide a --cache-dir ~/.crossplane flag, which will deposit the directory in the user profile folder.

We need to provide the provider's schemes.

For example grab a Composition and validate it against the AWS provider:

crossplane beta validate --cache-dir ~/.crossplane config/provider-aws-s3.yaml resources/public-bucket.yaml package schemas does not exist, downloading: xpkg.upbound.io/upbound/provider-aws-s3:v1.1.0 [✓] s3.aws.upbound.io/v1beta1, Kind=Bucket, crossplane-argocd-s3-bucket validated successfully [✓] s3.aws.upbound.io/v1beta1, Kind=BucketPublicAccessBlock, crossplane-argocd-s3-bucket-pab validated successfully [✓] s3.aws.upbound.io/v1beta1, Kind=BucketOwnershipControls, crossplane-argocd-s3-bucket-osc validated successfully [✓] s3.aws.upbound.io/v1beta1, Kind=BucketACL, crossplane-argocd-s3-bucket-acl validated successfully [✓] s3.aws.upbound.io/v1beta1, Kind=BucketWebsiteConfiguration, crossplane-argocd-s3-bucket-websiteconf validated successfully Total 5 resources: 0 missing schemas, 5 success cases, 0 failure cases

We can also validate a full directory:

crossplane beta validate --cache-dir ~/.crossplane provider-aws-s3/definition.yaml provider-aws-s3 [✓] crossplane.jonashackt.io/v1alpha1, Kind=ObjectStorage, managed-upbound-s3 validated successfully [!] could not find CRD/XRD for: apiextensions.crossplane.io/v1, Kind=Composition [!] could not find CRD/XRD for: pkg.crossplane.io/v1, Kind=Provider [!] could not find CRD/XRD for: aws.upbound.io/v1beta1, Kind=ProviderConfig [!] could not find CRD/XRD for: apiextensions.crossplane.io/v1, Kind=CompositeResourceDefinition [✓] s3.aws.upbound.io/v1beta1, Kind=Bucket, crossplane-argocd-s3-bucket validated successfully [✓] s3.aws.upbound.io/v1beta1, Kind=BucketPublicAccessBlock, crossplane-argocd-s3-bucket-pab validated successfully [✓] s3.aws.upbound.io/v1beta1, Kind=BucketOwnershipControls, crossplane-argocd-s3-bucket-osc validated successfully [✓] s3.aws.upbound.io/v1beta1, Kind=BucketACL, crossplane-argocd-s3-bucket-acl validated successfully [✓] s3.aws.upbound.io/v1beta1, Kind=BucketWebsiteConfiguration, crossplane-argocd-s3-bucket-websiteconf validated successfully Total 10 resources: 4 missing schemas, 6 success cases, 0 failure cases

crossplane beta validate --cache-dir ~/.crossplane provider-aws-s3/config/provider-aws-s3.yaml provider-aws-s3 [!] could not find CRD/XRD for: crossplane.jonashackt.io/v1alpha1, Kind=ObjectStorage [!] could not find CRD/XRD for: apiextensions.crossplane.io/v1, Kind=Composition [!] could not find CRD/XRD for: pkg.crossplane.io/v1, Kind=Provider [!] could not find CRD/XRD for: aws.upbound.io/v1beta1, Kind=ProviderConfig [!] could not find CRD/XRD for: apiextensions.crossplane.io/v1, Kind=CompositeResourceDefinition [✓] s3.aws.upbound.io/v1beta1, Kind=Bucket, crossplane-argocd-s3-bucket validated successfully [✓] s3.aws.upbound.io/v1beta1, Kind=BucketPublicAccessBlock, crossplane-argocd-s3-bucket-pab validated successfully [✓] s3.aws.upbound.io/v1beta1, Kind=BucketOwnershipControls, crossplane-argocd-s3-bucket-osc validated successfully [✓] s3.aws.upbound.io/v1beta1, Kind=BucketACL, crossplane-argocd-s3-bucket-acl validated successfully [✓] s3.aws.upbound.io/v1beta1, Kind=BucketWebsiteConfiguration, crossplane-argocd-s3-bucket-websiteconf validated successfully Total 10 resources: 5 missing schemas, 5 success cases, 0 failure cases

https://docs.crossplane.io/latest/cli/command-reference/#beta-render

https://blog.crossplane.io/building-crossplane-composition-functions-to-empower-your-control-plane/

To be able to validate Compositions & XRs, we need another command in the game: crossplane beta render:

https://docs.crossplane.io/latest/cli/command-reference/#validate-render-command-output

You can pipe the output of crossplane beta render into crossplane beta validate to validate complete Crossplane resource pipelines, including XRs, compositions and composition functions.

Therefore we need to use the --include-full-xr command with crossplane beta render and the - option with crossplane beta validate like that:

crossplane beta render composition.yaml --include-full-xr | crossplane beta validate config/provider-aws-s3.yaml -

Imagine a provider beeing configured like this:

apiVersion: pkg.crossplane.io/v1 kind: Provider metadata: name: provider-aws-ec2 spec: package: xpkg.upbound.io/upbound/provider-aws-ec2:v1.1.0 packagePullPolicy: Always revisionActivationPolicy: Automatic revisionHistoryLimit: 1

Now if provider-aws-ec2:v1.1.1 gets released, the revisionActivationPolicy: Automatic will lead to a automatically upgraded provider.

This can be great - or it can lead to a sitation, where the new provider version doesn't work as expected (e.g. because it presents a https://docs.crossplane.io/latest/concepts/providers/#unhealthypackagerevision) - and thus beeing rolled back, also automatically. This could lead to a situation, where the provider gets unusable from time to time.

If we want to do Provider upgrades in a GitOps fashion through a git commit to a git repository, we should configure the packagePullPolicy to IfNotPresent instead of Always (which means " Check for new packages every minute and download any matching package that isn’t in the cache", see https://docs.crossplane.io/master/concepts/packages/#configuration-package-pull-policy) - BUT leave the revisionActivationPolicy to Automatic! Since otherwise, the Provider will never get active and healty! See https://docs.crossplane.io/master/concepts/packages/#revision-activation-policy), but I didn't find it documented that way!

apiVersion: pkg.crossplane.io/v1 kind: Provider metadata: name: provider-aws-ec2 spec: package: xpkg.upbound.io/upbound/provider-aws-ec2:v1.1.0 packagePullPolicy: IfNotPresent # Only download the package if it isn’t in the cache. revisionActivationPolicy: Automatic # Otherwise our Provider never gets activate & healthy revisionHistoryLimit: 1

Renovate supports Crossplane by the end of November 2023:

• renovatebot/renovate#22363
• merged PR renovatebot/renovate#25911 & docs https://github.com/renovatebot/renovate/pull/25911/commits/d224eaeaee0283282d54bc86e2b7f3e7100de455

The Renovate docs tell us how we can configure Crossplane support:

https://docs.renovatebot.com/modules/manager/crossplane/

To use the crossplane manager you must set your own fileMatch pattern. The crossplane manager has no default fileMatch pattern, because there is no common filename or directory name convention for Crossplane YAML files. The crossplane manager supports these depTypes: configuration, function, provider

So we always need to explicitely configure Renovate to let it handle our Crossplane Provider & Configuration Package updates for us!

The simplest configuration would be:

"crossplane": { "fileMatch": ["\\.yaml$"] }

It makes sense, if most of the files are for Crossplane.

https://docs.crossplane.io/latest/concepts/compositions/#resource-readiness-checks

https://docs.crossplane.io/latest/concepts/compositions/#match-a-boolean

With uptest and kuttl:

https://github.com/upbound/platform-ref-s3-website/blob/main/Makefile

https://github.com/awslabs/crossplane-on-eks?tab=readme-ov-file#features

https://github.com/awslabs/crossplane-on-eks/blob/main/doc/patching-101.md

https://github.com/crossplane/crossplane/blob/master/design/design-doc-composition.md

Note that Crossplane packages were formerly known as 'Stacks'. This term has been rescoped to refer specifically to a package of infrastructure configuration but is frequently synonymous with packages in historical design documents and some remaining Crossplane APIs.

https://kuttl.dev/

example: https://github.com/jonashackt/crossplane-kuttl

A good advice is to use unique bucket names based on the branch to prevent interfering jobs with each other. Otherwise a Renovate enabled Crossplane project using real Cloud resources will create multiple crashed jobs in a row.

But we also need to split out the branch name, see https://stackoverflow.com/a/73467112/4964553 otherwise we'll run into errors like:

compose resources: cannot associate composed resources with Composition resource templates: cannot get composed resource: invalid resource name "spring2024-bucket-renovate/xpkg.upbound.io-upbound-provider-aws-s3-1.x": [may not contain '/']'

Therefore I created a BUCKET_NAME variable and Split branch name step which uses the predefined `{{ github.ref_name }}' containing the branch name with often with a slash (when they are opened by Renovate for example):

name: provision-aws on: [push] env: KIND_NODE_VERSION: v1.29.2 BUCKET_NAME: spring2024-bucket # AWS AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} AWS_DEFAULT_REGION: 'eu-central-1' jobs: crossplane-provision-aws: runs-on: ubuntu-latest steps: - name: Checkout uses: actions/checkout@master # Using branch unique name for our bucket to prevent interfering jobs # But we need to split out the branch name, see https://stackoverflow.com/a/73467112/4964553 # otherwise we'll run into errors like: 'compose resources: cannot associate composed resources with Composition resource templates: cannot get composed resource: invalid resource name "spring2024-bucket-renovate/xpkg.upbound.io-upbound-provider-aws-s3-1.x": [may not contain '/']' - name: Split branch name env: BRANCH: ${{ github.ref_name }} id: split run: echo "::set-output name=branchbucketsuffix::${BRANCH##*/}"

Later I create a inline Claim instead of using the yaml file to be able to override the bucket name dynamically:

 # Not using kubectl apply -f upbound/provider-aws-s3/claim.yaml currently to prevent jobs from interfering with each other - name: Create XRD, Composition & Claim to create S3 Bucket (now using Upbound official AWS Provider Families) run: |  ...   echo "### Create Claim, which should create S3 Bucket (inline here to prevent jobs from interfering with each other)"  kubectl apply -f - <<EOF  apiVersion: crossplane.jonashackt.io/v1alpha1  kind: ObjectStorage  metadata:  namespace: default  name: managed-upbound-s3  spec:  compositionRef:  name: objectstorage-composition  parameters:  bucketName: "$BUCKET_NAME-${{ steps.split.outputs.branchbucketsuffix }}"  region: eu-central-1  EOF   ...

Now in AWS console we can watch Renovate doing it's work without interfering with other branches:

And finally we need to use the bucket name also when we interact with AWS S3:

 - name: Upload index.html to S3 and check deployment works run: |  echo "### Upload index.html to Bucket via AWS CLI"  aws s3 sync static "s3://$BUCKET_NAME-${{ steps.split.outputs.branchbucketsuffix }}" --acl public-read   echo "### Access S3 Bucket static website"  curl "http://$BUCKET_NAME-${{ steps.split.outputs.branchbucketsuffix }}.s3-website.eu-central-1.amazonaws.com"   - name: Delete index.html and remove Claim for S3 Bucket deletion run: |  echo "### Delete index.html from S3 Bucket"  aws s3 rm "s3://$BUCKET_NAME-${{ steps.split.outputs.branchbucketsuffix }}/index.html"   ...