Troubleshoot Crossplane

Requested resource not found

If you use the Crossplane CLI to install a Provider or Configuration (for example, crossplane xpkg install provider xpkg.crossplane.io/crossplane-contrib/provider-aws-s3:v2.0.0) and get the server could not find the requested resource error, more often than not, that’s an indicator that your Crossplane CLI needs updating. In other words Crossplane graduated some API from alpha to beta or stable and the old plugin isn’t aware of this change.

Resource status and conditions

Most Crossplane resources have a status section that can represent the current state of that particular resource. Running kubectl describe against a Crossplane resource frequently gives insightful information about its condition. For example, to determine the status of a GCP CloudSQLInstance managed resource use kubectl describe for the resource.

1kubectl describe cloudsqlinstance my-db 2Status: 3 Conditions: 4 Last Transition Time: 2019-09-16T13:46:42Z 5 Reason: Creating 6 Status: False 7 Type: Ready 

Most Crossplane resources set the Ready condition. Ready represents the availability of the resource - whether it’s creating, deleting, available, unavailable, binding, etc.

Resource events

Most Crossplane resources emit events when something interesting happens. You can see the events associated with a resource by running kubectl describe - for example, kubectl describe cloudsqlinstance my-db. You can also see all events in a particular namespace by running kubectl get events.

1Events: 2 Type Reason Age From Message 3 ---- ------ ---- ---- ------- 4 Warning CannotConnectToProvider 16s (x4 over 46s) managed/postgresqlserver.database.azure.crossplane.io cannot get referenced ProviderConfig: ProviderConfig.azure.crossplane.io "default" not found 

Note that Kubernetes namespaces events, while most Crossplane resources (XRs, etc) are cluster scoped. Crossplane emits events for cluster scoped resources to the ‘default’ namespace.

Crossplane Logs

The next place to look to get more information or investigate a failure would be in the Crossplane pod logs, which should be running in the crossplane-system namespace. To get the current Crossplane logs, run the following:

1kubectl -n crossplane-system logs -lapp=crossplane 

Note that Crossplane emits minimal logs by default - events are typically the best place to look for information about what Crossplane is doing. You may need to restart Crossplane with the --debug flag if you can’t find what you’re looking for.

Provider logs

Remember that providers provide much of Crossplane’s features. You can use kubectl logs to view provider logs too. By convention, they also emit minimal logs by default.

1kubectl -n crossplane-system logs <name-of-provider-pod> 

All providers maintained by the Crossplane community mirror Crossplane’s support of the --debug flag. The easiest way to set flags on a provider is to create a DeploymentRuntimeConfig and reference it from the Provider:

 1apiVersion: pkg.crossplane.io/v1beta1  2kind: DeploymentRuntimeConfig  3metadata:  4 name: debug-config  5spec:  6 deploymentTemplate:  7 spec:  8 selector: {}  9 template: 10 spec: 11 containers: 12 - name: package-runtime 13 args: 14 - --debug 15--- 16apiVersion: pkg.crossplane.io/v1 17kind: Provider 18metadata: 19 name: crossplane-contrib-provider-aws-s3 20spec: 21 package: xpkg.crossplane.io/crossplane-contrib/provider-aws-s3:v2.0.0 22 runtimeConfigRef: 23 apiVersion: pkg.crossplane.io/v1beta1 24 kind: DeploymentRuntimeConfig 25 name: debug-config 

Note that you can add a reference to a DeploymentRuntimeConfig to an already installed Provider and it updates its Deployment accordingly.

Pausing Crossplane

Sometimes, for example when you encounter a bug, it can be useful to pause Crossplane if you want to stop it from actively attempting to manage your resources. To pause Crossplane without deleting all its resources, run the following command to scale down its deployment:

1kubectl -n crossplane-system scale --replicas=0 deployment/crossplane 

After you have been able to rectify the problem or smooth things out, you can unpause Crossplane by scaling its deployment back up:

1kubectl -n crossplane-system scale --replicas=1 deployment/crossplane 

Pausing Providers

You can also pause Providers when troubleshooting an issue or orchestrating a complex migration of resources. Creating and referencing a DeploymentRuntimeConfig is the easiest way to scale down a provider, and you can change the DeploymentRuntimeConfig or remove the reference to scale it back up:

 1apiVersion: pkg.crossplane.io/v1beta1  2kind: DeploymentRuntimeConfig  3metadata:  4 name: scale-config  5spec:  6 deploymentTemplate:  7 spec:  8 selector: {}  9 replicas: 0 10 template: {} 11--- 12apiVersion: pkg.crossplane.io/v1 13kind: Provider 14metadata: 15 name: crossplane-contrib-provider-aws-s3 16spec: 17 package: xpkg.crossplane.io/crossplane-contrib/provider-aws-s3:v2.0.0 18 runtimeConfigRef: 19 apiVersion: pkg.crossplane.io/v1beta1 20 kind: DeploymentRuntimeConfig 21 name: scale-config 

Note that you can add a reference to a DeploymentRuntimeConfig to an already installed Provider and it updates its Deployment accordingly.

Deleting when a resource hangs

The resources that Crossplane manages are automatically cleaned up so as not to leave anything running behind. Crossplane accomplishes this by using finalizers, but in certain scenarios the finalizer can prevent the Kubernetes object from getting deleted.

To deal with this, patch the object to remove its finalizer, which then allows Kubernetes to delete it. Note that this doesn’t necessarily delete the external resource that Crossplane was managing, so you want to go to your cloud provider’s console and look there for any lingering resources to clean up.

In general, you can remove a finalizer from an object with this command:

1kubectl patch <resource-type> <resource-name> -p '{"metadata":{"finalizers": []}}' --type=merge 

For example, for a CloudSQLInstance managed resource (database.gcp.crossplane.io) named my-db, you can remove its finalizer with:

1kubectl patch cloudsqlinstance my-db -p '{"metadata":{"finalizers": []}}' --type=merge 

Tips, tricks, and troubleshooting

This section covers some common tips, tricks, and troubleshooting steps for working with Composite Resources. If you’re trying to track down why your Composite Resources aren’t working the [Troubleshooting][trouble-ref] page also has some useful information.