Pulumi Kubernetes Operator

This page details how to use the Pulumi Kubernetes Operator (PKO) to automate the deployment of Pulumi stacks. The Pulumi program for a stack can come from a Program resource, from a Git repository, or from a Flux source, and may be authored in any supported Pulumi language (TypeScript, Python, Go, .NET, Java, YAML).

Overview

The Pulumi Kubernetes Operator provides custom resources to:

• Provision a workspace (an execution environment) for a Pulumi project
• Keep a Pulumi stack up-to-date using gitops
• Write Pulumi YAML programs as Kubernetes objects
• Run Pulumi deployment operations

Deploying Pulumi stacks using Kubernetes provides the capability to build out CI/CD and other automation systems, and to manage your infrastructure alongside your Kubernetes workloads or in dedicated control-plane clusters.

To work with the operator, we’ll need to follow these steps.

• Overview
• Install the Pulumi Kubernetes Operator
  • Using Helm
  • Dev Install
• Create a Service Account
• Configure Pulumi Cloud Access
• Create a Stack Resource
  • Using a Git repository
  • Using a Flux source
  • Using a Program object
• Explore other Features
  • Stack Configuration Values
  • Environment Variables
  • Drift Detection
  • State Refresh
  • Stack Cleanup
  • Stack Prerequisites
  • External Triggers
• Use With Argo CD
• More Information
  • Examples
  • Getting Help

Install the Pulumi Kubernetes Operator

Using Helm

Use Helm 3.x to install the Pulumi Kubernetes Operator into your cluster.

helm install --create-namespace -n pulumi-kubernetes-operator pulumi-kubernetes-operator \  oci://ghcr.io/pulumi/helm-charts/pulumi-kubernetes-operator --version 2.0.0 

Dev Install

A simple “quickstart” installation manifest is provided for non-production environments.

Install with kubectl:

kubectl apply -f https://raw.githubusercontent.com/pulumi/pulumi-kubernetes-operator/refs/tags/v2.2.0/deploy/quickstart/install.yaml 

Note: the installation manifest creates a usable Kubernetes service account named default/pulumi for your convenience.

Create a Service Account

The operator uses Kubernetes pods as the execution environment for Pulumi stack operations, with each Stack having a dedicated pod. A pod service account is needed to serve as the stack’s identity and to authenticate users.

Create a ServiceAccount named default/pulumi and grant the system:auth-delegator cluster role:

apiVersion: v1 kind: ServiceAccount metadata:  namespace: default  name: pulumi --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata:  name: default:pulumi:system:auth-delegator roleRef:  apiGroup: rbac.authorization.k8s.io  kind: ClusterRole  name: system:auth-delegator # permissions: TokenReview, SubjectAccessReview subjects: - kind: ServiceAccount  namespace: default  name: pulumi 

If your Pulumi program uses the Kubernetes Provider to manage resources within the cluster, the stack’s service account will need extra permissions, e.g. a ClusterRoleBinding to the cluster-admin cluster role.

See “Kubernetes: Service Accounts” for more information.

Configure Pulumi Cloud Access

By default, the operator uses Pulumi Cloud as the state backend for your stacks. Please create a Secret containing a Pulumi access token to be used to authenticate to Pulumi Cloud. Follow these instructions to create a personal, organization, or team access token.

Here’s an easy way to create a secret named default/pulumi-api-secret:

kubectl create secret generic -n default pulumi-api-secret \  --from-literal=accessToken=$PULUMI_ACCESS_TOKEN 

In the Stack specification, use spec.envRefs to reference the secret:

spec:  envRefs:  PULUMI_ACCESS_TOKEN:  type: Secret  secret:  name: pulumi-api-secret  key: accessToken 

To use a DIY state backend, set the spec.backend field to a storage endpoint URL. Use spec.envRefs to attach credentials and to set environment variables for the backend as necessary.

See “States & Backends” for more information.

Create a Stack Resource

The Stack Resource encapsulates a Pulumi project to provision infrastructure resources such as cloud VMs, object storage, and Kubernetes clusters and their workloads.

Set the spec.serviceAccountName field to the name of a ServiceAccount with the requisite permissions.

Set the spec.stack field to a unique Pulumi stack name, using a supported format.

Using a Git repository

In this scenario, the stack draws on a Git repository for the program source code.

The Stack specification can specify a commit SHA (spec.commit) or a branch reference (spec.branch). The repository URL is specified with spec.projectRepo plus an optional spec.repoDir.

If a branch reference is specified, the operator will periodically poll the branch for any new commits and roll out updates as they are found. Use the spec.resyncFrequencySeconds field to set the polling frequency.

Specify Git authentication options with the spec.gitAuth field.

In the example below, we’re creating a Stack for a Pulumi project called kubernetes-ts-nginx to deploy a simple NGINX server to your cluster. Without any configuration, the Kubernetes Provider uses the in-cluster Kubernetes context.

import * as pulumi from "@pulumi/pulumi"; import * as k8s from "@pulumi/kubernetes";  // Get the Pulumi API token. const pulumiConfig = new pulumi.Config(); const pulumiAccessToken = pulumiConfig.requireSecret("pulumiAccessToken")  // Create the API token as a Kubernetes Secret. const accessToken = new k8s.core.v1.Secret("accessToken", {  stringData: { accessToken: pulumiAccessToken }, });  // Create an NGINX deployment in-cluster. const mystack = new k8s.apiextensions.CustomResource("my-stack", {  apiVersion: 'pulumi.com/v1',  kind: 'Stack',  spec: {  serviceAccountName: "pulumi",  envRefs: {  PULUMI_ACCESS_TOKEN: {  type: "Secret",  secret: {  name: accessToken.metadata.name,  key: "accessToken"  },  },  },  stack: "<YOUR_ORG>/k8s-nginx/dev",  projectRepo: "https://github.com/pulumi/examples",  repoDir: "kubernetes-ts-nginx/",  commit: "03658b5514f08970f350618a6e6fdf1bd75f45d0",  // branch: "master", // Alternatively, track master branch.  destroyOnFinalize: true,  } }); 

import pulumi from pulumi_kubernetes import core, apiextensions  # Get the Pulumi API token. pulumi_config = pulumi.Config() pulumi_access_token = pulumi_config.require_secret("pulumiAccessToken")  # Create the API token as a Kubernetes Secret. access_token = core.v1.Secret("accessToken", string_data={ "access_token": pulumi_access_token })  # Create an NGINX deployment in-cluster. my_stack = apiextensions.CustomResource("my-stack",  api_version="pulumi.com/v1",  kind="Stack",  spec={  "serviceAccountName": "pulumi",  "envRefs": {  "PULUMI_ACCESS_TOKEN": {  "type": "Secret",  "secret": {  "name": access_token.metadata.name,  "key": "access_token",  }  },  },  "stack": "<YOUR_ORG>/k8s-nginx/dev",  "projectRepo": "https://github.com/pulumi/examples",  "repoDir": "kubernetes-ts-nginx/",  "commit": "03658b5514f08970f350618a6e6fdf1bd75f45d0",  # branch: "master", # Alternatively, track master branch.  "destroyOnFinalize": True,  } ) 

using Pulumi; using Pulumi.Kubernetes.ApiExtensions; using Pulumi.Kubernetes.Core.V1; using Pulumi.Kubernetes.Types.Inputs.Core.V1;  class StackArgs : CustomResourceArgs {  [Input("spec")]  public Input<StackSpecArgs>? Spec { get; set; }   public StackArgs() : base("pulumi.com/v1", "Stack")  {  } }  class StackSpecArgs : ResourceArgs {  [Input("serviceAccountName")]  public Input<string>? ServiceAccountName { get; set; }   [Input("accessTokenSecret")]  public Input<string>? AccessTokenSecret { get; set; }   [Input("stack")]  public Input<string>? Stack { get; set; }   [Input("projectRepo")]  public Input<string>? ProjectRepo { get; set; }   [Input("commit")]  public Input<string>? Commit { get; set; }   [Input("destroyOnFinalize")]  public Input<bool>? DestroyOnFinalize { get; set; } }  class MyStack : Stack {  public MyStack()  {  // Get the Pulumi API token.  var config = new Config();  var pulumiAccessToken = config.RequireSecret("pulumiAccessToken");   // Create the API token as a Kubernetes Secret.  var accessToken = new Secret("accessToken", new SecretArgs  {  StringData =  {  {"accessToken", pulumiAccessToken}  }  });   // Create an NGINX deployment in-cluster.  var myStack = new Pulumi.Kubernetes.ApiExtensions.CustomResource("nginx", new StackArgs  {  Spec = new StackSpecArgs  {  ServiceAccountName = "pulumi",  AccessTokenSecret = accessToken.Metadata.Apply(m => m.Name),  Stack = "<YOUR_ORG>/k8s-nginx/dev",  InitOnCreate = true,  ProjectRepo = "https://github.com/pulumi/examples",  RepoDir = "kubernetes-ts-nginx/",  Commit = "03658b5514f08970f350618a6e6fdf1bd75f45d0",  // branch: "master", // Alternatively, track master branch.  DestroyOnFinalize = true,  }  });  } } 

package main  import ( "github.com/pulumi/pulumi-kubernetes/sdk/v3/go/kubernetes" apiextensions "github.com/pulumi/pulumi-kubernetes/sdk/v3/go/kubernetes/apiextensions" corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v3/go/kubernetes/core/v1" "github.com/pulumi/pulumi/sdk/v3/go/pulumi" "github.com/pulumi/pulumi/sdk/v3/go/pulumi/config" )  func main() { pulumi.Run(func(ctx *pulumi.Context) error { // Get the Pulumi API token. c := config.New(ctx, "") pulumiAccessToken := c.Require("pulumiAccessToken")  // Create the API token as a Kubernetes Secret. accessToken, err := corev1.NewSecret(ctx, "accessToken", &corev1.SecretArgs{ StringData: pulumi.StringMap{"accessToken": pulumi.String(pulumiAccessToken)}, }) if err != nil { return err }  // Create an NGINX deployment in-cluster. _, err = apiextensions.NewCustomResource(ctx, "my-stack", &apiextensions.CustomResourceArgs{ ApiVersion: pulumi.String("pulumi.com/v1"), Kind: pulumi.String("Stack"), OtherFields: kubernetes.UntypedArgs{ "spec": map[string]interface{}{ "serviceAccountName": "pulumi", "envRefs": pulumi.Map{ "PULUMI_ACCESS_TOKEN": pulumi.Map{ "type": pulumi.String("Secret"), "secret": pulumi.Map{ "name": accessToken.Metadata.Name(), "key": pulumi.String("accessToken"), }, }, }, "stack": "<YOUR_ORG>/k8s-nginx/dev", "projectRepo": "https://github.com/pulumi/examples", "repoDir": "kubernetes-ts-nginx/", "commit": "03658b5514f08970f350618a6e6fdf1bd75f45d0", // "branch": "master", // Alternatively, track master branch. "destroyOnFinalize": true, }, }, }, pulumi.DependsOn([]pulumi.Resource{accessToken})) return err }) } 

Using a Flux source

Flux offers a powerful alternative for fetching Pulumi program source code from a variety of sources, including OCI repositories and cloud storage buckets. Flux also supports some advanced Git options. Flux sources are specified as Custom Resources in a Kubernetes cluster; examples of sources are GitRepository, OCIRepository, and Bucket resources.

To refer to a Flux source object, use the spec.fluxSource field. Use spec.fluxSource.dir to refer to a program directory within the source artifact.

Here is the TypeScript example from above, adjusted to create a Flux source for the Git repo and then use it in the Stack specification. This example assumes you’ve already installed Flux into your cluster (see “Flux installation”).

import * as pulumi from "@pulumi/pulumi"; import * as k8s from "@pulumi/kubernetes";  // Get the Pulumi API token. const pulumiConfig = new pulumi.Config(); const pulumiAccessToken = pulumiConfig.requireSecret("pulumiAccessToken")  // Create the API token as a Kubernetes Secret. const accessToken = new k8s.core.v1.Secret("accessToken", {  stringData: { accessToken: pulumiAccessToken }, });  // Create a GitRepository const gitrepo = new k8s.apiextensions.CustomResource("nginx-repo", {  apiVersion: "source.toolkit.fluxcd.io/v1",  kind: "GitRepository",  metadata: {},  spec: {  interval: '5m0s',  url: "https://github.com/pulumi/examples",  ref: { commit: "03658b5514f08970f350618a6e6fdf1bd75f45d0" },  }, });  // Create an NGINX deployment in-cluster. const mystack = new k8s.apiextensions.CustomResource("my-stack", {  apiVersion: 'pulumi.com/v1',  kind: 'Stack',  spec: {  serviceAccountName: "pulumi",  envRefs: {  PULUMI_ACCESS_TOKEN: {  type: "Secret",  secret: {  name: accessToken.metadata.name,  key: "accessToken"  },  },  },  stack: "<YOUR_ORG>/k8s-nginx/dev",  fluxSource: {  sourceRef: {  apiVersion: "source.toolkit.fluxcd.io/v1",  kind: "GitRepository",  name: gitrepo.metadata.name,  },  },  destroyOnFinalize: true,  } }); 

Using a Program object

With the Program resource, you can define a Pulumi YAML program directly as a Kubernetes resource. The reference docs for the Program Custom Resource details the wrapping; the reference docs for Pulumi YAML gives all the fields that are part of the program code.

Here is an example as a YAML manifest file:

--- apiVersion: pulumi.com/v1 kind: Program metadata:  name: staticwebsite program:  resources:  my-bucket:  type: aws:s3:Bucket  my-bucket-ownership-controls:  type: aws:s3:BucketOwnershipControls  properties:  bucket: ${my-bucket.id}  rule:  objectOwnership: ObjectWriter  my-bucket-acl:  type: aws:s3:BucketAclV2  properties:  bucket: ${my-bucket.bucket}  acl: public-read  options:  dependsOn:  - ${my-bucket-ownership-controls}  my-bucket-public-access-block:  type: aws:s3:BucketPublicAccessBlock  properties:  bucket: ${my-bucket.id}  blockPublicAcls: false  my-bucket-website:  type: aws:s3:BucketWebsiteConfigurationV2  properties:  bucket: ${my-bucket.bucket}  indexDocument:  suffix: index.html  index.html:  type: aws:s3:BucketObject  properties:  bucket: ${my-bucket}  source:  fn::stringAsset: <h1>Hello, world!</h1>  acl: public-read  contentType: text/html  outputs:  bucketEndpoint: http://${my-bucket-website.websiteEndpoint} 

You can then create a Stack object to deploy the program, by referring to it in the spec.programRef field:

--- apiVersion: pulumi.com/v1 kind: Stack metadata:  name: staticwebsite spec:  serviceAccountName: pulumi  stack: <YOUR ORG>/staticwebsite/dev  programRef:  name: staticwebsite  destroyOnFinalize: true  config:  aws:region: us-east-1 

Explore other Features

Here’s some advanced options provided by the Stack resource. Detailed documentation on the Stack API is available here.

Stack Configuration Values

In many cases, different stacks for a single project will need differing values. For instance, you may want to use a different size for your AWS EC2 instance, or a different number of replicas for a particular Kubernetes deployment. Pulumi offers a configuration system for managing such differences; see “Configuration” for more information.

Use the spec.config block to set stack configuration values. The values are merged into your project’s stack settings file.

Use the spec.secretsRef block to set configuration values containing secrets. The value may be a literal value or may be a reference to a Kubernetes Secret.

Use the spec.secretsProvider field to use an alternative encryption provider. See “Initializing a stack with alternative encryption” for more information.

Environment Variables

Use the spec.envRefs field to set environment variables for the Pulumi program, such as PULUMI_ACCESS_TOKEN or AWS_SECRET_ACCESS_KEY.

Values may be literals or based on the contents of a ConfigMap or Secret object.

Drift Detection

Drift detection means to detect unwanted changes to your provisioned infrastructure. The operator supports drift detection and remediation by periodically running pulumi up. This is referred to as re-synchronization.

Use the spec.continueResyncOnCommitMatch field to enable periodic resyncs. Use the spec.resyncFrequencySeconds field to set the resync frequency.

State Refresh

Use the spec.refresh field to refresh the state of the stack’s resources before each update.

Stack Cleanup

Use the spec.destroyOnFinalize field to automatically destroy the Pulumi stack (i.e. pulumi destroy -f) when the Stack object is deleted. Enable this option to link the lifecycle of the Pulumi stack, and the resources it contains, to its Stack object.

Stack Prerequisites

It is possible to declare that a particular Stack be dependent on another Stack. The dependent stack waits for the other stack to be successfully deployed. Use the succeededWithinDuration field to set a duration within which the prerequisite must have reached success; otherwise the dependency is automatically re-synced.

External Triggers

It is possible to trigger a stack update for a stack at any time by applying the pulumi.com/reconciliation-request annotation:

kubectl annotate stack $STACK_NAME "pulumi.com/reconciliation-request=$(date)" --overwrite 

The value of the annotation is arbitrary, and we recommend using a timestamp.

Use With Argo CD

We can use ArgoCD in combination with PKO to manage the lifetime of the Stack via the GitOps paradigm. This gives you the ability to use the ArgoCD UI or CLI to interact with the Stack, and to allow ArgoCD to reconcile changes to the Stack specification. The Pulumi Kubernetes Operator handles the details.

First, we need to define a Pulumi stack as a Kubernetes manifest that ArgoCD can deploy. We assume here that this manifest lives in the same repository as the Pulumi program, in the subfolder deploy/. However, this manifest could live in a separate repository, such as an “app-of-apps” repo. In this example, the manifest declares a service account and cluster role bindings to allow the stack to create resources in the cluster. Additionally, we expect a Secret to exist on the cluster containing a Pulumi access token.

Note that the Stack’s projectRepo and branch point to the location of the Pulumi program to be executed by the Pulumi Kubernetes Operator.

--- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata:  name: my-app:system:auth-delegator  annotations:  argocd.argoproj.io/sync-wave: "2"  labels:  app.kubernetes.io/instance: my-app roleRef:  apiGroup: rbac.authorization.k8s.io  kind: ClusterRole  name: system:auth-delegator subjects: - kind: ServiceAccount  name: my-app  namespace: some-namepace --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata:  name: my-app:cluster-admin  annotations:  argocd.argoproj.io/sync-wave: "2"  labels:  app.kubernetes.io/instance: my-app roleRef:  apiGroup: rbac.authorization.k8s.io  kind: ClusterRole  name: cluster-admin subjects: - kind: ServiceAccount  name: my-app  namespace: some-namepace --- apiVersion: pulumi.com/v1 kind: Stack metadata:  name: my-app-dev  namespace: some-namepace  labels:  app.kubernetes.io/instance: my-app  annotations:  argocd.argoproj.io/sync-wave: "3"  pulumi.com/reconciliation-request: "before-first-update"  link.argocd.argoproj.io/external-link: http://app.pulumi.com/my-org/my-prject/dev spec:  serviceAccountName: my-app  stack: my-org/my-project/dev  projectRepo: "https://github.com/my-repo/my-app.git"  branch: main  refresh: true  resyncFrequencySeconds: 60  destroyOnFinalize: true  envRefs:  PULUMI_ACCESS_TOKEN:  type: Secret  secret:  name: pulumi-access-token-secret  key: PULUMI_ACCESS_TOKEN  workspaceTemplate:  spec:  image: pulumi/pulumi:3.134.1-nonroot 

Next we create an ArgoCD Application object:

apiVersion: argoproj.io/v1alpha1 kind: Application metadata:  name: my-app  namespace: argocd  finalizers:  # best practice: use background cascading deletion when destroyOnFinalize is enabled.  - resources-finalizer.argocd.argoproj.io/background spec:  destination:  namespace: default  server: "https://kubernetes.default.svc"  syncPolicy:  automated:  prune: true  project: default  source:  repoURL: "https://github.com/my-repo/my-app.git"  path: "./deploy" # the location of the Stack maifest  targetRevision: main 

ArgoCD will sync the Application by applying the Stack object, which will in turn effect a Pulumi deployment. The result will look something like this in the ArgoCD UI:

More Information

Examples

More examples are available in the pulumi/pulumi-kubernetes-operator repository.

Getting Help

Check out troubleshooting for more details, look at known issues or open a new issue in GitHub.