Tailscale Kubernetes Operator

Apologizes to anyone looking for more #Pidgin content. I know it's been awhile and I'll get something out soon I promise, but there are quite a few Pidgin related things in this post too...

About a month ago I setup a Kubernetes cluster using Talos to handle my container load at home.

I have containers for all sorts of things including the graphs you all may have seen on my stream as well as an Ergo instance for developing our new IRCv3 protocol plugin.

I'll be adding to this cluster in the future too as we continue development on Xeme which is our new XMPP library as well as Myna which is our new Matrix library.

Having these servers running locally makes it easier to test interesting configurations, but more importantly, lets me use them on stream without leaking my IP address or any passwords.

Anyways, I'm not always at home when I want to work on something and I wanted to find a way to access services remotely. Well tonight I realized I should be able to do this with Tailscale.

I'm already using Tailscale to make sure we can access all of our build agents which are spread across a few sites, so being able to expose this to the other Pidgin developers who have access to that Tailscale network is just icing on the cake.

So I decided to give the Tailscale Kubernetes Operator a go. They have full documentation it here. As is pretty typical right now their installation instructions are based on using Helm.

I prefer to use Kustomize over Helm. There's plenty of reasons, but there's no reason to get into that now ;)

Thankfully, Tailscale provides a static manifest for deploying the operator. This is very easy to import into our kustomize and get running directly quite quickly.

But of course, it's not just that easy. Their instructions have you modifying that manifest and then applying that. This isn't a big deal, but it makes upgrading harder as you have to keep track of what you edited. Sure you could use version control (and you should have your manifest in version control) but kustomize makes this very easy.

First we needed to download operator.yaml which is the static manifest that Tailscale provides. We're going to use this file as is without modification which means we can just replace it if/when they update it.

Next we need to create our kustomization.yaml which is used to drive everything. My commented kustomization.yaml is below.

 --- # Here we reference the unmodified operator.yaml we downloaded from # Tailscale. resources: - operator.yaml patches: # Tailscale needs network admin and some other permissions, so # we use the following patch that will add the appropriate * annotations to the namespace. - patch: |- apiVersion: v1 kind: Namespace metadata: name: tailscale labels: pod-security.kubernetes.io/enforce: privileged pod-security.kubernetes.io/audit: privileged pod-security.kubernetes.io/warn: privileged secretGenerator: # As I mentioned earlier, operator.yaml defines a secret for the # tailscale oauth secrets which the documentation wants you to # modify manually. We can instead use a secret generator to read # those values from an `env` file and merge them into the # existing secret. - name: operator-oauth namespace: tailscale behavior: merge envs: - secrets/env 

With this all now setup, we can run kubectl kustomize and verify our output. A condensed output with just the namespace and secret are below.

 apiVersion: v1 kind: Namespace metadata: labels: pod-security.kubernetes.io/audit: privileged pod-security.kubernetes.io/enforce: privileged pod-security.kubernetes.io/warn: privileged name: tailscale --- apiVersion: v1 data: client_id: aGFjayB0aGUgcGxhbmV0 client_secret: aGFja2VycyBvZiB0aGUgd29ybGQgdW5pdGU= kind: Secret metadata: name: operator-oauth namespace: tailscale type: Opaque --- 

Now that we've confirmed everything looks good, we can get ready to apply it to our cluster. Normally I'd recommend doing a dry run first, but since we're applying a namespace as well most of the tests will fail because that namespace doesn't exist.

Anyways, we can apply this kustomization with the following command:

 kubectl apply -k . 

If everything is successful you'll see something like the following:

 namespace/tailscale configured customresourcedefinition.apiextensions.k8s.io/connectors.tailscale.com configured customresourcedefinition.apiextensions.k8s.io/dnsconfigs.tailscale.com configured customresourcedefinition.apiextensions.k8s.io/proxyclasses.tailscale.com configured serviceaccount/operator configured serviceaccount/proxies configured role.rbac.authorization.k8s.io/operator configured role.rbac.authorization.k8s.io/proxies configured clusterrole.rbac.authorization.k8s.io/tailscale-operator configured rolebinding.rbac.authorization.k8s.io/operator configured rolebinding.rbac.authorization.k8s.io/proxies configured clusterrolebinding.rbac.authorization.k8s.io/tailscale-operator configured secret/operator-oauth configured deployment.apps/operator configured ingressclass.networking.k8s.io/tailscale configured 

You can verify the operator is up and running the following command:

 kubectl -n tailscale get deployments 

You should see something like the following:

 NAME READY UP-TO-DATE AVAILABLE AGE operator 1/1 1 1 72m 

If that all looks good you can go ahead and verify that it shows up in the Machines section in Tailscale. You should see something like the following:

Now we're ready to expose a service via Tailscale!

As I mentioned earlier, I'm running an Ergo instance for local IRC development. I have an Service of type LoadBalancer to expose it to my LAN via MetalLB.

The Tailscale operator has multiple ways to integrate with your Kubernetes cluster, but we're just going to expose existing services to the tailnet and doing that is extremely simple.

To do so, you just need to add the tailscale.com/expose: "true" annotation to the service. Below is my updated Service for Ergo:

 --- apiVersion: v1 kind: Service metadata: name: ergo annotations: tailscale.com/expose: "true" labels: app: ergo spec: ports: - port: 6667 protocol: TCP name: irc - port: 6697 protocol: TCP name: ircs selector: app: ergo type: LoadBalancer 

Now that that's done, I can apply the Kustomization for Ergo:

 kubectl apply -k . 

The operator will automatically add it to the tailnet with a machine name of <namespace>-<service-name>. This ergo instance is running in the reaperworld namespace so it's machine name, and therefore DNS name is reaperworld-ergo which we can now see in the Tailscale machines page.

And that's it, that service is now available on the tailnet accessible to anyone else on the tailnet!

I hope you all found this interesting. I decided to write this as I had to learn about behavior: merge property of the secretGenerator and figured others might find this useful as well. Plus it gave me an excuse to show the power of Kustomize!