DEV Community

Cover image for Cluster-level multitenancy with vCluster
Ashok Nagaraj
Ashok Nagaraj

Posted on

Cluster-level multitenancy with vCluster

#kubernetes #vcluster
What are they?

Virtual clusters are fully working Kubernetes clusters that run on top of other Kubernetes clusters. Compared to fully separate "real" clusters, virtual clusters reuse worker nodes and networking of the host cluster. They have their own control plane and schedule all workloads into a single namespace of the host cluster. Like virtual machines, virtual clusters partition a single physical cluster into multiple separate ones.
Official documentation

Why do we need them?

To partition the cluster into multiple virtual clusters which can be provided to the tenant teams for isolation. They provide following advantages:

  • Ease of use - each tenant gets a cluster of their own!
  • Customization - ability to install and use (and test) different cluster level resources (CRDs)
  • Isolation - much stricter isolation than traditional namespace based multi tenancy
  • Manageability - operators need not handle (too) many clusters
  • Lightweight and full-fledged - based on the popular k3s distro and sqlite for DB (instead of etcd)

Architecture

How to use them

  1. Download vcluster cli
    brew install vcluster

  2. Install the cluster under a namespace team-abc

❯ time vcluster create -n team-abc vcluster-abc info Detected local kubernetes cluster kind. Will deploy vcluster with a NodePort & sync real nodes info Create vcluster vcluster-abc... done √ Successfully created virtual cluster vcluster-abc in namespace team-abc info Waiting for vcluster to come up... warn vcluster is waiting, because vcluster pod vcluster-abc-0 has status: ContainerCreating warn vcluster is waiting, because vcluster pod vcluster-abc-0 has status: ContainerCreating warn vcluster is waiting, because vcluster pod vcluster-abc-0 has ... info Starting proxy container... done √ Switched active kube context to vcluster_vcluster-abc_team-abc_kind-macbook - Use `vcluster disconnect` to return to your previous kube context - Use `kubectl get namespaces` to access the vcluster vcluster create -n team-abc vcluster-abc 0.57s user 0.49s system 0% cpu 2:18.09 total
Enter fullscreen mode Exit fullscreen mode
  1. Check the new cluster 
❯ kubectl cluster-info Kubernetes control plane is running at https://127.0.0.1:10754 CoreDNS is running at https://127.0.0.1:10754/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. ❯ kubectl ns default kube-system kube-public kube-node-lease
Enter fullscreen mode Exit fullscreen mode
  1. Deploy workloads 
❯ kubectl create ns test-ns namespace/test-ns created ❯ kubectl create deployment test-dep --image=nginx --replicas=2 -n test-ns deployment.apps/test-dep created ❯ kubectl get all -n test-ns NAME READY STATUS RESTARTS AGE pod/test-dep-574f5c6754-x4l45 0/1 ContainerCreating 0 51s pod/test-dep-574f5c6754-7tlr5 0/1 ContainerCreating 0 51s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/test-dep 0/2 2 0 51s NAME DESIRED CURRENT READY AGE replicaset.apps/test-dep-574f5c6754 2 2 0 51s
Enter fullscreen mode Exit fullscreen mode
  1. Check back from host cluster 
❯ vcluster disconnect ❯ k get pods -n team-abc NAME READY STATUS RESTARTS AGE coredns-5df468b6b7-dtmn6-x-kube-system-x-vcluster-abc 1/1 Running 0 7m38s test-dep-574f5c6754-7tlr5-x-test-ns-x-vcluster-abc 1/1 Running 0 2m47s test-dep-574f5c6754-x4l45-x-test-ns-x-vcluster-abc 1/1 Running 0 2m47s vcluster-abc-0 2/2 Running 0 9m30s
Enter fullscreen mode Exit fullscreen mode
House keeping 
❯ vcluster list NAME NAMESPACE STATUS CONNECTED CREATED AGE vcluster-abc team-abc Running 2022-06-26 21:38:38 +0530 IST 10m44s ❯ vcluster delete vcluster-abc info Stopping docker proxy... info Delete vcluster vcluster-abc... done √ Successfully deleted virtual cluster vcluster-abc in namespace team-abc done √ Successfully deleted virtual cluster pvc data-vcluster-abc-0 in namespace team-abc
Enter fullscreen mode Exit fullscreen mode
Get the kubeconfig of a vcluster 
❯ vcluster list NAME NAMESPACE STATUS CONNECTED CREATED AGE vcluster-abc team-abc Running True 2022-06-26 21:51:11 +0530 IST 5m29s ❯ vcluster disconnect # Look for a secret named vc-<cluster-name> in the <namespace> used ❯ kubectl get secret vc-vcluster-abc -n team-abc --template={{.data.config}} | base64 -D clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJkakNDQVIyZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWpNU0V3SHdZRFZRUUREQmhyTTNNdGMyVnkKZG1WeUxXTmhRREUyTlRZeU5qQTFNalV3SGhjTk1qSXdOakkyTVRZeU1qQTFXaGNOTXpJd05qSXpNVFl5TWpBMQpXakFqTVNFd0h3WURWUVFEREJock0zTXRjMlZ5ZG1WeUxXTmhRREUyTlRZeU5qQTFNalV3V1RBVEJnY3Foa2pPClBRSUJCZ2dxaGtqT1BRTUJCd05DQUFTVThuZGdKaWlNTFFHTEI0VmxSSjlQNm9LdTdGbkU4ZTM2OXl0eDhUYnUKeFJxNTg3NVE4SE1zRU9SMjl4a2puUmN5dUNrcVpOTTRHeDYzLzM1bW5TSjBvMEl3UURBT0JnTlZIUThCQWY4RQpCQU1DQXFRd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVU5PKzlHb1M5Z1lib0NMWjU3ck9GCmdndFFDZlF3Q2dZSUtv= ...
Enter fullscreen mode Exit fullscreen mode
TODO

There are a lot more features to be explored and will do in another post

  • pause/resume
  • applying manifests on init
  • passing down limits and quotas
  • extensions/plugins
  • using different distributions
What is impressive
  • Tailor made for testing CRDs
  • Probably well suited for CI use-cases to create clusters on the fly
  • Very useful in creating throw-away clusters (for DEV/R&D purposes)

Top comments (0)