DEV Community

Cover image for Kubernetes multi-tenancy with projectsveltos
Gianluca
Gianluca

Posted on • Edited on

Kubernetes multi-tenancy with projectsveltos

#kubernetes #multitenancy #opensource #showdev

Kubernetes is a complex system that relies on several components being configured correctly to have a working cluster.

Some solutions already exist to automate clusters' creation, upgrade, deletion, like for instance ClusterAPI. Such solutions require the existance of a management Kubernetes cluster from where tens of other Kubernetes clusters are then programmatically created and managed.

Managing the life-cycle of Kubernetes clusters is the first step. After a Kubernetes cluster is created, an handful number of features needs to be installed (CNI for instance). Kubernetes add-ons are used to extend the functionality of Kubernetes clusters.

So after cluster creation, the next problem that needs to be addressed is how to programmatically manage Kubernetes add-ons in each managed cluster from the management cluster. Luckily, even in this space, there are already existing solutions. For instance, Sveltos is an open source project to programmatically deploy Kubernetes add-ons in tens of Kubernetes clusters.

Summarizing, using a Kubernetes cluster to manage other clusters (creation/upgrade/deletion and add-ons deployment) provides several benefits, including:

  1. Centralized management: A cluster management cluster allows administrators to manage multiple clusters from a single place, making it easier to maintain consistency and reduce the risk of configuration issues.
  2. Scalability: A cluster management cluster can help organizations scale their infrastructure by making it easier to create, deploy, and manage multiple clusters.
  3. Better security: A cluster management cluster can be configured with security-related addons, such as network policies and secrets management, to help ensure that all managed clusters are secure.
  4. Increased automation: A cluster management cluster can be integrated with a continuous integration/continuous deployment (CI/CD) pipeline, making it easier to automate the deployment of new clusters and addons, and reducing the time and effort involved in managing the infrastructure.

Problem gets more complex when different organizations end up sharing the managed Kubernetes clusters. This scenario is commonly referred to as multi-tenancy.

Common forms of multi-tenancy are:

  1. share a cluster between multiple teams within an organization, each of whom may operate one or more workloads;
  2. one (or more) cluster(s) fully reserved for an organization.

In this scenario we need to introduce two roles:

  • platform admin: The role of a platform administrator is to manage and maintain the infrastructure of the managed Kubernetes clusters. This includes tasks such as setting up the cluster, managing nodes, installing and updating software components, and ensuring the overall health and security of the cluster. The platform administrator is also responsible for onboarding tenant admins and granting permissions to those;
  • tenant admin: Has admin access to the clusters/namespaces assigned to them by the platform admin and manages tenant applications in those clusters/namespaces.

After cluster creation/upgrade/deletion and the addons deployment, the next problem to solve is how to allow platform admin to onboard tenants and programmatically grant them permissions from the management cluster.

Sveltos addons management solution

Before we deep dive into granting permissions to tenants, lets briefly summarize Sveltos solution to manage Kubernetes addons.

Sveltos introduces a single CRD called ClusterProfile for that.

By creating a ClusterProfile instance, we can tell Sveltos:

  1. where to deploy addons (i.e, in all managed Clusters matching the ClusterSelector field);
  2. list of addons to deploy (either Helm charts or Kubernetes resource YAMLs contained in referenced ConfigMaps/Secrets). 
apiVersion: config.projectsveltos.io/v1alpha1 kind: ClusterProfile metadata: name: deploy-kyverno spec: clusterSelector: env=fv syncMode: Continuous helmCharts: - repositoryURL: https://kyverno.github.io/kyverno/ repositoryName: kyverno chartName: kyverno/kyverno chartVersion: v2.6.0 releaseName: kyverno-latest releaseNamespace: kyverno helmChartAction: Install policyRefs: - name: disallow-latest-tag # referenced ConfigMap contains a Kyverno ClusterPolicy namespace: default kind: ConfigMap
Enter fullscreen mode Exit fullscreen mode

Sveltos addons management

With Sveltos configuration drift detection we don't have to worry about managed cluster states. Sveltos monitors the state of managed clusters and when it detects a configuration drift for one of the resource deployed because of a ClusterProfile, it re-syncs the cluster state back to the state described in the management cluster.

Sveltos add-ons deployment with drift detection

Sveltos multi-tenancy solution

Let's now go back to the multi-tenancy problem.

We want to:

  1. allow platform admin to programmatically grant permissions to tenant admins;
  2. allow tenant admins to manage applications from management cluster, making sure each tenant admin only deploys resources it has permissions to and only in the clusters/namespaces it has permission to.

RoleRequest is the CRD introduced by Sveltos to allow platform admin to grant permissions to various tenant admins.

apiVersion: lib.projectsveltos.io/v1alpha1 kind: RoleRequest metadata: name: full-access spec: clusterSelector: env=pre-production admin: eng roleRefs: - name: full-access namespace: default kind: ConfigMap
Enter fullscreen mode Exit fullscreen mode

where:

  1. ClusterSelector field is used to select a set of managed clusters where permissions will be granted;
  2. Admin field selects a tenant admin (later on we will cover how this identify a tenant admin);
  3. RoleRefs field references one of more ConfigMaps/Secrets. Each of those ConfigMaps/Secrets contains one or more ClusterRole/Role defining the permissions granted to the tenant admin by the platform admin.

Platform admin is the only admin in the management cluster with permissions to create/update/delete RoleRequest instances.

Tenant admin managing applications

Once RoleRequest instances have been created, tenant admins can use Sveltos ClusterProfile CRD to instruct Sveltos on what Kubernetes resources to deploy on which clusters.

Sveltos requires that following label

projectsveltos.io/admin-name: <admin>

is set on a ClusterProfile instances created by a tenant admin.
How to achieve that, is up to platform admin.

For instance, if:

  • Kyverno is present in the management cluster;
  • tenant admin are represented by Kubernetes ServiceAccount;

following Kyverno ClusterPolicy is all that is needed for label to be automatically added to each instance of a ClusterProfile:

apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: add-labels annotations: policies.kyverno.io/title: Add Labels policies.kyverno.io/description: >- Adds projectsveltos.io/admin-name label on each ClusterProfile created by tenant admin. It assumes each tenant admin is represented in the management cluster by a ServiceAccount. spec: validationFailureAction: enforce background: false rules: - name: add-labels match: resources: kinds: - ClusterProfile mutate: patchStrategicMerge: metadata: labels: projectsveltos.io/admin-name: "{{serviceAccountName}}"
Enter fullscreen mode Exit fullscreen mode

Examples

I have created management cluster on my laptop. It is a Kind cluster.
In the management cluster, I have deployed both ClusterAPI (with Kind as infrastructure provider) and Sveltos.

kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE capd-system capd-controller-manager-59b7bdbcf9-zd4xq 1/1 Running 0 30m capi-kubeadm-bootstrap-system capi-kubeadm-bootstrap-controller-manager-68f975d954-9r7lm 1/1 Running 0 30m capi-kubeadm-control-plane-system capi-kubeadm-control-plane-controller-manager-8476fb8676-kpnkd 1/1 Running 0 30m capi-system capi-controller-manager-69fd995599-67fh5 1/1 Running 0 30m ... projectsveltos access-manager-569dd6cccd-m676t 2/2 Running 0 59s projectsveltos classifier-manager-ddc8c756-bmxc9 2/2 Running 0 59s projectsveltos fm-controller-manager-5964fb8bff-2j2gx 2/2 Running 0 59s projectsveltos sveltoscluster-manager-75ddcf7cbd-q47hp 2/2 Running 0 58s
Enter fullscreen mode Exit fullscreen mode

I have one ClusterAPI powered cluster:

kubectl get clusters -A NAMESPACE NAME PHASE AGE VERSION default sveltos-management-workload Provisioned 10m v1.25.2
Enter fullscreen mode Exit fullscreen mode

and I have one GKE cluster registered with Sveltos

kubectl get sveltoscluster -A NAMESPACE NAME AGE gke production 6s
Enter fullscreen mode Exit fullscreen mode

Cluster labels are set as follow:

Cluster Label
default/sveltos-management-workload env:development
gke/production env:production

Fully reserve set of cluster to a tenant 

apiVersion: v1 kind: ConfigMap metadata: name: full-access namespace: default data: role.yaml: | apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: system-admin-full-access rules: - apiGroups: ["*"] resources: ["*"] verbs: ["*"] --- apiVersion: lib.projectsveltos.io/v1alpha1 kind: RoleRequest metadata: name: full-access spec: clusterSelector: env=production admin: system-admin roleRefs: - name: full-access namespace: default kind: ConfigMap
Enter fullscreen mode Exit fullscreen mode

By referencing ConfigMap default/full-access, above RoleRequest is reserving any cluster matching clusterSelector env=production to admin system-admin.

Using sveltosctl we can verify Sveltos is aware system-admin has full access to managed cluster with label env:production

./bin/sveltosctl show admin-rbac +-------------------------------+--------------+-----------+------------+-----------+----------------+-------+ | CLUSTER | ADMIN | NAMESPACE | API GROUPS | RESOURCES | RESOURCE NAMES | VERBS | +-------------------------------+--------------+-----------+------------+-----------+----------------+-------+ | SveltosCluster:gke/production | system-admin | * | * | * | * | * | +-------------------------------+--------------+-----------+------------+-----------+----------------+-------+
Enter fullscreen mode Exit fullscreen mode

If system-admin tries to deploy addons to such cluster, that operation will go through.

For instance, if system-admin posts following ClusterProfile

apiVersion: config.projectsveltos.io/v1alpha1 kind: ClusterProfile metadata: name: deploy-kyverno labels: projectsveltos.io/admin-name: system-admin spec: clusterSelector: env=production syncMode: Continuous helmCharts: - repositoryURL: https://kyverno.github.io/kyverno/ repositoryName: kyverno chartName: kyverno/kyverno chartVersion: v2.6.0 releaseName: kyverno-latest releaseNamespace: kyverno helmChartAction: Install
Enter fullscreen mode Exit fullscreen mode

we can verify Kyverno is deployed by Sveltos on behalf of system-admin:

./bin/sveltosctl show features +----------------+---------------+-----------+----------------+---------+-------------------------------+------------------+ | CLUSTER | RESOURCE TYPE | NAMESPACE | NAME | VERSION | TIME | CLUSTER PROFILES | +----------------+---------------+-----------+----------------+---------+-------------------------------+------------------+ | gke/production | helm chart | kyverno | kyverno-latest | 2.6.0 | 2023-02-03 14:06:48 -0800 PST | deploy-kyverno | +----------------+---------------+-----------+----------------+---------+-------------------------------+------------------+
Enter fullscreen mode Exit fullscreen mode

If system-admin tries to deploy same add-ons on clusters matching clusterSelector env:development that operation will fail.

Share cluster between tenants

Platform admin here is granting following permissions in all clusters matching ClusterSelector env=development

  1. Admin eng full access to namespace eng and some read permission in namespace shared-service-access;
  2. Admin hr full access to namespace human-resource. 
# ConfigMap contains a Role which gives # service read access to namespace shared-services apiVersion: v1 kind: ConfigMap metadata: name: shared-service-access namespace: default data: role.yaml: | apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: service-read-role namespace: shared-services rules: - apiGroups: [""] resources: ["services"] verbs: ["get,list"] --- # ConfigMap contains a Role which gives # full access to namespace eng apiVersion: v1 kind: ConfigMap metadata: name: eng-full-access namespace: default data: role.yaml: | apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: edit-role namespace: eng rules: - apiGroups: ["*"] resources: ["*"] verbs: ["*"] --- # RoleRequest gives admin 'eng': # - full access to namespace eng # - some read permission in namespace shared-service-access # in all clusters matching the label # selector env=development apiVersion: lib.projectsveltos.io/v1alpha1 kind: RoleRequest metadata: name: eng-access spec: clusterSelector: env=development admin: eng roleRefs: - name: eng-full-access namespace: default kind: ConfigMap - name: shared-service-access namespace: default kind: ConfigMap # ConfigMap contains a Role which gives # full access to namespace human-resource apiVersion: v1 kind: ConfigMap metadata: name: hr-full-access namespace: default data: role.yaml: | apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: edit-role namespace: human-resource rules: - apiGroups: ["*"] resources: ["*"] verbs: ["*"] --- # RoleRequest gives admin 'hr' access to namespace # 'human-resource' in all clusters matching the label # selector env=development apiVersion: lib.projectsveltos.io/v1alpha1 kind: RoleRequest metadata: name: hr-access spec: clusterSelector: env=development admin: hr roleRefs: - name: hr-full-access namespace: default kind: ConfigMap
Enter fullscreen mode Exit fullscreen mode

As before, we can use sveltosctl to verify granted permission:

./bin/sveltosctl show admin-rbac --namespace=default +---------------------------------------------+-------+-----------------+------------+-----------+----------------+----------+ | CLUSTER | ADMIN | NAMESPACE | API GROUPS | RESOURCES | RESOURCE NAMES | VERBS | +---------------------------------------------+-------+-----------------+------------+-----------+----------------+----------+ | Cluster:default/sveltos-management-workload | eng | eng | * | * | * | * | | Cluster:default/sveltos-management-workload | eng | shared-services | | services | services | get,list | | Cluster:default/sveltos-management-workload | hr | human-resource | * | * | * | * | +---------------------------------------------+-------+-----------------+------------+-----------+----------------+----------+
Enter fullscreen mode Exit fullscreen mode

If eng admin tries to deploy nginx in eng namespace, that will succeed.

apiVersion: v1 kind: ConfigMap metadata: name: nginx namespace: default data: nginx.yaml: | apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment namespace: eng labels: app: nginx spec: replicas: 3 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.14.2 ports: - containerPort: 80 --- apiVersion: config.projectsveltos.io/v1alpha1 kind: ClusterProfile metadata: name: deploy-nginx labels: projectsveltos.io/admin-name: eng spec: clusterSelector: env=development syncMode: Continuous policyRefs: - name: nginx namespace: default kind: ConfigMap
Enter fullscreen mode Exit fullscreen mode

nginx is deployed

./bin/sveltosctl show features +-------------------------------------+-----------------+-----------+------------------+---------+-------------------------------+------------------+ | CLUSTER | RESOURCE TYPE | NAMESPACE | NAME | VERSION | TIME | CLUSTER PROFILES | +-------------------------------------+-----------------+-----------+------------------+---------+-------------------------------+------------------+ | default/sveltos-management-workload | apps:Deployment | eng | nginx-deployment | N/A | 2023-02-03 14:40:19 -0800 PST | deploy-nginx | +-------------------------------------+-----------------+-----------+------------------+---------+-------------------------------+------------------+
Enter fullscreen mode Exit fullscreen mode

Sveltos multi-tenancy summarized

Contribute

Sveltos

Sveltos is an open source project. Contributions are more welcome 🤗

If you would like to know more about it or would like to see a new feature added to sveltos, please reach to us on slack. Any feedback is very much appreciated ❤️

Documentation can be found here

Top comments (1)

Collapse
 
gianlucam76 profile image
Gianluca •

Sveltos is an open source project.

Contributions are more welcome 🤗
If you like golang and Kubernetes and would like to help, please contact us. We would love to have you part of the sveltos project.

Chat with us on the Slack in the #projectsveltos channel Slack