Flomesh Ingress Controller with Kubernetes Multi-tenancy

Background

It's very rare for organizations to have or provide a dedicated Kubernetes cluster to each tenant, as it's generally more cost-effective to share a cluster. Sharing clusters reduces expenses and streamlines management. Sharing clusters, however, also comes with difficulties like managing noisy neighbors and ensuring security.

Clusters can be shared in many ways. In some cases, different applications may run in the same cluster. In other cases, multiple instances of the same application may run in the same cluster, one for each end user. All these types of sharing are frequently described using the umbrella term multi-tenancy.

While Kubernetes does not have first-class concepts of end users or tenants, it provides several features to help manage different tenancy requirements. These are discussed below.

• A common form of multi-tenancy is to share a cluster between multiple teams within an organization, each of whom may operate one or more workloads. In this scenario, members of the teams often have direct access to Kubernetes resources via tools such as kubectl, or indirect access through GitOps controllers or other types of release automation tools. There is often some level of trust between members of different teams, but Kubernetes policies such as RBAC, quotas, and network policies are essential to safely and fairly share clusters.

• The other major form of multi-tenancy frequently involves a Software-as-a-Service (SaaS) vendor running multiple instances of a workload for customers. This business model is so strongly associated with this deployment style that many people call it "SaaS tenancy." In this scenario, the customers do not have access to the cluster; Kubernetes is invisible from their perspective and is only used by the vendor to manage the workloads. Cost optimization is frequently a critical concern, and Kubernetes policies are used to ensure that the workloads are strongly isolated from each other.

Tenant Isolation

There are several ways to design and build multi-tenant solutions with Kubernetes at its control plane, data plane, or at both levels. Each of these methods comes with its own set of tradeoffs that impact the isolation level, implementation effort, operational complexity, and cost of service.

Kubernetes Control plane isolation ensures that different tenants cannot access or affect each other's Kubernetes API resources, and Flomesh Service Mesh (FSM) Ingress controller provides isolated Ingress controllers for Kubernetes Namespaces.

In Kubernetes, a Namespace provides a mechanism for isolating groups of API resources within a single cluster. This isolation has two key dimensions:

• Object names within a namespace can overlap with names in other namespaces, similar to files in folders. This allows tenants to name their resources without having to consider what other tenants are doing.

• Many Kubernetes security policies are scoped to namespaces. For example, RBAC Roles and Network Policies are namespace-scoped resources. Using RBAC, Users and Service Accounts can be restricted to a namespace.

In a multi-tenant environment, a Namespace helps segment a tenant's workload into a logical and distinct management unit. A common practice is to isolate every workload in its namespace, even if multiple workloads are operated by the same tenant. This ensures that each workload has its own identity and can be configured with an appropriate security policy.

In this article, we will learn how to use the Flomesh Service Mesh Ingress controller to create physical isolation of Ingress controllers when hosting multiple tenants in your Kubernetes cluster.

Flomesh Service Mesh (FSM)

FSM is an open-source product from Flomesh for Kubernetes north-south traffic, gateway API controller, and multi-cluster management. FSM uses Pipy, a programmable proxy at its core, and provides an Ingress controller, Gateway API controller, load balancer, cross-cluster service registration discovery, and more.

FSM Ingress Controller supports a multi-tenancy model via its concept of NamespacedIngress CRD where it deploys a physically isolated Ingress Controller for requested Namespace.

For example, the YAML below defines an Ingress Controller that monitors on port 100 and also creates a LoadBalancer type service for it that listens on port 100.

apiVersion: flomesh.io/v1alpha1 kind: NamespacedIngress metadata: name: namespaced-ingress-100 namespace: test-100 spec: serviceType: LoadBalancer ports: - name: http port: 100 protocol: TCP 

Install FSM

FSM provides a standard Helm chart, which can be installed via the Helm CLI.

$ helm repo add fsm https://flomesh-io.github.io/fsm $ helm repo update $ helm install fsm fsm/fsm --namespace flomesh --create-namespace --set fsm.ingress.namespaced=true 

Verify that all pods are up and running properly.

$ kubectl get po -n flomesh NAME READY STATUS RESTARTS AGE fsm-manager-6857f96858-sjksm 1/1 Running 0 55s fsm-repo-59bbbfdc5f-w7vg6 1/1 Running 0 55s fsm-bootstrap-8576c5ff4f-7qr7k 1/1 Running 0 55s fsm-cluster-connector-local-8f8fb87f6-h7z9j 1/1 Running 0 32s 

Create Sample Application

In this demo, we will be deploying httpbin service under a namespace httpbin.

# Create Namespace kubectl create ns httpbin # Deploy sample kubectl apply -f https://raw.githubusercontent.com/flomesh-io/osm-edge-docs/main/manifests/samples/httpbin/httpbin.yaml -n httpbin 

Creating a standalone Ingress Controller

The next step is to create a separate Ingress Controller for the namespace httpbin.

$ kubectl apply -f - <<EOF apiVersion: flomesh.io/v1alpha1 kind: NamespacedIngress metadata: name: namespaced-ingress-httpbin namespace: httpbin spec: serviceType: LoadBalancer http: port: name: http port: 81 nodePort: 30081 resources: limits: cpu: 500m memory: 200Mi requests: cpu: 100m memory: 20Mi EOF 

After executing the above command, you will see an Ingress Controller running successfully under the namespace httpbin.

kubectl get po -n httpbin -l app=fsm-ingress-pipy NAME READY STATUS RESTARTS AGE fsm-ingress-pipy-httpbin-5594ffcfcc-zl5gl 1/1 Running 0 58s 

At this point, there should be a corresponding Service under this namespace.

$ kubectl get svc -n httpbin -l app.kubernetes.io/component=controller NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE fsm-ingress-pipy-httpbin LoadBalancer 10.43.62.120 192.168.1.11 81:30081/TCP 2m49s 

Once you have the Ingress Controller, it's time to create the Ingress resource.

kubectl apply -f - <<EOF apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: httpbin namespace: httpbin spec: ingressClassName: pipy rules: - host: httpbin.org http: paths: - path: / pathType: Prefix backend: service: name: httpbin port: number: 14001 EOF 

Now we have created Ingress resource, let's do a quick curl to see if things are working as expected.

For my local setup demo, LoadBalancer IP is 192.168.1.11, your IP might be different. So ensure you are performing a curl against your setup ExternalIP.

curl -sI http://192.168.1.11:81/get -H "Host: httpbin.org" HTTP/1.1 200 OK server: gunicorn/19.9.0 date: Mon, 03 Oct 2022 12:02:04 GMT content-type: application/json content-length: 239 access-control-allow-origin: * access-control-allow-credentials: true connection: keep-alive 

Conclusion

In this blog post, you learned about Kubernetes multi-tenancy, features provided by Kubernetes, tenancy isolation levels, and how to use Flomesh Service Mesh (FSM) Ingress controller to set up isolated Ingress controller for namespaces.

Flomesh Service Mesh(FSM) from Flomesh is Kubernetes North-South traffic manager, provides Ingress controllers, Gateway API, Load Balancer, and cross-cluster service registration and service discovery. FSM uses Pipy - a programmable network proxy, as its data plane and is suitable for cloud, edge, and IoT.