Mastering Certificate Challenges with Ease
By Amrutha Paladugu
Author LinkedIn: https://www.linkedin.com/in/amrutha-paladugu/
Introduction:
Kong Hybrid mode represents a groundbreaking approach to API management, merging the control plane (CP) and data plane (DP) capabilities. This mode offers many benefits that empower organizations to achieve enhanced control, security, and performance in their API ecosystem. In this article, we’ll delve into a real-world scenario showcasing the capabilities of Kong Hybrid mode. Specifically, we’ll explore the deployment of Kong Hybrid across Google Kubernetes Engine (GKE) and an On-Premise Kubernetes cluster. We will focus on achieving secure communication, certificate management, workload deployment, and external access. Join us as we navigate this journey through a hybrid API management deployment.
Kong Hybrid architecture with one DP on the cloud and one DP on-premise.
Understanding Certificates:
Mutual Transport Layer Security (mTLS) plays a vital role in Kong’s security features by facilitating secure communication between services. When operating in hybrid mode, the authentication process relies on an mTLS handshake to ensure that communication between CP and DP nodes remains secure, requiring the presentation of certificates. Kong manages certificate key pairs in two distinct modes: Shared and PKI.
In the Shared mode (the default setting), Kong CLI generates a certificate/key pair, and copies of this pair are then distributed across nodes. CP and DP nodes share and utilize this certificate/key pair.
On the other hand, in the PKI mode, certificates signed by a central certificate authority (CA) are provided. Kong performs validation on both ends by verifying if they originate from the same CA. This approach effectively mitigates the risks associated with the transportation of private keys and provides increased security. If one of the DP nodes is compromised, an attacker won’t be able to affect other nodes in the Kong cluster.
In this article, we will use PKI mode and generate three different cert/key pairs for CP and the 2 DPs. All three pairs need to be signed by the same CA discussed above.
Additionally, we will provision a Let’s Encrypt certificate for DP1’s public-facing endpoint and verify encrypted traffic.
Certificate generation:
Step1: Generate CA cert using OpenSSL:
openssl genrsa -out ca.key 4096 openssl req -x509 -new -nodes -key ca.key -sha256 -days 365 -out ca.crt Step2: Generate private key and CSR(certificate signing request) for CP and DPs
openssl req -new -newkey rsa:2048 -nodes -keyout cp-tls.key -out cp-tls.csr -subj "/CN=*.kong.example.com" openssl req -new -newkey rsa:2048 -nodes -keyout dp1-tls.key -out dp1-tls.csr -subj "/CN=*.kong.example.com" openssl req -new -newkey rsa:2048 -nodes -keyout dp2-tls.key -out dp2-tls.csr -subj "/CN=*.127.0.0.1.nip.io" The CN (Common Name) here would be your wildcard domain.
Step3: Use CA from Step1 to sign the generated CSRs from Step2
openssl x509 -req -in cp-tls.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out cp-tls.crt -days 365 openssl x509 -req -in dp1-tls.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out dp1-tls.crt -days 365 openssl x509 -req -in dp2-tls.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out dp2-tls.crt -days 365 Step4: Create bundle certs:
cat cp-tls.crt ca.crt > cp-tls-bundle.crt cat dp1-tls.crt ca.crt > dp1-tls-bundle.crt cat dp2-tls.crt ca.crt > dp2-tls-bundle.crt Note: Save these certificates in an accessible directory (certs) for the CP and the DPs.
Install CP and DP1 on GKE:
Connect to a GKE cluster and use the next set of commands to install CP and DP1.
Create ns, Create secrets, update helm repo:
sudo snap install helm - classic helm repo add kong https://charts.konghq.com helm repo update kubectl create namespace kong-cp kubectl create namespace kong-dp1 Create secrets for Kong Enterprise license, Session Config, Admin password, and Postgresql password:
kubectl create secret generic kong-enterprise-license -n kong-cp \ - from-file=license=license.json \ - dry-run=client -o yaml | kubectl apply -f - kubectl create secret generic kong-session-config -n kong-cp \ - from-literal=portal_session_conf='{"storage":"kong","secret":"super_secret_salt_string","cookie_name":"portal_session","cookie_samesite":"off","cookie_secure":false, "cookie_domain": ".example.com"}' \ - from-literal=admin_gui_session_conf='{"cookie_name":"admin_session","cookie_samesite":"off","secret":"super_secret_salt_string","cookie_secure":false,"storage":"kong", "cookie_domain": ".example.com"}' kubectl create secret generic kong-manager-password -n kong-cp \ - from-literal=password=password kubectl create secret generic kong-cp-postgresql -n kong-cp \ - from-literal=host="enterprise-postgresql.kong.svc.cluster.local" \ - from-literal=password=password Note: If you don’t have access to an Enterprise license, you can still go ahead and try the installation but will not be able to access the Admin GUI.
Now, create Kubernetes Secrets for the cluster cert and CA cert for ns kong-cp.
kubectl create secret tls kong-cluster-cert -n kong-cp \ - cert=./certs/cp-tls-bundle.crt \ - key=./certs/cp-tls.key kubectl create secret generic kong-ca-cert -n kong-cp \ - from-file=ca.crt=./certs/ca.crt We use Helm to install the CP and DPs and use custom values.yaml files.
Update the values-cp.yaml with the below values:
image: repository: kong/kong-gateway tag: "3.4" secretVolumes: - kong-cluster-cert - kong-ca-cert env: role: control_plane cluster_mtls: pki cluster_ca_cert: /etc/secrets/kong-ca-cert/ca.crt cluster_cert: /etc/secrets/kong-cluster-cert/tls.crt cluster_cert_key: /etc/secrets/kong-cluster-cert/tls.key nginx_http_ssl_protocols: TLSv1.2 admin_gui_ssl_cert: /etc/secrets/kong-cluster-cert/tls.crt admin_gui_ssl_cert_key: /etc/secrets/kong-cluster-cert/tls.key admin_ssl_cert: /etc/secrets/kong-cluster-cert/tls.crt admin_ssl_cert_key: /etc/secrets/kong-cluster-cert/tls.key portal_gui_ssl_cert: /etc/secrets/kong-cluster-cert/tls.crt portal_gui_ssl_cert_key: /etc/secrets/kong-cluster-cert/tls.key portal_api_ssl_cert: /etc/secrets/kong-cluster-cert/tls.crt portal_api_ssl_cert_key: /etc/secrets/kong-cluster-cert/tls.key lua_ssl_verify_depth: 3 admin_api_uri: https://admin.kong.example.com admin_gui_url: https://manager.kong.example.com portal_auth: basic-auth portal_api_url: https://portal-api.kong.example.com portal_gui_url: https://portal-dev.kong.example.com portal_gui_host: portal-dev.kong.example.com portal_gui_protocol: https database: postgres pg_host: kong-cp-postgresql pg_database: kong pg_user: postgres pg_schema: kong pg_password: valueFrom: secretKeyRef: name: kong-cp-postgresql key: password portal_session_conf: valueFrom: secretKeyRef: name: kong-session-config key: portal_session_conf user: kong password: valueFrom: secretKeyRef: name: kong-manager-password key: password admin: enabled: true type: ClusterIP annotations: konghq.com/protocol: https labels: {} http: enabled: false tls: enabled: true servicePort: 443 containerPort: 8444 parameters: - http2 ingress: annotations: konghq.com/https-redirect-status-code: "301" nginx.ingress.kubernetes.io/backend-protocol: HTTPS enabled: true ingressClassName: kong-dev hostname: admin.kong.example.com path: / tls: kong-cluster-cert manager: enabled: true type: ClusterIP annotations: konghq.com/protocol: https labels: {} http: enabled: false tls: enabled: true servicePort: 443 containerPort: 8445 parameters: - http2 ingress: annotations: konghq.com/https-redirect-status-code: "301" nginx.ingress.kubernetes.io/backend-protocol: HTTPS enabled: true ingressClassName: kong-dev hostname: manager.kong.example.com path: / tls: kong-cluster-cert cluster: enabled: true type: LoadBalancer annotations: {} http: enabled: false tls: enabled: true servicePort: 443 containerPort: 8005 ingress: annotations: konghq.com/https-redirect-status-code: "301" nginx.ingress.kubernetes.io/backend-protocol: HTTPS enabled: true ingressClassName: kong-dev hostname: cp.kong.example.com path: / tls: kong-cluster-cert clustertelemetry: enabled: true type: LoadBalancer http: enabled: false tls: enabled: true servicePort: 443 containerPort: 8006 ingress: annotations: konghq.com/https-redirect-status-code: "301" nginx.ingress.kubernetes.io/backend-protocol: HTTPS enabled: true ingressClassName: kong-dev hostname: tel.kong.example.com path: / tls: kong-cluster-cert proxy: enabled: false ingressController: enabled: true ingressClass: kong-dev resources: limits: cpu: 200m memory: 256Mi requests: cpu: 100m memory: 128Mi installCRDs: false env: publish_service: kong-dp/kong-dp-kong-proxy kong_admin_token: valueFrom: secretKeyRef: name: kong-manager-password key: password enterprise: enabled: true license_secret: kong-enterprise-license # CHANGEME vitals: enabled: false rbac: enabled: true session_conf_secret: kong-session-config admin_api_auth: basic-auth portal: enabled: true smtp: enabled: false portal: enabled: true type: ClusterIP annotations: konghq.com/protocol: https labels: {} http: enabled: false tls: enabled: true servicePort: 443 containerPort: 8446 ingress: annotations: konghq.com/https-redirect-status-code: "301" nginx.ingress.kubernetes.io/backend-protocol: HTTPS enabled: true ingressClassName: kong-dev hostname: portal-dev.kong.example.com path: / tls: kong-cluster-cert portalapi: enabled: true type: ClusterIP annotations: konghq.com/protocol: https labels: {} http: enabled: false tls: enabled: true servicePort: 443 containerPort: 8447 ingress: annotations: konghq.com/https-redirect-status-code: "301" nginx.ingress.kubernetes.io/backend-protocol: HTTPS enabled: true ingressClassName: kong-dev hostname: portal-api.kong.example.com path: / tls: kong-cluster-cert postgresql: enabled: true auth: database: kong username: postgres existingSecret: kong-cp-postgresql secretKeys: adminPasswordKey: password Let’s try and understand the above values file:
The secrets we created, kong-cluster-cert and kong-ca-cert are mounted as volumes for the SSL certificates and are accessed at the specified path, for example:
cluster_ca_cert: /etc/secrets/kong-ca-cert/ca.crt
The environment variables are for configuring Kong and its components. They control settings related to SSL/TLS, database connections, authentication, and more. Also, they specify the role as “control_plane” and enable cluster mutual TLS (mTLS) using the PKI method. The admin API and GUI URLs, authentication methods, and other configuration parameters are also defined using the env variables.
Admin, Manager, Portal, Portal API, and PostgreSQL:
- These sections configure various components of Kong. All these components are enabled for the CP and would be disabled in the DP values yaml. Their type ( ClusterIP), TLS, and Ingress settings are also defined, including annotations, hostnames, paths, and TLS certificates for these components.
- PostgreSQL settings specify authentication details for connecting to the PostgreSQL database used by Kong.
- Enterprise-related settings include the license secret, RBAC configuration, SMTP, and vitals.
Proxy and Ingress Controller:
- The proxy section specifies whether the Kong proxy is enabled. For CP values, this would be disabled as the DP does all the proxying.
- Ingress Controller settings define the resources and environment variables for the Kong Ingress Controller.
Cluster and Cluster Telemetry:
- Cluster and cluster telemetry are essential for Kong Gateway’s Control Plane (CP) and Data Plane (DP) connection. DP instances connect to the CP through the CP cluster, which provides redundancy, load balancing, and configuration synchronization.
- Cluster telemetry plays a crucial role in monitoring the health of both CP and DP components, ensuring they can work together effectively and respond to changes and issues in real time. This setup provides a reliable and resilient CP-DP connection in Kong Gateway deployments.
Install the CP:
helm upgrade - install kong-cp kong/kong - namespace kong-cp -f values-cp.yaml
Once the CP is installed, verify that all pods are running and the Cluster and Clusterterlemetry services’ external LoadBalancer IPs are ready.
To install DP in the same cluster, follow the steps below and create license and certificate secrets for DP1.
kubectl create secret generic kong-enterprise-license -n kong-dp1 \ --from-file=license=license.json \ --dry-run=client -o yaml | kubectl apply -f - kubectl create secret tls kong-cluster-cert -n kong-dp1 \ - cert=./certs/dp1-tls-bundle.crt \ - key=./certs/dp1-tls.key kubectl create secret generic kong-ca-cert -n kong-dp1 \ - from-file=ca.crt=./certs/ca.crt To encrypt the oneway traffic to the public-facing end point of DP1, we use Let’s Encrypt CA. Use the below link to generate one for the domain you use.
SSL Certificate Generator: Free letsencrypt SSL in minutes:
A free SSL Certificate Generator. No login is required. Secure your site with a letsencrypt certificate. Includes a…
punchsalad.com
Create a secret for this cert in the kong-dp1 namespace:
kubectl create secret tls kong-tls-cert -n kong-dp1 \ - cert=./letsencrypt/ca.crt \ - key=./letsencrypt/ca.key Create a values-dp1.yaml file with the below values: image: repository: kong/kong-gateway tag: "3.4" secretVolumes: - kong-cluster-cert - kong-ca-cert - kong-tls-cert env: role: data_plane database: "off" status_listen: 0.0.0.0:8100 cluster_control_plane: cp.kong.example.com:443 cluster_telemetry_endpoint: tel.kong.example.com:443 cluster_mtls: pki cluster_server_name: cp.kong.example.com cluster_telemetry_server_name: tel.kong.example.com lua_ssl_verify_depth: 3 cluster_ca_cert: /etc/secrets/kong-ca-cert/ca.crt lua_ssl_trusted_certificate: /etc/secrets/kong-cluster-cert/tls.crt cluster_cert: /etc/secrets/kong-cluster-cert/tls.crt cluster_cert_key: /etc/secrets/kong-cluster-cert/tls.key ssl_cert: /etc/secrets/kong-tls-cert/tls.crt ssl_cert_key: /etc/secrets/kong-tls-cert/tls.key ingressController: enabled: false enterprise: enabled: true license_secret: kong-enterprise-license vitals: enabled: false portal: enabled: false rbac: enabled: false admin: enabled: false manager: enabled: false portal: enabled: false portalapi: enabled: false proxy: enabled: true type: LoadBalancer annotations: {} tls: enabled: true servicePort: 443 containerPort: 8443 parameters: - http2 # stream: {} ingress: enabled: true Please take note of the key differences here from the earlier values-cp yaml. All the Manager, Admin, Portal, Portalapi, and IngressController sections are disabled, while the Proxy section is enabled as type LoadBalancer. As explained earlier, ensuring the correct configuration of cluster and cluster telemetry endpoints is crucial.
Another point worth mentioning is the secret volume for the Let’s Encrypt certificate:kong-tls-cert. This is accessed through the below variables:
ssl_cert: /etc/secrets/kong-tls-cert/tls.crt
ssl_cert_key: /etc/secrets/kong-tls-cert/tls.key
Install DP1:
helm upgrade --install kong-dp1 kong/kong --namespace kong-dp1 -f values-dp1.yaml
Note the LoadBalancer IPs for the Cluster and Clustertelemetry service in the CP and the Proxy service in the DP. Map these values to the domain you intend to use. In our scenario, it would be as follows:
cp.kong.example.com - - - - - - - > 34.136.32.39 cp-tel.kong.example.com - - - - >34.68.58.255 *.kong.example.com - - - - - - - - >34.132.64.159 You should now be able to access Kong Manager (in case you are using an Enterprise license) at manager.kong.example.com, but the browser won’t recognize the OpenSSL CA, and hence it will appear as NotSecure as shown in the below image.
Access Admi GUI and view certificate (Not Secure- openSSL)
Deploy a sample application:
Let’s deploy a sample nodeJS application in the same GKE cluster in the default namespace, try accessing it through the DP endpoint, and see if the Let’s Encrypt certificate is served.
kubectl create deploy --image debianmaster/nodejs-welcome welcome kubectl expose deployment welcome --type=LoadBalancer --name=welcome --port=8080 --target-port=8080 
Deploying a sample nodeJS ‘welcome’ application
Add a service and a route to this application through the Kong manager.
Now, the workload can be accessed through dp1.kong.example.com/welcome. Click on the lock icon🔒present right before the URL and view the certificate. You will notice that the Let’s Encrypt certificate is being served, as shown in the below image.
Access the workload through the DP endpoint and verify the Let’s Encrypt cert.
Install DP2 on local K3s:\
Connect to a local K3s (lightweight Kubernetes) cluster and install the following command set to install DP2.
Create namespace for DP2
kubectl create ns kong-dp2
Create secrets for enterprise licenses and certificates:
kubectl create secret generic kong-enterprise-license - from-file=license=./license.json -n kong-dp2 - dry-run=client -o yaml | kubectl apply -f - kubectl create secret tls kong-cluster-cert -n kong-dp2 \ - cert=./certs/dp2-tls-bundle.crt \ - key=./certs/dp2-tls.key kubectl create secret generic kong-ca-cert -n kong-dp2 \ - from-file=ca.crt=./certs/ca.crt Create a values-dp2.yaml file with the below values:
image: repository: kong/kong-gateway tag: "3.4" secretVolumes: - kong-cluster-cert - kong-ca-cert env: role: data_plane cluster_control_plane: cp.kong.example.com:443 cluster_telemetry_endpoint: tel.kong.example.com:443 lua_ssl_trusted_certificate: /etc/secrets/kong-cluster-cert/tls.crt cluster_mtls: pki cluster_server_name: cp.kong.example.com cluster_telemetry_server_name: tel.kong.example.com cluster_ca_cert: /etc/secrets/kong-ca-cert/ca.crt cluster_cert: /etc/secrets/kong-cluster-cert/tls.crt cluster_cert_key: /etc/secrets/kong-cluster-cert/tls.key database: "off" audit_log: "on" headers: "off" anonymous_reports: "off" untrusted_lua: "off" enterprise: enabled: true license_secret: kong-enterprise-license vitals: enabled: false portal: enabled: false rbac: enabled: false admin: enabled: false manager: enabled: false portal: enabled: false portalapi: enabled: false ingressController: enabled: false proxy: enabled: true type: LoadBalancer annotations: { konghq.com/protocol: "https" } tls: enabled: true servicePort: 443 containerPort: 8443 parameters: - http2 ingress: enabled: true Install DP2 helm upgrade - install kong-dp2 kong/kong - namespace kong-dp2 -f values-dp2.yaml Deploy a sample application in the K3s cluster:
Let’s deploy a sample nodeJS application in the same K3s cluster in the default namespace and try to access it through the DP2 endpoint and verify.
kubectl create deploy --image debianmaster/nodejs-welcome hello kubectl expose deployment hello --type=LoadBalancer --name=hello --port=8080 --target-port=8080 
Verify deployment of nodeJS ‘hello’ application.
Add a service and a route to this application through the Kong manager.
Now the workload can be accessed through dp2.127.0.0.1.nip.io/hello
Conclusion:
Throughout this article, we embarked on a journey to uncover the transformative power of Kong Hybrid mode within API management. By harmonizing the control plane (CP) and data plane (DP), Kong Hybrid empowers organizations to balance control, security, and performance seamlessly. Through a real-world scenario, we showcased the deployment of Kong Hybrid across Google Kubernetes Engine (GKE) and an On-Premise Kubernetes cluster.
We navigated the intricate landscape of certificate management, emphasizing the pivotal role of Mutual Transport Layer Security (mTLS) and certificates in fortifying Kong’s security architecture. By embracing PKI mode, we ensured secure communication and mitigated the risks inherent in key transportation.
By following our straightforward, step-by-step instructions, we achieved successful deployments of Kong CP and DP1 within the GKE cluster, followed by Kong DP2 in an On-Premise K3s cluster. We highlighted the deployment of sample applications, demonstrated access through DP endpoints, and verified Let’s Encrypt certificates, reinforcing the foundation of a robust and secure API ecosystem.
As we wrap up this journey, remember the importance of resource cleanup to prevent lingering costs. The world of Kong Hybrid mode introduces a revolutionary approach that bridges the gap between Cloud and On-Premise, enabling the creation of APIs that transcend traditional confines. With mastery of this approach, you’ll confidently and innovatively navigate the dynamic realm of modern API management.
For more details visit: [https://zelarsoft.com/kong-consulting-and-licensing/]
Top comments (0)