Déployer et exposer globalement une application multi-clusters via K8GB et Liqo …

doctl compute droplet create \ --image ubuntu-22-04-x64 \ --size s-8vcpu-16gb \ --region ams3 \ --vpc-uuid 2812643c-3dd8-484a-b9f4-1fce42903731 \ k8gb 

root@k8gb:~# curl -fsSL https://get.docker.com | sh - # Executing docker install script, commit: 4f282167c425347a931ccfd95cc91fab041d414f + sh -c apt-get update -qq >/dev/null + sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq apt-transport-https ca-certificates curl >/dev/null + sh -c mkdir -p /etc/apt/keyrings && chmod -R 0755 /etc/apt/keyrings + sh -c curl -fsSL "https://download.docker.com/linux/ubuntu/gpg" | gpg --dearmor --yes -o /etc/apt/keyrings/docker.gpg + sh -c chmod a+r /etc/apt/keyrings/docker.gpg + sh -c echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu jammy stable" > /etc/apt/sources.list.d/docker.list + sh -c apt-get update -qq >/dev/null + sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq --no-install-recommends docker-ce docker-ce-cli containerd.io docker-compose-plugin docker-scan-plugin >/dev/null + version_gte 20.10 + [-z] + return 0 + sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq docker-ce-rootless-extras >/dev/null + sh -c docker version Client: Docker Engine - Community Version: 20.10.21 API version: 1.41 Go version: go1.18.7 Git commit: baeda1f Built: Tue Oct 25 18:01:58 2022 OS/Arch: linux/amd64 Context: default Experimental: true Server: Docker Engine - Community Engine: Version: 20.10.21 API version: 1.41 (minimum version 1.12) Go version: go1.18.7 Git commit: 3056208 Built: Tue Oct 25 17:59:49 2022 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.6.9 GitCommit: 1c90a442489720eec95342e1789ee8a5e1b9536f runc: Version: 1.1.4 GitCommit: v1.1.4-0-g5fd4c4d docker-init: Version: 0.19.0 GitCommit: de40ad0 ================================================================================ To run Docker as a non-privileged user, consider setting up the Docker daemon in rootless mode for your user: dockerd-rootless-setuptool.sh install Visit https://docs.docker.com/go/rootless/ to learn about rootless mode. To run the Docker daemon as a fully privileged service, but granting non-root users access, refer to https://docs.docker.com/go/daemon-access/ WARNING: Access to the remote API on a privileged Docker daemon is equivalent to root access on the host. Refer to the 'Docker daemon attack surface' documentation for details: https://docs.docker.com/go/attack-surface/ ================================================================================ root@k8gb:~# docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES root@k8gb:~# git clone https://github.com/k8gb-io/k8gb Cloning into 'k8gb'... remote: Enumerating objects: 11114, done. remote: Counting objects: 100% (12/12), done. remote: Compressing objects: 100% (12/12), done. remote: Total 11114 (delta 2), reused 2 (delta 0), pack-reused 11102 Receiving objects: 100% (11114/11114), 10.55 MiB | 15.73 MiB/s, done. Resolving deltas: 100% (5805/5805), done. root@k8gb:~# cd k8gb/ root@k8gb:~/k8gb# curl -s https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash Preparing to install k3d into /usr/local/bin k3d installed into /usr/local/bin/k3d Run 'k3d --help' to see what you can do with it. root@k8gb:~/k8gb# root@k8gb:~/k8gb# k3d --help https://k3d.io/ k3d is a wrapper CLI that helps you to easily create k3s clusters inside docker. Nodes of a k3d cluster are docker containers running a k3s image. All Nodes of a k3d cluster are part of the same docker network. Usage: k3d [flags] k3d [command] Available Commands: cluster Manage cluster(s) completion Generate completion scripts for [bash, zsh, fish, powershell | psh] config Work with config file(s) help Help about any command image Handle container images. kubeconfig Manage kubeconfig(s) node Manage node(s) registry Manage registry/registries version Show k3d and default k3s version Flags: -h, --help help for k3d --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) --version Show k3d and default k3s version Use "k3d [command] --help" for more information about a command. root@k8gb:~/k8gb# snap install kubectl --classic snap "kubectl" is already installed, see 'snap help refresh' root@k8gb:~/k8gb# curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 11156 100 11156 0 0 51694 0 --:--:-- --:--:-- --:--:-- 51888 Downloading https://get.helm.sh/helm-v3.10.2-linux-amd64.tar.gz Verifying checksum... Done. Preparing to install helm into /usr/local/bin helm installed into /usr/local/bin/helm 

root@k8gb:~/k8gb# make deploy-full-local-setup Creating 3 k8s clusters make create-local-cluster CLUSTER_NAME=edge-dns make[1]: Entering directory '/root/k8gb' Create local cluster edge-dns k3d cluster create -c k3d/edge-dns.yaml INFO[0000] Using config file k3d/edge-dns.yaml (k3d.io/v1alpha4#simple) INFO[0000] Prep: Network INFO[0000] Re-using existing network 'k3d-action-bridge-network' (ee0b6bae78a46fd6d885e84a0af19bc4cfc93c3af1aee9c0815e198b1baeaf4a) INFO[0000] Created image volume k3d-edgedns-images INFO[0000] Starting new tools node... INFO[0000] Starting Node 'k3d-edgedns-tools' INFO[0001] Creating node 'k3d-edgedns-server-0' INFO[0001] Using the k3d-tools node to gather environment information INFO[0001] HostIP: using network gateway 172.18.0.1 address INFO[0001] Starting cluster 'edgedns' INFO[0001] Starting servers... INFO[0001] Starting Node 'k3d-edgedns-server-0' INFO[0009] All agents already running. INFO[0009] All helpers already running. INFO[0009] Injecting records for hostAliases (incl. host.k3d.internal) and for 1 network members into CoreDNS configmap... INFO[0011] Cluster 'edgedns' created successfully! INFO[0011] You can now use it like this: kubectl cluster-info make[1]: Leaving directory '/root/k8gb' make[1]: Entering directory '/root/k8gb' Create local cluster test-gslb1 k3d cluster create -c k3d/test-gslb1.yaml INFO[0000] Using config file k3d/test-gslb1.yaml (k3d.io/v1alpha4#simple) INFO[0000] Prep: Network INFO[0000] Re-using existing network 'k3d-action-bridge-network' (ee0b6bae78a46fd6d885e84a0af19bc4cfc93c3af1aee9c0815e198b1baeaf4a) INFO[0000] Created image volume k3d-test-gslb1-images INFO[0000] Starting new tools node... INFO[0000] Starting Node 'k3d-test-gslb1-tools' INFO[0001] Creating node 'k3d-test-gslb1-server-0' INFO[0001] Creating node 'k3d-test-gslb1-agent-0' INFO[0001] Using the k3d-tools node to gather environment information INFO[0001] HostIP: using network gateway 172.18.0.1 address INFO[0001] Starting cluster 'test-gslb1' INFO[0001] Starting servers... INFO[0001] Starting Node 'k3d-test-gslb1-server-0' INFO[0009] Starting agents... INFO[0009] Starting Node 'k3d-test-gslb1-agent-0' INFO[0014] All helpers already running. INFO[0014] Injecting records for hostAliases (incl. host.k3d.internal) and for 3 network members into CoreDNS configmap... INFO[0016] Cluster 'test-gslb1' created successfully! INFO[0016] You can now use it like this: kubectl cluster-info make[1]: Leaving directory '/root/k8gb' make[1]: Entering directory '/root/k8gb' Create local cluster test-gslb2 k3d cluster create -c k3d/test-gslb2.yaml INFO[0000] Using config file k3d/test-gslb2.yaml (k3d.io/v1alpha4#simple) INFO[0000] Prep: Network INFO[0000] Re-using existing network 'k3d-action-bridge-network' (ee0b6bae78a46fd6d885e84a0af19bc4cfc93c3af1aee9c0815e198b1baeaf4a) INFO[0000] Created image volume k3d-test-gslb2-images INFO[0000] Starting new tools node... INFO[0000] Starting Node 'k3d-test-gslb2-tools' INFO[0001] Creating node 'k3d-test-gslb2-server-0' INFO[0001] Creating node 'k3d-test-gslb2-agent-0' INFO[0001] Using the k3d-tools node to gather environment information INFO[0001] HostIP: using network gateway 172.18.0.1 address INFO[0001] Starting cluster 'test-gslb2' INFO[0001] Starting servers... INFO[0001] Starting Node 'k3d-test-gslb2-server-0' INFO[0009] Starting agents... INFO[0009] Starting Node 'k3d-test-gslb2-agent-0' INFO[0013] All helpers already running. INFO[0013] Injecting records for hostAliases (incl. host.k3d.internal) and for 5 network members into CoreDNS configmap... INFO[0016] Cluster 'test-gslb2' created successfully! INFO[0016] You can now use it like this: kubectl cluster-info make[1]: Leaving directory '/root/k8gb' make deploy-stable-version DEPLOY_APPS=true make[1]: Entering directory '/root/k8gb' Deploying EdgeDNS kubectl --context k3d-edgedns apply -f deploy/edge/ secret/ddns-key created deployment.apps/edge created service/bind created configmap/zone created make[2]: Entering directory '/root/k8gb' Deploy local cluster test-gslb1 kubectl config use-context k3d-test-gslb1 Switched to context "k3d-test-gslb1". Create namespace kubectl apply -f deploy/namespace.yaml namespace/k8gb created Deploy GSLB operator from v0.10.0 make deploy-k8gb-with-helm make[3]: Entering directory '/root/k8gb' # create rfc2136 secret kubectl -n k8gb create secret generic rfc2136 --from-literal=secret=96Ah/a2g0/nLeFGK+d/0tzQcccf9hCEIy34PoXX2Qg8= || true secret/rfc2136 created helm repo add --force-update k8gb https://www.k8gb.io "k8gb" has been added to your repositories cd chart/k8gb && helm dependency update walk.go:74: found symbolic link in path: /root/k8gb/chart/k8gb/LICENSE resolves to /root/k8gb/LICENSE. Contents of linked file included and used Getting updates for unmanaged Helm repositories... ...Successfully got an update from the "https://absaoss.github.io/coredns-helm" chart repository Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "k8gb" chart repository Update Complete. ⎈Happy Helming!⎈ Saving 1 charts Downloading coredns from repo https://absaoss.github.io/coredns-helm Deleting outdated charts helm -n k8gb upgrade -i k8gb k8gb/k8gb -f "" \ --set k8gb.clusterGeoTag='eu' --set k8gb.extGslbClustersGeoTags='us' \ --set k8gb.reconcileRequeueSeconds=10 \ --set k8gb.dnsZoneNegTTL=10 \ --set k8gb.imageTag=v0.10.0 \ --set k8gb.log.format=simple \ --set k8gb.log.level=debug \ --set rfc2136.enabled=true \ --set k8gb.edgeDNSServers[0]=172.18.0.1:1053 \ --set externaldns.image=absaoss/external-dns:rfc-ns1 \ --wait --timeout=2m0s Release "k8gb" does not exist. Installing it now. NAME: k8gb LAST DEPLOYED: Fri Nov 11 19:54:26 2022 NAMESPACE: k8gb STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: done _ ___ _ | | _( _ ) ___| |__ | |/ / _ \ / _` | '_ \ | < (_) | (_| | |_) | |_|\_\ ___/ \__ , |_.__/ & all dependencies are installed |___/ 1. Check if your DNS Zone is served by K8GB CoreDNS $ kubectl -n k8gb run -it --rm --restart=Never --image=infoblox/dnstools:latest dnstools --command -- /usr/bin/dig @k8gb-coredns SOA . +short If everything is fine then you are expected to see similar output: 

make[3]: Leaving directory '/root/k8gb' Deploy Ingress helm repo add --force-update nginx-stable https://kubernetes.github.io/ingress-nginx "nginx-stable" has been added to your repositories helm repo update Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "k8gb" chart repository ...Successfully got an update from the "nginx-stable" chart repository Update Complete. ⎈Happy Helming!⎈ helm -n k8gb upgrade -i nginx-ingress nginx-stable/ingress-nginx \ --version 4.0.15 -f deploy/ingress/nginx-ingress-values.yaml Release "nginx-ingress" does not exist. Installing it now. NAME: nginx-ingress LAST DEPLOYED: Fri Nov 11 19:54:37 2022 NAMESPACE: k8gb STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: The ingress-nginx controller has been installed. It may take a few minutes for the LoadBalancer IP to be available. You can watch the status by running 'kubectl --namespace k8gb get services -o wide -w nginx-ingress-ingress-nginx-controller' An example Ingress that makes use of the controller: apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: example namespace: foo spec: ingressClassName: nginx rules: - host: www.example.com http: paths: - backend: service: name: exampleService port: number: 80 path: / # This section is only required if TLS is to be enabled for the Ingress tls: - hosts: - www.example.com secretName: example-tls If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided: apiVersion: v1 kind: Secret metadata: name: example-tls namespace: foo data: tls.crt: <base64 encoded cert> tls.key: <base64 encoded key> type: kubernetes.io/tls make[3]: Entering directory '/root/k8gb' Deploy GSLB cr kubectl apply -f deploy/crds/test-namespace.yaml namespace/test-gslb created sed -i 's/cloud\.example\.com/cloud.example.com/g' "deploy/crds/k8gb.absa.oss_v1beta1_gslb_cr.yaml" kubectl apply -f "deploy/crds/k8gb.absa.oss_v1beta1_gslb_cr.yaml" gslb.k8gb.absa.oss/test-gslb created git checkout -- "deploy/crds/k8gb.absa.oss_v1beta1_gslb_cr.yaml" sed -i 's/cloud\.example\.com/cloud.example.com/g' "deploy/crds/k8gb.absa.oss_v1beta1_gslb_cr_failover.yaml" kubectl apply -f "deploy/crds/k8gb.absa.oss_v1beta1_gslb_cr_failover.yaml" gslb.k8gb.absa.oss/test-gslb-failover created git checkout -- "deploy/crds/k8gb.absa.oss_v1beta1_gslb_cr_failover.yaml" Deploy podinfo kubectl apply -f deploy/test-apps service/unhealthy-app created deployment.apps/unhealthy-app created helm repo add podinfo https://stefanprodan.github.io/podinfo "podinfo" has been added to your repositories helm upgrade --install frontend --namespace test-gslb -f deploy/test-apps/podinfo/podinfo-values.yaml \ --set ui.message="` kubectl -n k8gb describe deploy k8gb | awk '/CLUSTER_GEO_TAG/ { printf $2 }'`" \ --set image.repository="ghcr.io/stefanprodan/podinfo" \ podinfo/podinfo \ --version 5.1.1 Release "frontend" does not exist. Installing it now. NAME: frontend LAST DEPLOYED: Fri Nov 11 19:54:41 2022 NAMESPACE: test-gslb STATUS: deployed REVISION: 1 NOTES: 1. Get the application URL by running these commands: echo "Visit http://127.0.0.1:8080 to use your application" kubectl -n test-gslb port-forward deploy/frontend-podinfo 8080:9898 make[3]: Leaving directory '/root/k8gb' Wait until Ingress controller is ready kubectl -n k8gb wait --for=condition=Ready pod -l app.kubernetes.io/name=ingress-nginx --timeout=600s pod/nginx-ingress-ingress-nginx-controller-7cs7g condition met pod/nginx-ingress-ingress-nginx-controller-qx5db condition met test-gslb1 deployed! make[2]: Leaving directory '/root/k8gb' make[2]: Entering directory '/root/k8gb' Deploy local cluster test-gslb2 kubectl config use-context k3d-test-gslb2 Switched to context "k3d-test-gslb2". Create namespace kubectl apply -f deploy/namespace.yaml namespace/k8gb created Deploy GSLB operator from v0.10.0 make deploy-k8gb-with-helm make[3]: Entering directory '/root/k8gb' # create rfc2136 secret kubectl -n k8gb create secret generic rfc2136 --from-literal=secret=96Ah/a2g0/nLeFGK+d/0tzQcccf9hCEIy34PoXX2Qg8= || true secret/rfc2136 created helm repo add --force-update k8gb https://www.k8gb.io "k8gb" has been added to your repositories cd chart/k8gb && helm dependency update walk.go:74: found symbolic link in path: /root/k8gb/chart/k8gb/LICENSE resolves to /root/k8gb/LICENSE. Contents of linked file included and used Getting updates for unmanaged Helm repositories... ...Successfully got an update from the "https://absaoss.github.io/coredns-helm" chart repository Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "k8gb" chart repository ...Successfully got an update from the "podinfo" chart repository ...Successfully got an update from the "nginx-stable" chart repository Update Complete. ⎈Happy Helming!⎈ Saving 1 charts Downloading coredns from repo https://absaoss.github.io/coredns-helm Deleting outdated charts helm -n k8gb upgrade -i k8gb k8gb/k8gb -f "" \ --set k8gb.clusterGeoTag='us' --set k8gb.extGslbClustersGeoTags='eu' \ --set k8gb.reconcileRequeueSeconds=10 \ --set k8gb.dnsZoneNegTTL=10 \ --set k8gb.imageTag=v0.10.0 \ --set k8gb.log.format=simple \ --set k8gb.log.level=debug \ --set rfc2136.enabled=true \ --set k8gb.edgeDNSServers[0]=172.18.0.1:1053 \ --set externaldns.image=absaoss/external-dns:rfc-ns1 \ --wait --timeout=2m0s Release "k8gb" does not exist. Installing it now. NAME: k8gb LAST DEPLOYED: Fri Nov 11 19:55:03 2022 NAMESPACE: k8gb STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: done _ ___ _ | | _( _ ) ___| |__ | |/ / _ \ / _` | '_ \ | < (_) | (_| | |_) | |_|\_\ ___/ \__ , |_.__/ & all dependencies are installed |___/ 1. Check if your DNS Zone is served by K8GB CoreDNS $ kubectl -n k8gb run -it --rm --restart=Never --image=infoblox/dnstools:latest dnstools --command -- /usr/bin/dig @k8gb-coredns SOA . +short If everything is fine then you are expected to see similar output: 

make[3]: Leaving directory '/root/k8gb' Deploy Ingress helm repo add --force-update nginx-stable https://kubernetes.github.io/ingress-nginx "nginx-stable" has been added to your repositories helm repo update Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "k8gb" chart repository ...Successfully got an update from the "podinfo" chart repository ...Successfully got an update from the "nginx-stable" chart repository Update Complete. ⎈Happy Helming!⎈ helm -n k8gb upgrade -i nginx-ingress nginx-stable/ingress-nginx \ --version 4.0.15 -f deploy/ingress/nginx-ingress-values.yaml Release "nginx-ingress" does not exist. Installing it now. NAME: nginx-ingress LAST DEPLOYED: Fri Nov 11 19:55:14 2022 NAMESPACE: k8gb STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: The ingress-nginx controller has been installed. It may take a few minutes for the LoadBalancer IP to be available. You can watch the status by running 'kubectl --namespace k8gb get services -o wide -w nginx-ingress-ingress-nginx-controller' An example Ingress that makes use of the controller: apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: example namespace: foo spec: ingressClassName: nginx rules: - host: www.example.com http: paths: - backend: service: name: exampleService port: number: 80 path: / # This section is only required if TLS is to be enabled for the Ingress tls: - hosts: - www.example.com secretName: example-tls If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided: apiVersion: v1 kind: Secret metadata: name: example-tls namespace: foo data: tls.crt: <base64 encoded cert> tls.key: <base64 encoded key> type: kubernetes.io/tls make[3]: Entering directory '/root/k8gb' Deploy GSLB cr kubectl apply -f deploy/crds/test-namespace.yaml namespace/test-gslb created sed -i 's/cloud\.example\.com/cloud.example.com/g' "deploy/crds/k8gb.absa.oss_v1beta1_gslb_cr.yaml" kubectl apply -f "deploy/crds/k8gb.absa.oss_v1beta1_gslb_cr.yaml" gslb.k8gb.absa.oss/test-gslb created git checkout -- "deploy/crds/k8gb.absa.oss_v1beta1_gslb_cr.yaml" sed -i 's/cloud\.example\.com/cloud.example.com/g' "deploy/crds/k8gb.absa.oss_v1beta1_gslb_cr_failover.yaml" kubectl apply -f "deploy/crds/k8gb.absa.oss_v1beta1_gslb_cr_failover.yaml" gslb.k8gb.absa.oss/test-gslb-failover created git checkout -- "deploy/crds/k8gb.absa.oss_v1beta1_gslb_cr_failover.yaml" Deploy podinfo kubectl apply -f deploy/test-apps service/unhealthy-app created deployment.apps/unhealthy-app created helm repo add podinfo https://stefanprodan.github.io/podinfo "podinfo" already exists with the same configuration, skipping helm upgrade --install frontend --namespace test-gslb -f deploy/test-apps/podinfo/podinfo-values.yaml \ --set ui.message="` kubectl -n k8gb describe deploy k8gb | awk '/CLUSTER_GEO_TAG/ { printf $2 }'`" \ --set image.repository="ghcr.io/stefanprodan/podinfo" \ podinfo/podinfo \ --version 5.1.1 Release "frontend" does not exist. Installing it now. NAME: frontend LAST DEPLOYED: Fri Nov 11 19:55:20 2022 NAMESPACE: test-gslb STATUS: deployed REVISION: 1 NOTES: 1. Get the application URL by running these commands: echo "Visit http://127.0.0.1:8080 to use your application" kubectl -n test-gslb port-forward deploy/frontend-podinfo 8080:9898 make[3]: Leaving directory '/root/k8gb' Wait until Ingress controller is ready kubectl -n k8gb wait --for=condition=Ready pod -l app.kubernetes.io/name=ingress-nginx --timeout=600s pod/nginx-ingress-ingress-nginx-controller-tvqn5 condition met pod/nginx-ingress-ingress-nginx-controller-nnw6p condition met test-gslb2 deployed! make[2]: Leaving directory '/root/k8gb' make[1]: Leaving directory '/root/k8gb' 

root@k8gb:~# k3d cluster list NAME SERVERS AGENTS LOADBALANCER edgedns 1/1 0/0 false test-gslb1 1/1 1/1 false test-gslb2 1/1 1/1 false root@k8gb:~# kubectl cluster-info --context k3d-edgedns && kubectl cluster-info --context k3d-test-gslb1 && kubectl cluster-info --context k3d-test-gslb2 Kubernetes control plane is running at https://0.0.0.0:43459 CoreDNS is running at https://0.0.0.0:43459/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. Kubernetes control plane is running at https://0.0.0.0:36415 CoreDNS is running at https://0.0.0.0:36415/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. Kubernetes control plane is running at https://0.0.0.0:43505 CoreDNS is running at https://0.0.0.0:43505/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. 

root@k8gb:~/k8gb# make deploy-prometheus 

root@k8gb:~# dig @localhost -p 1053 roundrobin.cloud.example.com +short +tcp ;; Connection to ::1#1053(::1) for roundrobin.cloud.example.com failed: connection refused. ;; Connection to ::1#1053(::1) for roundrobin.cloud.example.com failed: connection refused. ;; Connection to ::1#1053(::1) for roundrobin.cloud.example.com failed: connection refused. 172.18.0.4 172.18.0.6 172.18.0.3 172.18.0.5 root@k8gb:~# for c in k3d-test-gslb{1,2}; do kubectl get no -ocustom-columns="NAME:.metadata.name,IP:status.addresses[0].address" --context $c; done NAME IP k3d-test-gslb1-server-0 172.18.0.3 k3d-test-gslb1-agent-0 172.18.0.4 NAME IP k3d-test-gslb2-server-0 172.18.0.5 k3d-test-gslb2-agent-0 172.18.0.6 

root@k8gb:~# curl localhost:80 -H "Host:roundrobin.cloud.example.com" && curl localhost:81 -H "Host:roundrobin.cloud.example.com" { "hostname": "frontend-podinfo-7cf64f696-952m9", "version": "5.1.1", "revision": "", "color": "#34577c", "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", "message": "eu", "goos": "linux", "goarch": "amd64", "runtime": "go1.15.6", "num_goroutine": "9", "num_cpu": "8" }{ "hostname": "frontend-podinfo-59888d86d6-fm77n", "version": "5.1.1", "revision": "", "color": "#34577c", "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", "message": "us", "goos": "linux", "goarch": "amd64", "runtime": "go1.15.6", "num_goroutine": "9", "num_cpu": "8" 

root@k8gb:~/k8gb# make init-failover kubectl config use-context k3d-test-gslb2 Switched to context "k3d-test-gslb2". kubectl apply -f "deploy/crds/k8gb.absa.oss_v1beta1_gslb_cr_failover.yaml" gslb.k8gb.absa.oss/test-gslb-failover unchanged kubectl config use-context k3d-test-gslb1 Switched to context "k3d-test-gslb1". kubectl apply -f "deploy/crds/k8gb.absa.oss_v1beta1_gslb_cr_failover.yaml" gslb.k8gb.absa.oss/test-gslb-failover unchanged make start-test-app make[1]: Entering directory '/root/k8gb' kubectl scale deployment frontend-podinfo -n test-gslb --replicas=2 deployment.apps/frontend-podinfo scaled make[1]: Leaving directory '/root/k8gb' root@k8gb:~/k8gb# make test-failover { "hostname": "frontend-podinfo-7cf64f696-c8s5s", "version": "5.1.1", "revision": "", "color": "#34577c", "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", "message": "eu", "goos": "linux", "goarch": "amd64", "runtime": "go1.15.6", "num_goroutine": "8", "num_cpu": "8" }pod "busybox" deleted root@k8gb:~/k8gb# make stop-test-app kubectl scale deployment frontend-podinfo -n test-gslb --replicas=0 deployment.apps/frontend-podinfo scaled root@k8gb:~/k8gb# make test-failover { "hostname": "frontend-podinfo-59888d86d6-fm77n", "version": "5.1.1", "revision": "", "color": "#34577c", "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", "message": "us", "goos": "linux", "goarch": "amd64", "runtime": "go1.15.6", "num_goroutine": "13", "num_cpu": "8" }pod "busybox" deleted root@k8gb:~/k8gb# make start-test-app kubectl scale deployment frontend-podinfo -n test-gslb --replicas=2 deployment.apps/frontend-podinfo scaled root@k8gb:~/k8gb# make test-failover { "hostname": "frontend-podinfo-7cf64f696-jn54t", "version": "5.1.1", "revision": "", "color": "#34577c", "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", "message": "eu", "goos": "linux", "goarch": "amd64", "runtime": "go1.15.6", "num_goroutine": "9", "num_cpu": "8" }pod "busybox" deleted 

root@k8gb:~/k8gb# make test-round-robin { "hostname": "frontend-podinfo-7cf64f696-shlbd", "version": "5.1.1", "revision": "", "color": "#34577c", "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", "message": "eu", "goos": "linux", "goarch": "amd64", "runtime": "go1.15.6", "num_goroutine": "9", "num_cpu": "8" }pod "busybox" deleted root@k8gb:~/k8gb# make test-round-robin { "hostname": "frontend-podinfo-59888d86d6-fm77n", "version": "5.1.1", "revision": "", "color": "#34577c", "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", "message": "us", "goos": "linux", "goarch": "amd64", "runtime": "go1.15.6", "num_goroutine": "12", "num_cpu": "8" }pod "busybox" deleted 

root@k8gb:~/k8gb# make destroy-full-local-setup 

root@k8gb:~# liqoctl liqoctl is a CLI tool to install and manage Liqo. Liqo is a platform to enable dynamic and decentralized resource sharing across Kubernetes clusters, either on-prem or managed. Liqo allows to run pods on a remote cluster seamlessly and without any modification of Kubernetes and the applications. With Liqo it is possible to extend the control and data plane of a Kubernetes cluster across the cluster's boundaries, making multi-cluster native and transparent: collapse an entire remote cluster to a local virtual node, enabling workloads offloading, resource management and cross-cluster communication compliant with the standard Kubernetes approach. Usage: liqoctl [command] Available Commands: completion Generate the autocompletion script for the specified shell generate Generate data/commands to perform additional operations help Help about any command install Install/upgrade Liqo in the selected cluster move Move an object to a different cluster offload Offload a resource to remote clusters peer Enable a peering towards a remote cluster status Show the status of Liqo uninstall Uninstall Liqo from the selected cluster unoffload Unoffload a resource from remote clusters unpeer Disable a peering towards a remote cluster version Print the liqo CLI version and the deployed Liqo version Flags: --cluster string The name of the kubeconfig cluster to use --context string The name of the kubeconfig context to use -h, --help help for liqoctl --kubeconfig string Path to the kubeconfig file to use for CLI requests --user string The name of the kubeconfig user to use -v, --verbose Enable verbose logs (default false) Use "liqoctl [command] --help" for more information about a command. root@k8gb:~# git clone https://github.com/liqotech/liqo Cloning into 'liqo'... remote: Enumerating objects: 28194, done. remote: Counting objects: 100% (261/261), done. remote: Compressing objects: 100% (170/170), done. remote: Total 28194 (delta 132), reused 173 (delta 81), pack-reused 27933 Receiving objects: 100% (28194/28194), 34.98 MiB | 23.34 MiB/s, done. Resolving deltas: 100% (18901/18901), done. root@k8gb:~# cd liqo/examples/global-ingress/ root@k8gb:~/liqo/examples/global-ingress# ls manifests setup.sh 

root@k8gb:~/liqo/examples/global-ingress# ./setup.sh SUCCESS No cluster "edgedns" is running. SUCCESS No cluster "gslb-eu" is running. SUCCESS No cluster "gslb-us" is running. SUCCESS Cluster "edgedns" has been created. SUCCESS Cluster "gslb-eu" has been created. SUCCESS Cluster "gslb-us" has been created. SUCCESS Bind server has been deployed. SUCCESS K8gb has been installed on cluster. SUCCESS Ingress-nginx has been installed on cluster. SUCCESS Liqo has been installed on cluster "gslb-eu". SUCCESS K8gb has been installed on cluster. SUCCESS Ingress-nginx has been installed on cluster. SUCCESS Liqo has been installed on cluster "gslb-us". 

root@k8gb:~/liqo/examples/global-ingress# export KUBECONFIG_DNS=$(k3d kubeconfig write edgedns) export KUBECONFIG=$(k3d kubeconfig write gslb-eu) export KUBECONFIG_US=$(k3d kubeconfig write gslb-us) root@k8gb:~/liqo/examples/global-ingress# PEER_US=$(liqoctl generate peer-command --only-command --kubeconfig $KUBECONFIG_US) root@k8gb:~/liqo/examples/global-ingress# echo "$PEER_US" | bash INFO Peering enabled INFO Authenticated to cluster "gslb-us" INFO Outgoing peering activated to the remote cluster "gslb-us" INFO Network established to the remote cluster "gslb-us" INFO Node created for remote cluster "gslb-us" INFO Peering successfully established root@k8gb:~/liqo/examples/global-ingress# kubectl get foreignclusters NAME TYPE OUTGOING PEERING INCOMING PEERING NETWORKING AUTHENTICATION AGE gslb-us OutOfBand Established None Established Established 18s 

root@k8gb:~/liqo/examples/global-ingress# kubectl get node --selector=liqo.io/type=virtual-node NAME STATUS ROLES AGE VERSION liqo-gslb-us Ready agent 93s v1.22.6+k3s1 

root@k8gb:~/liqo/examples/global-ingress# kubectl create namespace podinfo namespace/podinfo created root@k8gb:~/liqo/examples/global-ingress# liqoctl offload namespace podinfo --namespace-mapping-strategy EnforceSameName INFO Offloading of namespace "podinfo" correctly enabled INFO Offloading completed successfully root@k8gb:~/liqo/examples/global-ingress# helm upgrade --install podinfo --namespace podinfo \ podinfo/podinfo -f manifests/values/podinfo.yaml Release "podinfo" does not exist. Installing it now. NAME: podinfo LAST DEPLOYED: Fri Nov 11 21:46:59 2022 NAMESPACE: podinfo STATUS: deployed REVISION: 1 NOTES: 1. Get the application URL by running these commands: http://liqo.cloud.example.com/ root@k8gb:~/liqo/examples/global-ingress# kubectl get ingress -n podinfo NAME CLASS HOSTS ADDRESS PORTS AGE podinfo nginx liqo.cloud.example.com 172.21.0.3,172.21.0.4 80 42s 

root@k8gb:~/liqo/examples/global-ingress# kubectl get gslbs.k8gb.absa.oss -n podinfo podinfo -o yaml apiVersion: k8gb.absa.oss/v1beta1 kind: Gslb metadata: annotations: k8gb.io/strategy: roundRobin meta.helm.sh/release-name: podinfo meta.helm.sh/release-namespace: podinfo creationTimestamp: "2022-11-11T21:47:01Z" finalizers: - k8gb.absa.oss/finalizer generation: 2 name: podinfo namespace: podinfo ownerReferences: - apiVersion: networking.k8s.io/v1 blockOwnerDeletion: true controller: true kind: Ingress name: podinfo uid: e4fe5d03-5aa6-4883-b204-5e63ef8a0b31 resourceVersion: "1858" uid: a450617c-7922-4272-865b-4cbc7e09ce03 spec: ingress: ingressClassName: nginx rules: - host: liqo.cloud.example.com http: paths: - backend: service: name: podinfo port: number: 9898 path: / pathType: ImplementationSpecific strategy: dnsTtlSeconds: 30 splitBrainThresholdSeconds: 300 type: roundRobin status: geoTag: eu healthyRecords: liqo.cloud.example.com: - 172.21.0.3 - 172.21.0.4 - 172.21.0.5 - 172.21.0.6 serviceHealth: liqo.cloud.example.com: Healthy root@k8gb:~/liqo/examples/global-ingress# kubectl get gslbs.k8gb.absa.oss -n podinfo podinfo -o yaml --kubeconfig $KUBECONFIG_US apiVersion: k8gb.absa.oss/v1beta1 kind: Gslb metadata: annotations: k8gb.io/strategy: roundRobin meta.helm.sh/release-name: podinfo meta.helm.sh/release-namespace: podinfo creationTimestamp: "2022-11-11T21:47:01Z" finalizers: - k8gb.absa.oss/finalizer generation: 2 name: podinfo namespace: podinfo ownerReferences: - apiVersion: networking.k8s.io/v1 blockOwnerDeletion: true controller: true kind: Ingress name: podinfo uid: 7519e30f-0a46-4c23-86b4-5970c9c030bb resourceVersion: "1692" uid: 1c3422c8-8c4f-4d2c-a659-9f0907694ab0 spec: ingress: ingressClassName: nginx rules: - host: liqo.cloud.example.com http: paths: - backend: service: name: podinfo port: number: 9898 path: / pathType: ImplementationSpecific strategy: dnsTtlSeconds: 30 splitBrainThresholdSeconds: 300 type: roundRobin status: geoTag: us healthyRecords: liqo.cloud.example.com: - 172.21.0.5 - 172.21.0.6 - 172.21.0.3 - 172.21.0.4 serviceHealth: liqo.cloud.example.com: Healthy 

root@k8gb:~/liqo/examples/global-ingress# kubectl get po,svc -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system pod/coredns-96cc4f57d-ss69w 1/1 Running 0 18m k8gb pod/k8gb-6ff7b8d894-mqv4g 1/1 Running 0 18m k8gb pod/external-dns-6bd44c4856-47k5q 1/1 Running 0 18m k8gb pod/nginx-ingress-ingress-nginx-controller-68jkc 1/1 Running 0 18m k8gb pod/nginx-ingress-ingress-nginx-controller-w6jgj 1/1 Running 0 18m liqo pod/liqo-network-manager-7f775cdbf9-q2dcb 1/1 Running 0 18m liqo pod/liqo-route-tmqfl 1/1 Running 0 18m liqo pod/liqo-controller-manager-db9cf996d-qvdtl 1/1 Running 0 18m liqo pod/liqo-route-h5mk7 1/1 Running 0 18m liqo pod/liqo-gateway-59b8745db9-t9gtw 1/1 Running 0 18m liqo pod/liqo-crd-replicator-68cb95df4f-9r2cm 1/1 Running 0 18m liqo pod/liqo-proxy-54bf9dbb8d-h48vm 1/1 Running 0 18m liqo pod/liqo-auth-6f6d6946fb-pqdj9 1/1 Running 0 18m liqo pod/liqo-metric-agent-94f8b96fc-62p7s 1/1 Running 0 18m k8gb pod/k8gb-coredns-5bc6689949-pv8bs 1/1 Running 0 18m liqo-tenant-gslb-us-726836 pod/virtual-kubelet-5657cbd4bb-p79jv 1/1 Running 0 15m podinfo pod/podinfo-68f5575b95-wqqbg 1/1 Running 0 10m podinfo pod/podinfo-68f5575b95-vlbgp 1/1 Running 0 10m NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default service/kubernetes ClusterIP 10.30.0.1 <none> 443/TCP 19m kube-system service/kube-dns ClusterIP 10.30.0.10 <none> 53/UDP,53/TCP,9153/TCP 19m k8gb service/k8gb-coredns ClusterIP 10.30.154.126 <none> 53/UDP 18m liqo service/liqo-network-manager ClusterIP 10.30.183.107 <none> 6000/TCP 18m liqo service/liqo-metric-agent ClusterIP 10.30.194.131 <none> 443/TCP 18m liqo service/liqo-auth NodePort 10.30.38.82 <none> 443:31758/TCP 18m liqo service/liqo-controller-manager ClusterIP 10.30.137.139 <none> 9443/TCP 18m liqo service/liqo-proxy ClusterIP 10.30.181.12 <none> 8118/TCP 18m liqo service/liqo-gateway NodePort 10.30.49.231 <none> 5871:31366/UDP 18m podinfo service/podinfo ClusterIP 10.30.150.228 <none> 9898/TCP,9999/TCP 10m root@k8gb:~/liqo/examples/global-ingress# k3d cluster list NAME SERVERS AGENTS LOADBALANCER edgedns 1/1 0/0 false gslb-eu 1/1 1/1 false gslb-us 1/1 1/1 false root@k8gb:~/liqo/examples/global-ingress# kubectl get po,svc -A --kubeconfig $KUBECONFIG_US NAMESPACE NAME READY STATUS RESTARTS AGE kube-system pod/coredns-96cc4f57d-88b9s 1/1 Running 0 21m k8gb pod/external-dns-768c97f5c8-dqstb 1/1 Running 0 20m k8gb pod/k8gb-864c495ddb-9jv2b 1/1 Running 0 20m k8gb pod/nginx-ingress-ingress-nginx-controller-g26vh 1/1 Running 0 20m k8gb pod/nginx-ingress-ingress-nginx-controller-jrjjl 1/1 Running 0 20m liqo pod/liqo-route-796x7 1/1 Running 0 20m liqo pod/liqo-crd-replicator-68cb95df4f-zbdsb 1/1 Running 0 20m liqo pod/liqo-proxy-54bf9dbb8d-l77mr 1/1 Running 0 20m liqo pod/liqo-metric-agent-94f8b96fc-rc8vz 1/1 Running 0 20m liqo pod/liqo-controller-manager-d54767454-z2zg8 1/1 Running 0 20m liqo pod/liqo-gateway-59b8745db9-4n6gv 1/1 Running 0 20m liqo pod/liqo-network-manager-7f775cdbf9-7tn7j 1/1 Running 0 20m liqo pod/liqo-route-zcqmc 1/1 Running 0 20m k8gb pod/k8gb-coredns-5bc6689949-twc27 1/1 Running 0 20m liqo pod/liqo-auth-95c87b89b-nk9jb 1/1 Running 0 20m podinfo pod/podinfo-68f5575b95-wqqbg 1/1 Running 0 13m NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default service/kubernetes ClusterIP 10.30.0.1 <none> 443/TCP 21m kube-system service/kube-dns ClusterIP 10.30.0.10 <none> 53/UDP,53/TCP,9153/TCP 21m k8gb service/k8gb-coredns ClusterIP 10.30.180.140 <none> 53/UDP 20m liqo service/liqo-network-manager ClusterIP 10.30.133.122 <none> 6000/TCP 20m liqo service/liqo-metric-agent ClusterIP 10.30.157.244 <none> 443/TCP 20m liqo service/liqo-proxy ClusterIP 10.30.27.174 <none> 8118/TCP 20m liqo service/liqo-auth NodePort 10.30.128.127 <none> 443:31335/TCP 20m liqo service/liqo-controller-manager ClusterIP 10.30.138.46 <none> 9443/TCP 20m liqo service/liqo-gateway NodePort 10.30.156.207 <none> 5871:30349/UDP 20m podinfo service/podinfo ClusterIP 10.30.48.175 <none> 9898/TCP,9999/TCP 13m 

root@k8gb:~/liqo/examples/global-ingress# HOSTNAME="liqo.cloud.example.com" K8GB_COREDNS_IP=$(kubectl get svc k8gb-coredns -n k8gb -o custom-columns='IP:spec.clusterIP' --no-headers) kubectl run -it --rm curl --restart=Never --image=curlimages/curl:7.82.0 --command \ --overrides "{\"spec\":{\"dnsConfig\":{\"nameservers\":[\"${K8GB_COREDNS_IP}\"]},\"dnsPolicy\":\"None\"}}" \ -- curl $HOSTNAME -v * Trying 172.21.0.3:80... * Connected to liqo.cloud.example.com (172.21.0.3) port 80 (#0) > GET / HTTP/1.1 > Host: liqo.cloud.example.com > User-Agent: curl/7.82.0-DEV > Accept: */* > * Mark bundle as not supporting multiuse < HTTP/1.1 200 OK < Date: Fri, 11 Nov 2022 21:55:06 GMT < Content-Type: application/json; charset=utf-8 < Content-Length: 392 < Connection: keep-alive < X-Content-Type-Options: nosniff < { "hostname": "podinfo-68f5575b95-vlbgp", "version": "6.2.3", "revision": "8615cb75d926ea0ba5353b1d56867868c737bf5e", "color": "#34577c", "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", "message": "greetings from podinfo v6.2.3", "goos": "linux", "goarch": "amd64", "runtime": "go1.19.3", "num_goroutine": "8", "num_cpu": "8" * Connection #0 to host liqo.cloud.example.com left intact }pod "curl" deleted root@k8gb:~/liqo/examples/global-ingress# kubectl run -it --rm curl --restart=Never --image=curlimages/curl:7.82.0 --command \ --overrides "{\"spec\":{\"dnsConfig\":{\"nameservers\":[\"${K8GB_COREDNS_IP}\"]},\"dnsPolicy\":\"None\"}}" \ -- curl $HOSTNAME -v * Trying 172.21.0.4:80... * Connected to liqo.cloud.example.com (172.21.0.4) port 80 (#0) > GET / HTTP/1.1 > Host: liqo.cloud.example.com > User-Agent: curl/7.82.0-DEV > Accept: */* > * Mark bundle as not supporting multiuse < HTTP/1.1 200 OK < Date: Fri, 11 Nov 2022 21:55:31 GMT < Content-Type: application/json; charset=utf-8 < Content-Length: 392 < Connection: keep-alive < X-Content-Type-Options: nosniff < { "hostname": "podinfo-68f5575b95-wqqbg", "version": "6.2.3", "revision": "8615cb75d926ea0ba5353b1d56867868c737bf5e", "color": "#34577c", "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", "message": "greetings from podinfo v6.2.3", "goos": "linux", "goarch": "amd64", "runtime": "go1.19.3", "num_goroutine": "8", "num_cpu": "8" * Connection #0 to host liqo.cloud.example.com left intact }pod "curl" deleted