Why k3s and not stick with k8s?
I wanted to experiment with k3s. They package everything you need in a single binary, don’t package in deprecated parts of k8s, and it works on Intel, ARMv7 and ARM64. It seemed like it’d be a less painful way to runn Kubernetes on my ARM cluster.
Prerequisites
You must have set up DNS entries for the nodes you want to cluster, or update /etc/hosts on all the nodes so they can find each other.
Installing k3s
I chose to install k3s without the built-in traefik install so I could install that with a custom configuration. I also chose to use docker instead of the baked-in containerd so that I could also run containers outside k3s on my worker nodes without wasting RAM.
Installing the master
curl -sfL https://get.k3s.io > install-k3s.sh && chmod 755 ./install-k3s.sh sudo ./install-k3s.sh --no-deploy traefik --docker sudo chgrp docker /etc/rancher/k3s/k3s.yaml sudo chmod g+r /etc/rancher/k3s/k3s.yaml I also updated /etc/systemd/system/k3s.service to add
After=network-online.target cluster-mfsmount.service docker.service
because I don’t want k3s to attempt to start until after the docker service has started and the cluster’s moosefs distributed filesystem is mounted.
Once all that is done, copy the node token from /var/lib/rancher/k3s/server/node-token to each of the worker nodes.
Installing the workers
Copy /var/lib/rancher/k3s/server/node-token from the server to your worker.
Run
./install-k3s.sh --agent --server https://master-server:6443 --kubelet-arg="address=0.0.0.0" --token "$(cat node-token)" --docker Remove the --docker if you want to use the containerd bundled into k3s - I wanted to be able to also run apps in docker on my nodes and didn’t want it using extra RAM for another containerd.
If you’re using a distributed filesystem like I am, add
After=network-online.target cluster-mfsmount.service docker.service
to /etc/systemd/system/k3s-agent.service, and
After=network-online.target cluster-mfsmount.service
to /lib/systemd/system/docker.service to keep docker from starting until after the distributed filesystem is mounted.
Set up Networking
MetallB
I wanted to be able to use LoadBalancerIP entries in my cluster services to make using Traefik easier.
Installing MetallB
On my master node, I ran
kubectl apply -f https://raw.githubusercontent.com/danderson/metallb/master/manifests/metallb.yaml Configuring MetallB
I used the following configuration for metallb (in metallb-conf.yaml)
apiVersion: v1 kind: ConfigMap metadata: namespace: metallb-system name: config data: config: | address-pools: - name: default protocol: layer2 addresses: - 10.0.1.16/28 And applied it with kubectl apply -f metallb-conf.yaml.
This allows me to use 10.0.1.17 through 10.0.1.30 as LoadBalancerIP entries in my k8s service configurations. 14 entries should be more than enough for my immediate needs.
You will want to change the addresses entry to conform to your own network.
Traefik
Installed traefik with my own configuration, which I have posted on github:
Here are the configuration files I used - you’ll need to tweak them for your own network.
traefik-rbac.yaml
--- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: traefik-ingress-controller rules: - apiGroups: - "" resources: - services - endpoints - secrets verbs: - get - list - watch - apiGroups: - extensions resources: - ingresses verbs: - get - list - watch --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: traefik-ingress-controller roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: traefik-ingress-controller subjects: - kind: ServiceAccount name: traefik-ingress-controller namespace: kube-system traefik-configmap.yaml
apiVersion: v1 kind: ConfigMap metadata: name: traefik-conf namespace: kube-system data: traefik.toml: | defaultEntryPoints = ["http","https"] debug = false logLevel = "INFO" # Do not verify backend certificates (use https backends) InsecureSkipVerify = true [entryPoints] [entryPoints.http] address = ":80" compress = true [entryPoints.https] address = ":443" [entryPoints.https.tls] #Config to redirect http to https #[entryPoints] # [entryPoints.http] # address = ":80" # compress = true # [entryPoints.http.redirect] # entryPoint = "https" # [entryPoints.https] # address = ":443" # [entryPoints.https.tls] [web] address = ":8080" [kubernetes] [metrics] [metrics.prometheus] buckets=[0.1,0.3,1.2,5.0] entryPoint = "traefik" [ping] entryPoint = "http" traefik-deployment.yaml
--- apiVersion: v1 kind: ServiceAccount metadata: name: traefik-ingress-controller namespace: kube-system --- kind: Deployment apiVersion: extensions/v1beta1 metadata: name: traefik-ingress-controller namespace: kube-system labels: k8s-app: traefik-ingress-lb spec: replicas: 2 selector: matchLabels: k8s-app: traefik-ingress-lb template: metadata: labels: k8s-app: traefik-ingress-lb name: traefik-ingress-lb spec: serviceAccountName: traefik-ingress-controller terminationGracePeriodSeconds: 60 containers: - image: traefik:1.7.9 name: traefik-ingress-lb volumeMounts: - mountPath: /config name: config ports: - name: http containerPort: 80 - name: https containerPort: 443 - name: admin containerPort: 8080 args: - --api - --kubernetes - --configfile=/config/traefik.toml livenessProbe: httpGet: path: /ping port: 80 initialDelaySeconds: 3 periodSeconds: 3 timeoutSeconds: 1 affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: role operator: In values: - data topologyKey: kubernetes.io/hostname volumes: - name: config configMap: name: traefik-conf traefik-service.yaml
--- kind: Service apiVersion: v1 metadata: name: traefik-ingress-service namespace: kube-system labels: k8s-app: traefik-ingress-lb spec: selector: k8s-app: traefik-ingress-lb externalTrafficPolicy: Local ports: - protocol: TCP port: 80 name: web - protocol: TCP port: 443 name: https - protocol: TCP port: 8080 name: admin type: LoadBalancer loadBalancerIP: 10.0.1.20 --- apiVersion: extensions/v1beta1 kind: Ingress metadata: name: traefik-ingress-lb namespace: kube-system spec: rules: - host: traefik.example.com http: paths: - path: / backend: serviceName: traefik-ingress-service servicePort: admin You’ll want to change the loadBalancerIP entry and the host entry in the spec section to match your network and DNS configurations.
for traefik_yaml in traefik-rbac.yaml traefik-configmap.yaml traefik-deployment.yaml traefik-service.yaml do kubectl apply -f $traefik_yaml done Updates
- Updated URL for metallb install yaml file to use latest instead of pinning a specific version. k3s is actively being updated and the old version no longer worked.