3

Resource Management for Pods and Containers describes how to set resource requets and limits for "regular" pods in Kubernetes. Is there a supported/reccomended way to set these limits for control plane components such as kube-apiserver?

Things I considered:

  • Modifying static manifests, e.g. in /etc/kubernetes/manifests/kube-apiserver.yaml. This could work but it will be overwritten by kubeadm during next upgrade.
  • Setting kube-reserved or system-reserved flags. This could work too however again - they are defined in just one ConfigMap (e.g. kubelet-config-1.21) and will be overwritten by kubeadm during node upgrade. The same limits will apply to control plane nodes and worker nodes and I don't want that.

I can overcome this with something like ansible but then ansible will be "figting" with kubeadm and I'd like to avoid that.

What problem am I trying to solve?

I have a small homelab kubernetes installation. I'd like to allow running regular pods on control plane node(s), but I'd like to be able to reserve some resources (primarily memory) for control plane components. I.e. I'd like to be able to set requests on things like kube-apiserver so that scheduler knows not to put any other pods (they will also have appropriate requests) in its place.

Improve this question
2
  • How exactly did you set up your cluster? What does it mean "I have a small homelab kubernetes installation."? Did you use Minikube? Commented Jan 12, 2022 at 9:34
  • @MikołajGłodziak no, "regular" cluster on 3 nodes built by kubeadm. Commented Jan 13, 2022 at 9:40

1 Answer 1

Reset to default
2

Is there a supported/reccomended way to set these limits for control plane components such as kube-apiserver?

Yes, you can use kubeadm init with patches command line flag. Look at this github page. The documentation of this thing could be also interested. See also official documentation: Customizing the control plane with patches:

Kubeadm allows you to pass a directory with patch files to InitConfiguration and JoinConfiguration on individual nodes. These patches can be used as the last customization step before the control plane component manifests are written to disk.

You can pass this file to kubeadm init with --config <YOUR CONFIG YAML>:

apiVersion: kubeadm.k8s.io/v1beta3 kind: InitConfiguration patches: directory: /home/user/somedir

Here's example how to set resources on kube-apiserver:

Create kube-apiserver.yaml file in some directory (e.g. /home/user/patches) with following contents:

spec: containers: - name: kube-apiserver resources: requests: memory: 512Mi limits: memory: 1024Mi

Then use --patches flag every time during node upgrade: use kubeadm upgrade node --patches /home/user/patches/ or kubeadm upgrade apply v1.22.4 --patches /home/user/patches/

The other option will be supply extra flags to control-plane components. For this, check this guide: Customizing the control plane with flags in ClusterConfiguration:

The kubeadm ClusterConfiguration object exposes a way for users to override the default flags passed to control plane components such as the APIServer, ControllerManager, Scheduler and Etcd. The components are defined using the following structures:

  • apiServer
  • controllerManager
  • scheduler
  • etcd
Improve this answer
1
  • 1
    Thank you, the patches route works. I've updated your answer with a more specific example of how I did that. Now I can use ansible to maintain different files in patches directory (if needed) and remember to use --patches flag when doing init or upgrades. Commented Jan 13, 2022 at 10:25

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.