Customizing kubelet configuration

Customizing kubelet configuration

System resources can be reserved through the configuration of the kubelet. This is recommended, because in the case of resource starvation the kubelet might not be able to evict pods and eventually make the node become NotReady. To do this, config files can include the kubeletExtraConfig field which accepts a free form yaml that will be embedded into the kubelet.yaml.

Some fields in the kubelet.yaml are set by eksctl and therefore are not overwritable, such as the address, clusterDomain, authentication, authorization, or serverTLSBootstrap.

The following example config file creates a nodegroup that reserves 300m vCPU, 300Mi of memory and 1Gi of ephemeral-storage for the kubelet; 300m vCPU, 300Mi of memory and 1Giof ephemeral storage for OS system daemons; and kicks in eviction of pods when there is less than 200Mi of memory available or less than 10% of the root filesystem.

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: dev-cluster-1
  region: eu-north-1

nodeGroups:
  - name: ng-1
    instanceType: m5a.xlarge
    desiredCapacity: 1
    kubeletExtraConfig:
        kubeReserved:
            cpu: "300m"
            memory: "300Mi"
            ephemeral-storage: "1Gi"
        kubeReservedCgroup: "/kube-reserved"
        systemReserved:
            cpu: "300m"
            memory: "300Mi"
            ephemeral-storage: "1Gi"
        evictionHard:
            memory.available:  "200Mi"
            nodefs.available: "10%"
        featureGates:
            DynamicKubeletConfig: true
            RotateKubeletServerCertificate: true # has to be enabled, otherwise it will be disabled

In this example, given instances of type m5a.xlarge which have 4 vCPUs and 16GiB of memory, the Allocatable amount of CPUs would be 3.4 and 15.4 GiB of memory. In addition, the DynamicKubeletConfig feature gate is also enabled. It is important to know that the values specified in the config file for the the fields in kubeletExtraconfig will completely overwrite the default values specified by eksctl.

IMPORTANT: by default eksctl sets featureGates.RotateKubeletServerCertificate=true, but when custom featureGates are provided, it will be unset. You should always include featureGates.RotateKubeletServerCertificate=true, unless you have to disable it.