Modifying kernel parameters in a Kubernetes cluster using a DaemonSet

Kernel parameters control the behavior of the operating system at runtime. They allow you to configure and fine-tune various aspects of the Linux kernel, such as networking, memory management, process handling, and security. These parameters are located in the /proc/sys directory on each node and can be dynamically modified at runtime using the sysctl command.

This guide outlines how to modify kernel parameters across all nodes in a Kubernetes cluster using a DaemonSet.

Identifying the kernel parameters to modify

Kernel parameters, managed via the sysctl command, are grouped into different categories depending on which part of the kernel they influence:

• Networking (net.*): Controls network-related settings such as buffer sizes, TCP/IP settings, and routing.
  Example: net.ipv4.ip_forward enables or disables IP packet forwarding, often used in routing scenarios.

• Memory Management (vm.*): Manages memory and swap behaviors.
  Example: vm.swappiness controls how aggressively the system swaps memory pages to disk.

• File System (fs.*): Configures file system-related limits and behaviors.
  Example: fs.file-max sets the maximum number of file descriptors the system can allocate.

• General Kernel Settings (kernel.*): Configures overall kernel behaviors.
  Example: kernel.hostname defines the system’s hostname.

• Security (kernel.random.*, net.ipv4.conf.*, etc.): Manages security settings such as IP forwarding, source address validation, and firewall rules.
  Example: net.ipv4.conf.all.rp_filter enables reverse path filtering for added network security.

• Process Limits (kernel.*): Controls limits for processes, such as the maximum number of processes or threads.
  Example: kernel.pid_max sets the maximum number of process IDs (PIDs) the system can allocate.

Creating a DaemonSet to modify kernel parameters

To apply kernel parameter changes across all nodes in the cluster, you can create a Kubernetes DaemonSet that runs privileged pods. This will ensure the changes are applied to every node.

Create a YAML file (e.g., sysctl-daemonset.yaml), copy/paste the following content into the file, save it and exit the text editor:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: sysctl-tuning
  namespace: kube-system
  labels:
    app: sysctl-tuning
spec:
  selector:
    matchLabels:
      app: sysctl-tuning
  template:
    metadata:
      labels:
        app: sysctl-tuning
    spec:
      hostNetwork: true  # Share the host's network namespace for network-related sysctl changes
      hostPID: true  # Access the host's PID namespace for sysctl commands
      initContainers:
      - name: sysctl-init  # Init container to set sysctl parameters
        image: busybox:latest
        command:
        - /bin/sh
        - -c
        - |
          sysctl -w net.core.rmem_max=7500000  # Set the maximum receive buffer size
          sysctl -w net.core.wmem_max=7500000  # Set the maximum send buffer size
        securityContext:
          privileged: true  # Privileged access to modify sysctl settings on the host
      containers:
      - name: sleep-container  # Main container to keep the pod running
        image: busybox:latest
        command:
        - /bin/sh
        - -c
        - sleep infinity  # Keep the pod alive indefinitely

Applying the DaemonSet

To apply the configuration, use the following command:

kubectl apply -f sysctl-daemonset.yaml

This command deploys the DaemonSet, which ensures that the kernel parameters are modified on all nodes.

Verifying changes

To verify that the DaemonSet is running on all nodes, use the following command:

kubectl get daemonset -n kube-system

To check if the kernel parameters were successfully updated on a node, SSH into the node and run:

ssh <node-name>
sysctl net.core.rmem_max
sysctl net.core.wmem_max

Cleaning up (Optional)

If the DaemonSet is no longer needed after the kernel parameters have been modified, you can delete it with the following command:

kubectl delete -f sysctl-daemonset.yaml