Monday, August 18, 2025

What are steps involved in setting up a multi-node Kubernetes cluster on Ubuntu VMs using kubeadm.

0) Plan & prerequisites (all nodes)

Give each VM a unique hostname and ensure full network connectivity between them.

Recommended (comfortable) sizes: control-plane ≥2 vCPU / 4 GB RAM; workers ≥2 GB RAM.

Make sure your firewall or cloud security groups allow the required Kubernetes ports (you can adjust later).



Open these ports (typical defaults):

Control-plane inbound: 6443/tcp (API server), 2379-2380/tcp (etcd), 10250/tcp (kubelet), 10257/tcp (controller), 10259/tcp (scheduler).  

Workers inbound: 10250/tcp (kubelet), 10256/tcp (kube-proxy), 30000-32767/tcp,udp (NodePort services). 


1) System prep (run on all nodes)


# Update OS

sudo apt-get update -y


# 1.1 Disable swap (kubeadm default expects swap off)

sudo swapoff -a

sudo sed -ri '/\sswap\s/s/^/#/' /etc/fstab



Kernel modules & sysctls for container networking


# 1.2 Load required modules on boot

cat <<'EOF' | sudo tee /etc/modules-load.d/k8s.conf

overlay

br_netfilter

EOF

sudo modprobe overlay

sudo modprobe br_netfilter


# 1.3 Allow bridged traffic to be seen by iptables and enable forwarding

cat <<'EOF' | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf

net.bridge.bridge-nf-call-iptables  = 1

net.bridge.bridge-nf-call-ip6tables = 1

net.ipv4.ip_forward                 = 1

EOF

sudo sysctl --system


These sysctls and modules are the standard container runtime prerequisites.)  


2) Install and configure containerd (all nodes)


# Install containerd

sudo apt-get install -y containerd


# Generate a default config and switch to systemd cgroups (recommended)

sudo mkdir -p /etc/containerd

containerd config default | sudo tee /etc/containerd/config.toml >/dev/null

sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml


# Enable and restart

sudo systemctl enable --now containerd

sudo systemctl restart containerd




3) Install kubeadm, kubelet, kubectl (all nodes)


# Add Kubernetes APT keyring & repo (Kubernetes v1.33 line shown here)

sudo apt-get install -y apt-transport-https ca-certificates curl gpg

sudo mkdir -p /etc/apt/keyrings

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.33/deb/Release.key \

  | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg


echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] \

https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /' \

| sudo tee /etc/apt/sources.list.d/kubernetes.list


sudo apt-get update

sudo apt-get install -y kubelet kubeadm kubectl

sudo apt-mark hold kubelet kubeadm kubectl


4) Initialize the control-plane (control-plane node only)


Pick a Pod CIDR that matches your CNI choice. Two popular options:

Calico defaults to 192.168.0.0/16

Flannel defaults to 10.244.0.0/16


Below shows Calico (you can swap to Flannel later—see Step 6):


# Replace the advertise-address with this node's primary IP

POD_CIDR=192.168.0.0/16

API_ADVERTISE_IP=$(hostname -I | awk '{print $1}')


sudo kubeadm init \

  --pod-network-cidr=${POD_CIDR} \

  --apiserver-advertise-address=${API_ADVERTISE_IP}



When it completes, it prints two important things:

A kubeadm join ... command for workers

A note to set up your kubeconfig for kubectl on this node


Set kubeconfig for your current user:


mkdir -p $HOME/.kube

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

sudo chown "$(id -u)":"$(id -g)" $HOME/.kube/config


# Verify

kubectl get nodes


(Init/join workflow per kubeadm docs.) 



5) Install a CNI network plugin (control-plane)

You need a CNI so Pods can talk to each other. Choose one:


Option A — Calico (NetworkPolicy-capable)


# Install the Tigera operator

kubectl apply -f https://docs.projectcalico.org/manifests/tigera-operator.yaml

# Create a default Calico installation (uses 192.168.0.0/16 by default)

kubectl apply -f https://docs.projectcalico.org/manifests/custom-resources.yaml


# Wait for calico pods to be Ready

kubectl get pods -n tigera-operator

kubectl get pods -n calico-system


Option B — Flannel (simple & lightweight)


If you prefer Flannel, ensure you used --pod-network-cidr=10.244.0.0/16 in step 4, then:


kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml


(Official Flannel manifest.)  


Give the CNI a minute to roll out. kubectl get nodes should show the control-plane Ready once CNI is settled.


6) Join worker nodes (run on each worker)


On each worker VM, paste the kubeadm join ... command that kubeadm init printed.

It looks like:


sudo kubeadm join <API_SERVER_IP>:6443 \

  --token <token> \

  --discovery-token-ca-cert-hash sha256:<hash>


If you lost it, re-create a fresh join command on the control-plane:


sudo kubeadm token create --print-join-command


(Join procedure is part of standard kubeadm workflow.)  


Verify from the control-plane:


kubectl get nodes -o wide


7) (Optional) Basic sanity tests


# Test DNS & scheduling with a simple deployment and NodePort service

kubectl create deploy hello --image=nginx

kubectl expose deploy hello --port=80 --type=NodePort

kubectl get svc hello -o wide  # Note the NodePort to test via workerIP:nodePort



8) Firewalls and security groups (recap)


If you run a host firewall (ufw, firewalld) or cloud SGs, ensure the required ports from step 0 are open; otherwise, components may be NotReady. Official list here.


Common gotchas

Swap not fully disabled: kubelet won’t start cleanly. Re-run the swap commands.  

Cgroups mismatch: If kubelet logs complain about cgroups, ensure SystemdCgroup = true in /etc/containerd/config.toml, then systemctl restart containerd and systemctl restart kubelet.  

CNI not installed: Nodes stay NotReady. Install Calico/Flannel as in step 5 and wait for pods to be Ready.   

Ports blocked: API at :6443 unreachable or workers can’t join—open the ports listed earlier.




No comments:

Post a Comment