Can I run workloads on server nodes?

Yes. By default k3s server nodes are not tainted, so the scheduler will place pods on them. In HA clusters this is common for small deployments, but dedicated agents give you clearer separation of control-plane and workload resources.

Do I need an external load balancer for HA?

Strictly speaking no — agents can list multiple server addresses — but a single stable endpoint (LB, VIP via kube-vip, or round-robin DNS) simplifies kubeconfig management and is strongly recommended for production.

How does embedded etcd compare to an external etcd cluster?

Embedded etcd is managed automatically by k3s and is suitable for most clusters. External etcd gives you independent backup and scaling of the datastore, but adds significant operational complexity. Start with embedded etcd unless you already manage an etcd cluster.

What happens if one server node goes down?

With three server nodes, etcd can tolerate one failure and still maintain quorum. The remaining two nodes continue serving the API. Workloads already running on agent nodes are unaffected; new scheduling continues normally.

How do I back up the embedded etcd data?

k3s includes a built-in snapshot command: run 'k3s etcd-snapshot save' on any server node. Snapshots are written to /var/lib/rancher/k3s/server/db/snapshots by default. Schedule this via a systemd timer or cron for automated backups.

Set Up a Multi-Node k3s Cluster

k3s turns a lightweight binary into a fully functional Kubernetes distribution. Its stripped-down footprint makes it practical on VMs, bare metal, and edge hardware alike. This guide walks through building a production-grade multi-node cluster with embedded etcd for high availability — no external database required. You will end up with three server nodes forming a quorum and one or more agent nodes ready to schedule workloads.

Architecture Overview

k3s collapses the Kubernetes control plane into a single binary. There are two node roles:

Server nodes — run the API server, scheduler, controller manager, and (when using embedded etcd) the etcd member. You need an odd number (3 or 5) for quorum.
Agent nodes — run only the kubelet and kube-proxy equivalents. They register against the server cluster and run workloads.

For HA you must bootstrap the first server with --cluster-init, then join additional servers. A load balancer or a virtual IP in front of the three API servers is strongly recommended for production. This guide uses a simple round-robin DNS entry for the cluster endpoint; swap in HAProxy, kube-vip, or a cloud LB as appropriate.

Prerequisites and Node Preparation

Each node needs a resolvable hostname, time synchronization, and open firewall ports before you touch k3s. Use at least three server nodes and at least one agent node. Tested with Ubuntu 24.04 LTS, Rocky Linux 9, and Arch (rolling, kernel 6.x).

Set unique hostnames

hostnamectl set-hostname k3s-server-1   # run the matching name on each node

Synchronize time

systemctl enable --now chronyd    # Fedora/RHEL/Rocky
systemctl enable --now systemd-timesyncd  # Debian/Ubuntu/Arch

Open required ports (choose your firewall)

The minimum ports between all nodes:

Port	Protocol	Purpose
6443	TCP	Kubernetes API server
2379-2380	TCP	etcd client and peer (server nodes only)
10250	TCP	kubelet metrics
8472	UDP	Flannel VXLAN overlay
51820-51821	UDP	WireGuard (if using encrypted flannel)

Ubuntu/Debian (ufw):

ufw allow 6443/tcp
ufw allow 2379:2380/tcp
ufw allow 10250/tcp
ufw allow 8472/udp
ufw reload

Fedora/Rocky/RHEL (firewalld):

firewall-cmd --permanent --add-port=6443/tcp
firewall-cmd --permanent --add-port=2379-2380/tcp
firewall-cmd --permanent --add-port=10250/tcp
firewall-cmd --permanent --add-port=8472/udp
firewall-cmd --reload

Arch (nftables): Add matching rules to /etc/nftables.conf and run systemctl restart nftables.

Bootstrap the First Server Node

All k3s nodes use the same official install script. On the first server only, pass --cluster-init to start a new embedded etcd cluster. Pick a cluster-wide token — treat it like a password.

curl -sfL https://get.k3s.io | sh -s - server \
  --cluster-init \
  --token "REPLACE_WITH_STRONG_SECRET" \
  --tls-san k3s-api.example.com \
  --tls-san 192.168.10.50

--tls-san adds extra Subject Alternative Names to the API server certificate. List every hostname and IP that clients or other nodes will use to reach the API — your load balancer VIP, DNS name, and individual node IPs. You can pass it multiple times.

Wait for the service to become active:

systemctl status k3s
journalctl -u k3s -f   # watch logs; etcd startup takes 20-30 s

Verify the first node is Ready

kubectl get nodes
# NAME            STATUS   ROLES                       AGE   VERSION
# k3s-server-1   Ready    control-plane,etcd,master   90s   v1.30.x+k3s1

Join Additional Server Nodes

Run this on each additional server node. Replace the IP with the first server's address or your load balancer endpoint.

curl -sfL https://get.k3s.io | sh -s - server \
  --server https://192.168.10.11:6443 \
  --token "REPLACE_WITH_STRONG_SECRET" \
  --tls-san k3s-api.example.com \
  --tls-san 192.168.10.50

Repeat for your third server. After both join, confirm the etcd quorum:

kubectl get nodes
# NAME            STATUS   ROLES                       AGE
# k3s-server-1   Ready    control-plane,etcd,master   5m
# k3s-server-2   Ready    control-plane,etcd,master   2m
# k3s-server-3   Ready    control-plane,etcd,master   1m

Join Agent Nodes

Agents do not run etcd or the control plane. Point them at the cluster endpoint (load balancer preferred) and provide the same token.

curl -sfL https://get.k3s.io | sh -s - agent \
  --server https://k3s-api.example.com:6443 \
  --token "REPLACE_WITH_STRONG_SECRET"

The agent service is named k3s-agent, not k3s:

systemctl status k3s-agent
journalctl -u k3s-agent -f

Configure kubectl on a Workstation

The kubeconfig lives on each server node at /etc/rancher/k3s/k3s.yaml. Copy it to your workstation and update the server address.

scp root@k3s-server-1:/etc/rancher/k3s/k3s.yaml ~/.kube/config
sed -i 's|127.0.0.1|k3s-api.example.com|g' ~/.kube/config
chmod 600 ~/.kube/config

Upgrading the Cluster

The safest upgrade path uses the System Upgrade Controller (SUC), which coordinates rolling upgrades via Kubernetes-native Plan resources. Manual upgrades work too but require careful sequencing.

Manual upgrade sequence

Upgrade server nodes one at a time. Drain workloads from each server (if it also runs pods) before upgrading.
On each server node, re-run the install script with the target version:

curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.31.0+k3s1 sh -s - server \
  --cluster-init   # only on first server; omit on subsequent servers
  --token "REPLACE_WITH_STRONG_SECRET" \
  --tls-san k3s-api.example.com

After all servers are upgraded, upgrade agents the same way using the agent subcommand and the new version string.

curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.31.0+k3s1 sh -s - agent \
  --server https://k3s-api.example.com:6443 \
  --token "REPLACE_WITH_STRONG_SECRET"

Never skip more than one minor version at a time. Check the k3s release notes for breaking changes between versions.

Verification

kubectl get nodes -o wide
kubectl get pods -A
kubectl get endpoints kubernetes

Deploy a quick smoke-test:

kubectl create deployment nginx --image=nginx --replicas=3
kubectl rollout status deployment/nginx
kubectl delete deployment nginx

Troubleshooting

Node stays NotReady

Check the node's k3s or k3s-agent log with journalctl -u k3s[-agent] -n 100. Common causes: firewall blocking port 8472/UDP (flannel VXLAN), time skew greater than 2 seconds between nodes, or an incorrect token.

etcd quorum lost

If two of three server nodes are permanently gone, etcd is stuck. You must force-promote a single member to a new cluster — this is a last resort:

# On the surviving server node
systemctl stop k3s
k3s server --cluster-reset   # DESTRUCTIVE — removes other etcd members
systemctl start k3s

After this, re-join your replacement server nodes from scratch.

TLS errors connecting to API

If you forgot to add a SAN at install time, regenerate the certificate. Edit /etc/rancher/k3s/config.yaml to add the missing tls-san entries, then delete the auto-generated certificate and restart:

rm /var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt
rm /var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key
systemctl restart k3s

k3s regenerates the certificate on startup using the current config.