RKE2 High Availability Installation: From Zero to Bulletproof in 60 Minutes

So you've learned about RKE2's high availability architecture and now you're ready to get your hands dirty? Perfect timing! In this follow-up to my RKE2 High Availability Architecture post, I'll walk through the actual installation process of building a production-ready RKE2 HA cluster with a multi-tier load balancing setup.

Think of this as building a fortress, we'll have multiple layers of defense against failures. Our architecture includes a main load balancer, dual HAProxy servers for redundancy, and three master nodes. If any component fails, your cluster keeps running. By the end of this guide, you'll have a bulletproof Kubernetes cluster that can weather any storm.

Let's remember our architecture one more time before starting with the installation.

RKE2 high availability deployment architecture: This production setup ensures your Kubernetes cluster survives failures at any layer through redundant HAProxy servers and multiple master nodes

Prerequisites

Before we start, ensure you have:

• 3 servers for master nodes (for true HA)

• 2 servers for HAProxy load balancers

• 1 main load balancer (cloud provider LB, self provisioned LB or another HAProxy)

• At least 2 servers for worker nodes

• Operating System: Ubuntu 20.04/22.04, RHEL 8.x, or Rocky Linux 8.x (I prefer Ubuntu)

• Resources per master node: 4 CPU cores, 8GB RAM, 100GB disk

• Resources per HAProxy node: 2 CPU cores, 4GB RAM, 20GB disk

• Resources per worker node: 2 CPU cores, 4GB RAM, 50GB disk

• Network connectivity: All nodes can communicate with each other

• Root or sudo access on all nodes

Part 1: Preparing Your Servers - The Foundation Matters

Making Your Nodes Friends: Hostname Resolution

First, let's ensure all nodes can resolve each other by hostname. When your cluster components are communicating thousands of times per second, hostname resolution beats memorizing IP addresses!

On all nodes (masters, workers, and HAProxy servers), update the /etc/hosts file:

# Edit hosts file on all nodes
sudo nano /etc/hosts

# Add entries for all cluster nodes
# Replace with your actual IP addresses
192.168.1.10    rke2-master-1
192.168.1.11    rke2-master-2
192.168.1.12    rke2-master-3
192.168.1.20    rke2-worker-1
192.168.1.21    rke2-worker-2
192.168.1.30    rke2-haproxy-1
192.168.1.31    rke2-haproxy-2
192.168.1.100   rke2-lb         # Main load balancer VIP

The Secret Sauce: Kernel Modules and Network Magic

Here's where things get interesting! Kubernetes needs specific kernel modules to work its container networking magic. Let me explain what's happening under the hood.

Run the following on all master and worker nodes (skip HAProxy servers):

# Configure kernel modules to load at boot
# We use 'tee' to write the content and redirect output to /dev/null for cleaner execution
# This creates a persistent configuration file that tells the system which modules to load
sudo tee /etc/modules-load.d/k8s.conf > /dev/null << EOF
overlay
br_netfilter
EOF

Why these specific modules?

• overlay: This is the storage driver for containers. Think of it as a clever filing system that lets containers share common layers (like base images) while keeping their unique changes separate. It's like having multiple transparent sheets stacked on top of each other, you see the combined image, but each layer remains independent. Without this, every container would need its own complete copy of the operating system, wasting massive amounts of disk space!

• br_netfilter: This module enables bridge netfilter support, allowing iptables rules to work on bridged traffic. In Kubernetes terms, it lets your firewall rules apply to traffic moving between containers on the same host. Without this, pods on the same node could bypass your network policies - imagine having security guards at the main gate but leaving the back door wide open!

# Load the modules immediately without rebooting
# modprobe is the command that inserts modules into the running kernel
sudo modprobe overlay
sudo modprobe br_netfilter

# Verify modules are loaded - should see both modules in the output
lsmod | grep -E 'overlay|br_netfilter'

Now, for the network parameters that make Kubernetes networking possible:

# Enable iptables to see bridged traffic and IPv4 forwarding
# Again using tee to create a persistent sysctl configuration
cat << EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF

The networking magic explained:

• bridge-nf-call-iptables = 1: This tells the Linux kernel to pass bridged IPv4 traffic through iptables chains. Kubernetes uses iptables extensively for service routing and network policies. Without this, your services won't be reachable! It's like having a postal system where letters can be sent but never delivered.

• bridge-nf-call-ip6tables = 1: Same as above but for IPv6 traffic. Even if you're not using IPv6 today, enabling this future-proofs your cluster.

• ip_forward = 1: This is crucial! It turns your Linux server into a router, allowing it to forward packets between network interfaces. Without this, your node becomes a dead end - packets come in but can't go anywhere else. This is essential for pod-to-pod communication across nodes. Think of it as turning your server from a parking lot into a highway interchange.

# Apply the kernel parameters immediately
# --system tells sysctl to load settings from all system configuration files
sudo sysctl --system

# Verify the settings took effect - all should return "1"
sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables net.ipv4.ip_forward

Part 2: Building Your Load Balancing Fortress - HAProxy Setup

Why Multi-Tier Load Balancing?

Before diving into configuration, let's understand our architecture. We're building a three-layer cake of availability:

1. Main Load Balancer: The single entry point (often a cloud provider LB with built-in HA)

2. Dual HAProxy Servers: Provides redundancy and intelligent health checking

3. Three Master Nodes: The actual Kubernetes control plane

This design means you can lose an HAProxy server AND a master node, and your cluster still runs perfectly!

Configuring HAProxy Servers

On both HAProxy servers (rke2-haproxy-1 and rke2-haproxy-2):

# Install HAProxy
sudo apt-get update && sudo apt-get install -y haproxy

# Backup the original config
sudo cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.backup

# Create our RKE2-optimized configuration
sudo tee /etc/haproxy/haproxy.cfg > /dev/null << 'EOF'
global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin
        stats timeout 30s
        user haproxy
        group haproxy
        daemon
        
        # SSL configuration for future HTTPS endpoints
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private
        
        # Modern SSL configuration - only secure protocols
        ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
        ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
        ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

#---------------------------------------------------------------------
# RKE2 Supervisor/Registration Frontend
# This is where new nodes join the cluster - the secret handshake!
#---------------------------------------------------------------------
frontend rke2_registration_frontend
    bind *:9345
    mode tcp
    option tcplog
    default_backend rke2_registration_backend

#---------------------------------------------------------------------
# RKE2 Supervisor/Registration Backend
# Round-robin between masters for node registration
#---------------------------------------------------------------------
backend rke2_registration_backend
    mode tcp
    balance roundrobin
    option tcp-check
    # Health check ensures we only send traffic to healthy masters
    server rke2-master-1 rke2-master-1:9345 check
    server rke2-master-2 rke2-master-2:9345 check
    server rke2-master-3 rke2-master-3:9345 check

#---------------------------------------------------------------------
# Kubernetes API Frontend
# This is where kubectl commands and apps connect
#---------------------------------------------------------------------
frontend k8s_api_frontend
    bind *:6443
    mode tcp
    option tcplog
    default_backend k8s_api_backend

#---------------------------------------------------------------------
# Kubernetes API Backend
# Distributes API requests across all masters
#---------------------------------------------------------------------
backend k8s_api_backend
    mode tcp
    balance roundrobin
    option tcp-check
    # TCP health checks on the API port
    server rke2-master-1 rke2-master-1:6443 check
    server rke2-master-2 rke2-master-2:6443 check
    server rke2-master-3 rke2-master-3:6443 check

#---------------------------------------------------------------------
# Statistics Page (Optional but useful for monitoring)
#---------------------------------------------------------------------
listen stats
    bind *:8080
    stats enable
    stats uri /stats
    stats refresh 30s
    stats show-node
    stats auth admin:strongpassword  # Change this password!
EOF

# Enable and start HAProxy
sudo systemctl enable haproxy
sudo systemctl restart haproxy

# Verify it's running and listening
sudo systemctl status haproxy
sudo netstat -tlnp | grep -E '6443|9345|8080'

Understanding the HAProxy configuration:

• mode tcp: We're using Layer 4 (TCP) load balancing, not Layer 7 (HTTP). This means HAProxy doesn't decrypt or inspect the traffic → it just forwards TCP connections. This is crucial for Kubernetes API traffic which uses mutual TLS authentication.

• balance roundrobin: Distributes connections evenly across all healthy backends. Simple and effective.

• option tcp-check: HAProxy actively checks if backend servers are responding on the specified port. If a master goes down, HAProxy automatically stops sending traffic to it.

• Port 9345: RKE2's supervisor port → where new nodes register to join the cluster.

• Port 6443: Kubernetes API server → where kubectl and applications connect.

Testing HAProxy Health Checks

# Check HAProxy statistics (browse to http://haproxy-ip:8080/stats)
curl -u admin:strongpassword http://localhost:8080/stats

# Test connectivity through HAProxy (should return 401 Unauthorized - that's good!)
curl -k https://localhost:6443

# Check HAProxy logs for backend health
sudo tail -f /var/log/haproxy.log

Configuring the Main Load Balancer

The main load balancer sits in front of your HAProxy servers and needs to forward TCP traffic on ports 6443 and 9345 to both HAProxy servers. This is pure Layer 4 (TCP) load balancing, so no SSL termination or HTTP inspection needed.

Your main load balancer should:

• Forward port 6443 → HAProxy servers port 6443 (Kubernetes API)

• Forward port 9345 → HAProxy servers port 9345 (RKE2 Supervisor)

• Use TCP mode with health checks

• Balance traffic between both HAProxy servers

The specific configuration depends on your environment (cloud provider LB, F5, another HAProxy, etc.), but the concept remains the same: simple TCP forwarding with health checks.

Part 3: Birth of a Cluster - Installing the First Master

The Genesis Node: Your First Master

With our load balancers ready, let's create the cluster! The first master initializes everything.

On rke2-master-1:

# Download and install RKE2
curl -sfL https://get.rke2.io | sudo sh -

# Enable but don't start yet
sudo systemctl enable rke2-server.service

# Create the configuration
sudo mkdir -p /etc/rancher/rke2

# Master-1 configuration - note the tls-san entries include our LB
sudo tee /etc/rancher/rke2/config.yaml > /dev/null << EOF
# RKE2 Server Configuration - First Master Node
write-kubeconfig-mode: "0644"

# CRITICAL: Add all possible API server access points to the certificate
tls-san:
  - "rke2-lb"              # Main load balancer DNS
  - "192.168.1.100"        # Main load balancer IP
  - "rke2-haproxy-1"       # HAProxy-1 DNS
  - "rke2-haproxy-2"       # HAProxy-2 DNS
  - "192.168.1.30"         # HAProxy-1 IP
  - "192.168.1.31"         # HAProxy-2 IP
  - "rke2-master-1"        # Master hostnames
  - "rke2-master-2"
  - "rke2-master-3"
  - "192.168.1.10"         # Master IPs
  - "192.168.1.11"
  - "192.168.1.12"

# Network configuration
cluster-cidr: "10.42.64.0/18"
service-cidr: "10.42.0.0/18"
cluster-dns: "10.42.0.10"

# Security hardening
profile: "cis-1.6"
selinux: true
secrets-encryption: true

# Node configuration
node-label:
  - "node-role.kubernetes.io/master=true"
node-taint:
  - "node-role.kubernetes.io/master=true:NoSchedule"
EOF

# Start the first master
sudo systemctl start rke2-server.service

# Watch it come to life
sudo journalctl -u rke2-server -f

# Once running, grab the join token
sudo cat /var/lib/rancher/rke2/server/node-token

Why These CIDR Blocks? 

# Network configuration
cluster-cidr: "10.42.64.0/18"
service-cidr: "10.42.0.0/18"
cluster-dns: "10.42.0.10"

This network design is deliberately crafted for production scalability and troubleshooting ease:

• cluster-cidr: "10.42.64.0/18" - Provides 16,384 IP addresses for pods (10.42.64.0 - 10.42.127.255). This /18 subnet can handle massive deployments while staying within a predictable range.

• service-cidr: "10.42.0.0/18" - Allocates another 16,384 IPs for services (10.42.0.0 - 10.42.63.255). Notice how it's perfectly adjacent to but not overlapping with the pod network, this makes network debugging much cleaner.

• Non-overlapping design: The service and pod networks are contiguous within the 10.42.0.0/16 space but completely separate. Services get the lower half (0-63), pods get the upper half (64-127). This means you can immediately tell if an IP belongs to a pod or service just by looking at the third octet!

• Avoiding conflicts: The 10.42.0.0/16 range is less commonly used than 10.244.0.0 (Flannel default) or 192.168.0.0 (home networks), reducing the chance of conflicts with existing infrastructure or VPN connections.

• cluster-dns: "10.42.0.10" Placed safely within the service range, leaving room for system services that might claim lower IPs.

This setup gives you room for 16,000+ pods and services each, enough headroom for even the most ambitious deployments while maintaining a logical, easy-to-troubleshoot network architecture.

Setting Up kubectl Access

# Configure kubectl
echo 'export PATH=/var/lib/rancher/rke2/bin:$PATH' >> ~/.bashrc
echo 'export KUBECONFIG=/etc/rancher/rke2/rke2.yaml' >> ~/.bashrc
source ~/.bashrc

# Verify cluster is running
kubectl get nodes

Growing the Control Plane: Additional Masters

On rke2-master-2 and rke2-master-3:

# Install RKE2
curl -sfL https://get.rke2.io | sudo sh -

# Create configuration
sudo mkdir -p /etc/rancher/rke2

# Additional master configuration - connects through the first master initially
sudo tee /etc/rancher/rke2/config.yaml > /dev/null << EOF
# RKE2 Server Configuration - Additional Master
server: https://rke2-master-1:9345  # Direct connection for initial join
token: "YOUR_TOKEN_HERE"  # Token from master-1
write-kubeconfig-mode: "0644"

# Same TLS SANs as master-1 - consistency is crucial!
tls-san:
  - "rke2-lb"
  - "192.168.1.100"
  - "rke2-haproxy-1"
  - "rke2-haproxy-2"
  - "192.168.1.30"
  - "192.168.1.31"
  - "rke2-master-1"
  - "rke2-master-2"
  - "rke2-master-3"
  - "192.168.1.10"
  - "192.168.1.11"
  - "192.168.1.12"

# Identical network configuration
cluster-cidr: "10.42.64.0/18"
service-cidr: "10.42.0.0/18"
cluster-dns: "10.42.0.10"

# Same backup and security settings
profile: "cis-1.6"
selinux: true
secrets-encryption: true

# Node configuration
node-label:
  - "node-role.kubernetes.io/master=true"
node-taint:
  - "node-role.kubernetes.io/master=true:NoSchedule"
EOF

# Join the cluster
sudo systemctl enable rke2-server.service
sudo systemctl start rke2-server.service

# Verify it joined
sudo systemctl status rke2-server.service

Part 4: Adding Muscle - Worker Node Installation

Now for the nodes that run your actual workloads. Notice that, they are connecting through the load balancer!

On each worker node:

# Install RKE2 agent
curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE="agent" sudo sh -

# Create configuration
sudo mkdir -p /etc/rancher/rke2

# Worker configuration - connects through the load balancer!
sudo tee /etc/rancher/rke2/config.yaml > /dev/null << EOF
# RKE2 Agent Configuration
server: https://rke2-lb:9345  # Using the main load balancer!
token: "YOUR_TOKEN_HERE"  # Same token from master-1

# Node labels for workload scheduling
node-label:
  - "node-role.kubernetes.io/worker=true"
  - "workload-type=general"

# Optional: Reserve resources for system stability
# kubelet-arg:
#   - "system-reserved=cpu=500m,memory=1Gi"
#   - "kube-reserved=cpu=500m,memory=1Gi"
EOF

# Start the worker
sudo systemctl enable rke2-agent.service
sudo systemctl start rke2-agent.service

# Check status
sudo systemctl status rke2-agent.service

Part 5: Verifying RKE2 High Availability - Chaos Testing Your Setup

Cluster Health Check

# From any master, check all nodes
kubectl get nodes -o wide

# Verify etcd cluster has 3 healthy members
sudo /var/lib/rancher/rke2/bin/etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt \
  --cert=/var/lib/rancher/rke2/server/tls/etcd/server-client.crt \
  --key=/var/lib/rancher/rke2/server/tls/etcd/server-client.key \
  member list

# Check HAProxy is balancing correctly
# From your workstation, run this multiple times
curl -k https://rke2-lb:6443/version
# You should see different server IPs in the response headers

The Ultimate HA Test: Chaos Engineering

Let's prove our HA setup actually works:

# Deploy a test application
kubectl create deployment nginx-ha-test --image=nginx:latest --replicas=6
kubectl expose deployment nginx-ha-test --type=NodePort --port=80

# Test 1: Stop one HAProxy server
# On rke2-haproxy-1:
sudo systemctl stop haproxy

# From your workstation - kubectl should still work!
kubectl get pods

# Test 2: Stop one master node (while HAProxy-1 is still down!)
# On rke2-master-2:
sudo systemctl stop rke2-server

# Your cluster should still be fully functional
kubectl get nodes  # master-2 will show NotReady
kubectl create deployment test --image=busybox -- sleep 3600

# Bring everything back up
# On rke2-haproxy-1:
sudo systemctl start haproxy
# On rke2-master-2:
sudo systemctl start rke2-server

Remote Access Through the Load Balancer

# From your local
scp root@rke2-master-1:/etc/rancher/rke2/rke2.yaml ~/.kube/config-rke2-ha

# Update the server address to use the load balancer
sed -i 's/127.0.0.1/rke2-lb/g' ~/.kube/config-rke2-ha
# OR use the IP
sed -i 's/127.0.0.1/192.168.1.100/g' ~/.kube/config-rke2-ha

# Use the HA configuration
export KUBECONFIG=~/.kube/config-rke2-ha
kubectl get nodes

Conclusion

Gratz! Now you've built not just a Kubernetes cluster, but a fortress of high availability. With your multi-tier load balancing architecture, you can now:

• Lose any single HAProxy server without impact

• Lose any single master node without losing cluster access

• Perform rolling updates on any component without downtime

• Scale your cluster by simply adding nodes through the load balancer

The beauty of this setup is its resilience. Your main load balancer distributes traffic to dual HAProxy servers, which intelligently route to healthy master nodes. Even if multiple components fail simultaneously, your cluster keeps running. The load balancers handle the complexity of health checking and failover, while RKE2 handles the Kubernetes orchestration.

Remember, the HAProxy statistics page is your window into the cluster's health. Monitor it regularly, set up alerts, and test your HA setup periodically. Chaos engineering isn't just fun, it's how you build confidence in your infrastructure.

What's next for your bulletproof cluster? Consider adding observability with Prometheus and Grafana to monitor all these components, implement Cluster Autoscaler for dynamic scaling, or set up Velero for disaster recovery. How will you push your HA setup even further?