Raspberry Pi Kubernetes Cluster

Notes from setting up a three node Raspberry Pi 3 Model B Kubernetes cluster using HypriotOS 64-bit.

Originally from Setup Kubernetes on a Raspberry Pi Cluster easily the official way! with some additions to fix things I ran into when following the guide. This guide uses a 64-bit version of HypriotOS and only armv8 64-bit Docker images will work.

RPI Cluster

Installing and Configuring HypriotOS

Flash HypriotOS v1.6.0 64-bit to SD card. By using HypriotOS we can avoid a lot of the issues that comes with installing Docker on ARM.

To begin, boot the Raspberry Pi to Hypriot, login and update system,

sudo apt update
sudo apt upgrade -y

Installing Kubernetes

Install Kubernetes from official package repositories on each node,

sudo su -
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" > /etc/apt/sources.list.d/kubernetes.list
apt update
apt install -y kubeadm kubelet

Setup the Master Node

As root, init the cluster with the network CIDR for Flannel,

As of this writing this will install and configure Kubernetes 1.8

kubeadm init --pod-network-cidr 10.244.0.0/16

As the pirate user setup kube config to run kubectl commands as non-root,

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
echo "export KUBECONFIG=${HOME}/.kube/config" >> ~/.bashrc
source ~/.bashrc

Setup Flannel CNI

By default Kubernetes does not configure a Container Network Interface and needs to have one installed. Flannel has an arm64 version available and works reasonably well on the Raspberry Pi 3 and HypriotOS.

Install flannel using arm64 images,

curl -sSL https://raw.githubusercontent.com/coreos/flannel/v0.9.0/Documentation/kube-flannel.yml | sed "s/amd64/arm64/g" | kubectl create -f -

Setup Worker Nodes

On each worker node run the kubeadm join command that was output after successfully running kubeadm init on the master node.

Join the node to the cluster,

sudo kubeadm join --token=$TOKEN

Setup Iptables Rules

Add some additional iptables rules in order for external DNS and forwarding in containers to work properly. See this issue for more information. Run these commands on the master and all worker nodes.

Install iptables-persistent package to save iptables rules,

sudo apt install -y iptables-persistent

Setup iptables for flannel CNI,

sudo iptables -P FORWARD ACCEPT
sudo iptables -t nat -A POSTROUTING -s 10.244.0.0/16 ! -d 10.244.0.0/16 -j MASQUERADE
sudo iptables -I FORWARD 1 -i cni0 -j ACCEPT -m comment --comment "flannel subnet"
sudo iptables -I FORWARD 1 -o cni0 -j ACCEPT -m comment --comment "flannel subnet"

Save iptable rules so they persist after reboot,

sudo netfilter-persistent save

Verifying

Show Node Status,

$ kubectl get nodes -o wide
NAME      STATUS    ROLES     AGE       VERSION   EXTERNAL-IP   OS-IMAGE                       KERNEL-VERSION    CONTAINER-RUNTIME
navi      Ready     master    15m       v1.8.0    <none>        Debian GNU/Linux 9 (stretch)   4.9.13-bee42-v8   docker://Unknown
tael      Ready     <none>    9m        v1.8.0    <none>        Debian GNU/Linux 9 (stretch)   4.9.13-bee42-v8   docker://Unknown
tatl      Ready     <none>    8m        v1.8.0    <none>        Debian GNU/Linux 9 (stretch)   4.9.13-bee42-v8   docker://Unknown

Show Pod Status,

$ kubectl get pods --all-namespaces
NAMESPACE     NAME                           READY     STATUS    RESTARTS   AGE
kube-system   etcd-navi                      1/1       Running   0          15m
kube-system   kube-apiserver-navi            1/1       Running   0          15m
kube-system   kube-controller-manager-navi   1/1       Running   1          15m
kube-system   kube-dns-596cf7c484-qrqsx      3/3       Running   0          14m
kube-system   kube-flannel-ds-2rzg7          1/1       Running   0          8m
kube-system   kube-flannel-ds-852gj          1/1       Running   0          13m
kube-system   kube-flannel-ds-qxmws          1/1       Running   0          10m
kube-system   kube-proxy-92762               1/1       Running   0          10m
kube-system   kube-proxy-r78jd               1/1       Running   0          14m
kube-system   kube-proxy-tfdjr               1/1       Running   0          8m
kube-system   kube-scheduler-navi            1/1       Running   0          15

Run A Test Pod

Using Docker Hub Official Multi-Platform Images makes running official Docker hub images on arm64 hardware much easier. Since Docker Hub now understands architecture manifest, no specific architecture tags are required and any official images will work on a Raspberry Pi k8s cluster without a specific tag.

Official images based off Alpine Linux currently do not work - see Issue #304 for more information.

In this example, we’ll run the official nginx image and have it listen on port 80.

First, deploy a nginx service with 3 replicas,

$ kubectl run nginx --image=nginx --replicas=3 --port=80
deployment "nginx" created

Expose pods in nginx service on port 80,

$ kubectl expose deployment nginx --port 80
service "nginx" exposed

Get endpoints for nginx service,

$ kubectl get endpoints
NAME         ENDPOINTS                                   AGE
kubernetes   192.168.7.220:6443                          37m
nginx        10.244.1.2:80,10.244.1.3:80,10.244.2.2:80   23s

Run curl against an endpoint IP to test,

curl should work against all endpoint IPs on all nodes, if not double check the iptables rules above

$ curl 10.244.2.2 | head -n 5
<DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
Written on September 11, 2017