# Process

1. Install rasperry pi os, and run rpi-eeprom-update, to make sure it's the latest.

Turned out to be unnecessary, but good habit anyhow.

1. Install Ubuntu 20.04 LTS 64-bit

Login, change password, assign static DHCP lease
Setup CNAME record for k8s to point to k8s1

1. Configure access

```
ssh ubuntu@k8s1
sudo useradd -m -s /bin/bash -G sudo fdamstra
sudo passwd fdamstra
exit
ssh-copy-id k8s1
ssh k8s1
sudo cp /etc/sudoers.d/90-cloud-init-users /etc/sudoers.d/91-nopasswd
sudo vim /etc/sudoers.d/91-nopasswd
# change 'ubuntu' to 'fdamstra'
```

1. Basics

```
git clone https://fdamstra@io.monkeybox.org/git/fdamstra/MyEnvironment
cd MyEnvironment
./CreateLinks.sh
cd ..
source .bashrc
sudo apt install thefuck vim net-tools nfs-common dbus avahi-daemon
sudo apt update
sudo apt upgrade
```

1. K8S stuff

```
vim /boot/firmware/cmdline.txt
# Append:  cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1
cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF
cat <<EOF | sudo tee /etc/modules-load.d/netfilter.conf
overlay
br_netfilter
EOF
sudo reboot
```

1. CRI-O

docker is deprecated, we're going to start here...  may be a mistake, but who knows.

Prep the repos
```
export OS=xUbuntu_20.04
# find the latest k8s version at https://storage.googleapis.com/kubernetes-release/release/stable.txt
export VERSION=1.20
cat <<EOF | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/ /
EOF
cat <<EOF | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.list
deb http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/$VERSION/$OS/ /
EOF
curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/Release.key | sudo apt-key --keyring /etc/apt/trusted.gpg.d/libcontainers.gpg add -
# NOTE: This url is wrong on teh page, opened a github issue
curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/$VERSION/$OS/Release.key | sudo apt-key --keyring /etc/apt/trusted.gpg.d/libcontainers-cri-o.gpg add -
```

Install it
```
sudo apt-get update
sudo apt-get install cri-o cri-o-runc
sudo apt-mark hold cri-o # prevent auto updates
sudo systemctl daemon-reload
sudo systemctl enable crio --now
sudo vim /etc/hostname
# set the short hostname
```

1. Kubernetes Install

```
sudo apt-get update && sudo apt-get install -y apt-transport-https curl ca-certificates
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
sudo apt-get update # I had to run this multiple times
sudo apt-get install -y kubelet kubeadm kubectl
# note: "The kubelet is now restarting every few seconds, as it waits in a crashloop for kubeadm to tell it what to do"
```

1. Bootstrapping the master (first node only!) *MASTER NODE ONLY*

* NOTE: The 10.244.0.0/16 is required for flannel, our networking layer
```
sudo kubeadm init --pod-network-cidr 10.244.0.0/16 --control-plane-endpoint k8s.home.monkeybox.org
# Note, to do it a second time, run:
sudo kubeadm reset --kubeconfig /etc/kubernetes/admin.conf
```


Recording OUTPUT here:
```
[init] Using Kubernetes version: v1.21.0
[preflight] Running pre-flight checks
        [WARNING SystemVerification]: missing optional cgroups: hugetlb
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s.home.monkeybox.org k8s1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.42.42.201]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s1 localhost] and IPs [10.42.42.201 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s1 localhost] and IPs [10.42.42.201 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[apiclient] All control plane components are healthy after 106.511915 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.21" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s1 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node k8s1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: ajr5er.7nj7x0idsy8ekbw0
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:

  kubeadm join k8s.home.monkeybox.org:6443 --token ajr5er.7nj7x0idsy8ekbw0 \
        --discovery-token-ca-cert-hash sha256:2a4fa0e06c3f32743e05c15fa03ee3702638d57940ef6774960af7496fb2d0fb \
        --control-plane

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join k8s.home.monkeybox.org:6443 --token ajr5er.7nj7x0idsy8ekbw0 \
        --discovery-token-ca-cert-hash sha256:2a4fa0e06c3f32743e05c15fa03ee3702638d57940ef6774960af7496fb2d0fb
```

Run the commands on the master it tells you to:
```
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
```

Apply a good network
```
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
```

Allow the master to run stuff:
```
kubectl taint nodes --all node-role.kubernetes.io/master-
```

Install helm (k8s package manager):
```
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
chmod 755 get_helm.sh
./get_helm.sh # Must self-sudo
```

1. Have slaves join the cluster * ON THE REMAINING NODES *

It's tempting here to make node #2 another control node, but we're going to do without for now.

Run the _second_ 'kubeadm join' commadn from above (the first one adds control nodes!)

1. Install the dashboard

Back on the master:
```
helm repo add kubernetes-dashboard https://kubernetes.github.io/dashboard/
helm install kubernetes-dashboard kubernetes-dashboard/kubernetes-dashboard --set protocolHttp=true,ingress.enabled=true,rbac.create=true,serviceAccount.create=true,service.externalPort=9090,networkPolicy.enabled=true,podLabels.app=dashboard
# CONCERN: It says the server URLs are, but doesn't list anything
kubectl create clusterrolebinding kubernetes-dashboard --clusterrole=cluster-admin --serviceaccount=default:kubernetes-dashboard
```

1. Install MetalLB Load balancer

From https://metallb.universe.tf/installation/

```
kubectl edit configmap -n kube-system kube-proxy
# Chagne "mode" from "" to "ipvs"
# Change "strictARP" from "false" to "true"
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.6/manifests/namespace.yaml
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.6/manifests/metallb.yaml
# On first install only
kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"
```

On all nodes, set the interface to promiscuous:
```
sudo ifconfig eth0 promisc
cat <<EOF | sudo tee /etc/networkd-dispatcher/routable.d/50-ifup-promisc
#!/bin/sh
ip link set eth0 promisc on
EOF
sudo chmod 755 /etc/networkd-dispatcher/routable.d/50-ifup-promisc
```

Reboot the nodes and verify via `ifconfig eth0`

See where the dashboard is via:
```
kubectl get svc k8s-dashboard
```

1. Configure the shared filesystems

```
sudo mkdir /mnt/{shared,kubernetes}
sudo chown nobody:nogroup /mnt/{shared,kubernetes}
```

add the following to /etc/fstab:
```
10.42.42.10:/mnt/DroboFS/Shares/Public     /mnt/shared     nfs auto,nofail,noatime,nolock,intr,tcp,actimeo=1800 0 0
10.42.42.10:/mnt/DroboFS/Shares/Kubernetes /mnt/kubernetes nfs auto,nofail,noatime,nolock,intr,tcp,actimeo=1800 0 0
```

```
sudo mount -a
```

1. Miscellany

Install traefik:
```
helm repo add traefik https://helm.traefik.io/traefik
helm repo update
helm install traefik traefik/traefik
```

Make avahi work?
```
sudo vim /etc/avahi/avahi-daemon.conf
# uncomment domain-name=local
sudo systemctl restart avahi-daemon
```

k9s?
```
wget https://github.com/derailed/k9s/releases/download/v0.24.7/k9s_Linux_arm64.tar.gz
tar xvzf k9s_Linux_arm64.tar.gz
sudo mv k9s /usr/local/bin/
sudo chmod 755 /usr/local/bin/k9s
```

Assign labels to the nodes:
```
# I assigned 'node:node1' and 'zone:zoneA' etc to the nodes, but didn't write down how...
```

Login into docker
follow directions at https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/

Install kompose
```
wget https://github.com/kubernetes/kompose/releases/download/v1.22.0/kompose-linux-arm64
chmod +x kompose-linux-arm64
sudo mv kompose-linux-arm64 /usr/local/bin/kompose
```