Added original install notes

original_notes.md

@@ -0,0 +1,318 @@
+# Process
+
+1. Install rasperry pi os, and run rpi-eeprom-update, to make sure it's the latest.
+
+Turned out to be unnecessary, but good habit anyhow.
+
+1. Install Ubuntu 20.04 LTS 64-bit
+
+Login, change password, assign static DHCP lease
+Setup CNAME record for k8s to point to k8s1
+
+1. Configure access
+
+```
+ssh ubuntu@k8s1
+sudo useradd -m -s /bin/bash -G sudo fdamstra
+sudo passwd fdamstra
+exit
+ssh-copy-id k8s1
+ssh k8s1
+sudo cp /etc/sudoers.d/90-cloud-init-users /etc/sudoers.d/91-nopasswd
+sudo vim /etc/sudoers.d/91-nopasswd
+# change 'ubuntu' to 'fdamstra'
+```
+
+1. Basics
+
+```
+git clone https://fdamstra@io.monkeybox.org/git/fdamstra/MyEnvironment
+cd MyEnvironment
+./CreateLinks.sh
+cd ..
+source .bashrc
+sudo apt install thefuck vim net-tools nfs-common dbus avahi-daemon
+sudo apt update
+sudo apt upgrade
+```
+
+1. K8S stuff
+
+```
+vim /boot/firmware/cmdline.txt
+# Append:  cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1 cgroup_enable=hugetlb
+cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
+net.bridge.bridge-nf-call-iptables  = 1
+net.ipv4.ip_forward                 = 1
+net.bridge.bridge-nf-call-ip6tables = 1
+EOF
+cat <<EOF | sudo tee /etc/modules-load.d/netfilter.conf
+overlay
+br_netfilter
+EOF
+sudo reboot
+```
+
+1. CRI-O
+
+docker is deprecated, we're going to start here...  may be a mistake, but who knows.
+
+Prep the repos
+```
+export OS=xUbuntu_20.04
+# find the latest k8s version at https://storage.googleapis.com/kubernetes-release/release/stable.txt
+export VERSION=1.20
+cat <<EOF | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
+deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/ /
+EOF
+cat <<EOF | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.list
+deb http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/$VERSION/$OS/ /
+EOF
+curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/Release.key | sudo apt-key --keyring /etc/apt/trusted.gpg.d/libcontainers.gpg add -
+# NOTE: This url is wrong on teh page, opened a github issue
+curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/$VERSION/$OS/Release.key | sudo apt-key --keyring /etc/apt/trusted.gpg.d/libcontainers-cri-o.gpg add -
+```
+
+Install it
+```
+sudo apt-get update
+sudo apt-get install cri-o cri-o-runc
+sudo apt-mark hold cri-o # prevent auto updates
+sudo systemctl daemon-reload
+sudo systemctl enable crio --now
+sudo vim /etc/hostname
+# set the short hostname
+```
+
+1. Kubernetes Install
+
+```
+sudo apt-get update && sudo apt-get install -y apt-transport-https curl ca-certificates
+curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
+sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
+sudo apt-get update # I had to run this multiple times
+sudo apt-get install -y kubelet kubeadm kubectl
+# note: "The kubelet is now restarting every few seconds, as it waits in a crashloop for kubeadm to tell it what to do"
+```
+
+1. Bootstrapping the master (first node only!) *MASTER NODE ONLY*
+
+* NOTE: The 10.244.0.0/16 is required for flannel, our networking layer
+```
+sudo kubeadm init --pod-network-cidr 10.244.0.0/16 --control-plane-endpoint k8s.home.monkeybox.org
+# Note, to do it a second time, run:
+sudo kubeadm reset --kubeconfig /etc/kubernetes/admin.conf
+```
+
+
+Recording OUTPUT here:
+```
+[init] Using Kubernetes version: v1.21.0
+[preflight] Running pre-flight checks
+        [WARNING SystemVerification]: missing optional cgroups: hugetlb
+[preflight] Pulling images required for setting up a Kubernetes cluster
+[preflight] This might take a minute or two, depending on the speed of your internet connection
+[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
+[certs] Using certificateDir folder "/etc/kubernetes/pki"
+[certs] Generating "ca" certificate and key
+[certs] Generating "apiserver" certificate and key
+[certs] apiserver serving cert is signed for DNS names [k8s.home.monkeybox.org k8s1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.42.42.201]
+[certs] Generating "apiserver-kubelet-client" certificate and key
+[certs] Generating "front-proxy-ca" certificate and key
+[certs] Generating "front-proxy-client" certificate and key
+[certs] Generating "etcd/ca" certificate and key
+[certs] Generating "etcd/server" certificate and key
+[certs] etcd/server serving cert is signed for DNS names [k8s1 localhost] and IPs [10.42.42.201 127.0.0.1 ::1]
+[certs] Generating "etcd/peer" certificate and key
+[certs] etcd/peer serving cert is signed for DNS names [k8s1 localhost] and IPs [10.42.42.201 127.0.0.1 ::1]
+[certs] Generating "etcd/healthcheck-client" certificate and key
+[certs] Generating "apiserver-etcd-client" certificate and key
+[certs] Generating "sa" key and public key
+[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
+[kubeconfig] Writing "admin.conf" kubeconfig file
+[kubeconfig] Writing "kubelet.conf" kubeconfig file
+[kubeconfig] Writing "controller-manager.conf" kubeconfig file
+[kubeconfig] Writing "scheduler.conf" kubeconfig file
+[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
+[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
+[kubelet-start] Starting the kubelet
+[control-plane] Using manifest folder "/etc/kubernetes/manifests"
+[control-plane] Creating static Pod manifest for "kube-apiserver"
+[control-plane] Creating static Pod manifest for "kube-controller-manager"
+[control-plane] Creating static Pod manifest for "kube-scheduler"
+[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
+[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
+[kubelet-check] Initial timeout of 40s passed.
+[apiclient] All control plane components are healthy after 106.511915 seconds
+[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
+[kubelet] Creating a ConfigMap "kubelet-config-1.21" in namespace kube-system with the configuration for the kubelets in the cluster
+[upload-certs] Skipping phase. Please see --upload-certs
+[mark-control-plane] Marking the node k8s1 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
+[mark-control-plane] Marking the node k8s1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
+[bootstrap-token] Using token: ajr5er.7nj7x0idsy8ekbw0
+[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
+[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
+[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
+[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
+[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
+[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
+[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
+[addons] Applied essential addon: CoreDNS
+[addons] Applied essential addon: kube-proxy
+
+Your Kubernetes control-plane has initialized successfully!
+
+To start using your cluster, you need to run the following as a regular user:
+
+  mkdir -p $HOME/.kube
+  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
+  sudo chown $(id -u):$(id -g) $HOME/.kube/config
+
+Alternatively, if you are the root user, you can run:
+
+  export KUBECONFIG=/etc/kubernetes/admin.conf
+
+You should now deploy a pod network to the cluster.
+Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
+  https://kubernetes.io/docs/concepts/cluster-administration/addons/
+
+You can now join any number of control-plane nodes by copying certificate authorities
+and service account keys on each node and then running the following as root:
+
+  kubeadm join k8s.home.monkeybox.org:6443 --token ajr5er.7nj7x0idsy8ekbw0 \
+        --discovery-token-ca-cert-hash sha256:2a4fa0e06c3f32743e05c15fa03ee3702638d57940ef6774960af7496fb2d0fb \
+        --control-plane
+
+Then you can join any number of worker nodes by running the following on each as root:
+
+kubeadm join k8s.home.monkeybox.org:6443 --token ajr5er.7nj7x0idsy8ekbw0 \
+        --discovery-token-ca-cert-hash sha256:2a4fa0e06c3f32743e05c15fa03ee3702638d57940ef6774960af7496fb2d0fb
+```
+
+Run the commands on the master it tells you to:
+```
+mkdir -p $HOME/.kube
+sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
+sudo chown $(id -u):$(id -g) $HOME/.kube/config
+```
+
+Apply a good network
+```
+kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
+```
+
+Allow the master to run stuff:
+```
+kubectl taint nodes --all node-role.kubernetes.io/master-
+```
+
+Install helm (k8s package manager):
+```
+curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
+chmod 755 get_helm.sh
+./get_helm.sh # Must self-sudo
+```
+
+1. Have slaves join the cluster * ON THE REMAINING NODES *
+
+It's tempting here to make node #2 another control node, but we're going to do without for now.
+
+Run the _second_ 'kubeadm join' commadn from above (the first one adds control nodes!)
+
+1. Install the dashboard
+
+Back on the master:
+```
+helm repo add kubernetes-dashboard https://kubernetes.github.io/dashboard/
+helm install kubernetes-dashboard kubernetes-dashboard/kubernetes-dashboard --set protocolHttp=true,ingress.enabled=true,rbac.create=true,serviceAccount.create=true,service.externalPort=9090,networkPolicy.enabled=true,podLabels.app=dashboard
+# CONCERN: It says the server URLs are, but doesn't list anything
+kubectl create clusterrolebinding kubernetes-dashboard --clusterrole=cluster-admin --serviceaccount=default:kubernetes-dashboard
+```
+
+1. Install MetalLB Load balancer
+
+From https://metallb.universe.tf/installation/
+
+```
+kubectl edit configmap -n kube-system kube-proxy
+# Chagne "mode" from "" to "ipvs"
+# Change "strictARP" from "false" to "true"
+kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.6/manifests/namespace.yaml
+kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.6/manifests/metallb.yaml
+# On first install only
+kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"
+```
+
+On all nodes, set the interface to promiscuous:
+```
+sudo ifconfig eth0 promisc
+cat <<EOF | sudo tee /etc/networkd-dispatcher/routable.d/50-ifup-promisc
+#!/bin/sh
+ip link set eth0 promisc on
+EOF
+sudo chmod 755 /etc/networkd-dispatcher/routable.d/50-ifup-promisc
+```
+
+Reboot the nodes and verify via `ifconfig eth0`
+
+See where the dashboard is via:
+```
+kubectl get svc k8s-dashboard
+```
+
+1. Configure the shared filesystems
+
+```
+sudo mkdir /mnt/{shared,kubernetes}
+sudo chown nobody:nogroup /mnt/{shared,kubernetes}
+```
+
+add the following to /etc/fstab:
+```
+10.42.42.10:/mnt/DroboFS/Shares/Public     /mnt/shared     nfs auto,nofail,noatime,nolock,intr,tcp,actimeo=1800 0 0
+10.42.42.10:/mnt/DroboFS/Shares/Kubernetes /mnt/kubernetes nfs auto,nofail,noatime,nolock,intr,tcp,actimeo=1800 0 0
+```
+
+```
+sudo mount -a
+```
+
+1. Miscellany
+
+Install traefik:
+```
+helm repo add traefik https://helm.traefik.io/traefik
+helm repo update
+helm install traefik traefik/traefik
+```
+
+Make avahi work?
+```
+sudo vim /etc/avahi/avahi-daemon.conf
+# uncomment domain-name=local
+sudo systemctl restart avahi-daemon
+```
+
+k9s?
+```
+wget https://github.com/derailed/k9s/releases/download/v0.24.7/k9s_Linux_arm64.tar.gz
+tar xvzf k9s_Linux_arm64.tar.gz
+sudo mv k9s /usr/local/bin/
+sudo chmod 755 /usr/local/bin/k9s
+```
+
+Assign labels to the nodes:
+```
+# I assigned 'node:node1' and 'zone:zoneA' etc to the nodes, but didn't write down how...
+```
+
+Login into docker
+follow directions at https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
+
+Install kompose
+```
+wget https://github.com/kubernetes/kompose/releases/download/v1.22.0/kompose-linux-arm64
+chmod +x kompose-linux-arm64
+sudo mv kompose-linux-arm64 /usr/local/bin/kompose
+```
+