Strona główna Odkrywaj Pomoc
Zarejestruj się Zaloguj się
Containers
/
monkeybox_kubernetes
2
0
Forkuj 0
Pliki Problemy 0 Oczekujące zmiany 0 Wiki
Drzewo: d3a80dc4c0
Gałęzie Tagi
monkeybox_kuber...
/
original_notes.md

original_notes.md 12 KB
Historia Czysty

Process

  1. Install rasperry pi os, and run rpi-eeprom-update, to make sure it's the latest.

Turned out to be unnecessary, but good habit anyhow.

  1. Install Ubuntu 20.04 LTS 64-bit

Login, change password, assign static DHCP lease Setup CNAME record for k8s to point to k8s1

  1. Configure access

    ssh ubuntu@k8s1
sudo useradd -m -s /bin/bash -G sudo fdamstra
sudo passwd fdamstra
exit
ssh-copy-id k8s1
ssh k8s1
sudo cp /etc/sudoers.d/90-cloud-init-users /etc/sudoers.d/91-nopasswd
sudo vim /etc/sudoers.d/91-nopasswd
# change 'ubuntu' to 'fdamstra'

  2. Basics

    git clone https://fdamstra@io.monkeybox.org/git/fdamstra/MyEnvironment
cd MyEnvironment
./CreateLinks.sh
cd ..
source .bashrc
sudo apt install thefuck vim net-tools nfs-common dbus avahi-daemon
sudo apt update
sudo apt upgrade

  3. K8S stuff

    vim /boot/firmware/cmdline.txt
# Append:  cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1
cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF
cat <<EOF | sudo tee /etc/modules-load.d/netfilter.conf
overlay
br_netfilter
EOF
sudo reboot

  4. CRI-O

docker is deprecated, we're going to start here... may be a mistake, but who knows.

Prep the repos

export OS=xUbuntu_20.04
# find the latest k8s version at https://storage.googleapis.com/kubernetes-release/release/stable.txt
export VERSION=1.20
cat <<EOF | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/ /
EOF
cat <<EOF | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.list
deb http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/$VERSION/$OS/ /
EOF
curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/Release.key | sudo apt-key --keyring /etc/apt/trusted.gpg.d/libcontainers.gpg add -
# NOTE: This url is wrong on teh page, opened a github issue
curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/$VERSION/$OS/Release.key | sudo apt-key --keyring /etc/apt/trusted.gpg.d/libcontainers-cri-o.gpg add -

Install it

sudo apt-get update
sudo apt-get install cri-o cri-o-runc
sudo apt-mark hold cri-o # prevent auto updates
sudo systemctl daemon-reload
sudo systemctl enable crio --now
sudo vim /etc/hostname
# set the short hostname

  1. Kubernetes Install

    sudo apt-get update && sudo apt-get install -y apt-transport-https curl ca-certificates
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
sudo apt-get update # I had to run this multiple times
sudo apt-get install -y kubelet kubeadm kubectl
# note: "The kubelet is now restarting every few seconds, as it waits in a crashloop for kubeadm to tell it what to do"

  2. Bootstrapping the master (first node only!) MASTER NODE ONLY

  • NOTE: The 10.244.0.0/16 is required for flannel, our networking layer

    sudo kubeadm init --pod-network-cidr 10.244.0.0/16 --control-plane-endpoint k8s.home.monkeybox.org
# Note, to do it a second time, run:
sudo kubeadm reset --kubeconfig /etc/kubernetes/admin.conf

Recording OUTPUT here:

[init] Using Kubernetes version: v1.21.0
[preflight] Running pre-flight checks
        [WARNING SystemVerification]: missing optional cgroups: hugetlb
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s.home.monkeybox.org k8s1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.42.42.201]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s1 localhost] and IPs [10.42.42.201 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s1 localhost] and IPs [10.42.42.201 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[apiclient] All control plane components are healthy after 106.511915 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.21" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s1 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node k8s1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: ajr5er.7nj7x0idsy8ekbw0
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:

  kubeadm join k8s.home.monkeybox.org:6443 --token ajr5er.7nj7x0idsy8ekbw0 \
        --discovery-token-ca-cert-hash sha256:2a4fa0e06c3f32743e05c15fa03ee3702638d57940ef6774960af7496fb2d0fb \
        --control-plane

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join k8s.home.monkeybox.org:6443 --token ajr5er.7nj7x0idsy8ekbw0 \
        --discovery-token-ca-cert-hash sha256:2a4fa0e06c3f32743e05c15fa03ee3702638d57940ef6774960af7496fb2d0fb

Run the commands on the master it tells you to:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Apply a good network

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

Allow the master to run stuff:

kubectl taint nodes --all node-role.kubernetes.io/master-

Install helm (k8s package manager):

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
chmod 755 get_helm.sh
./get_helm.sh # Must self-sudo
  1. Have slaves join the cluster * ON THE REMAINING NODES *

It's tempting here to make node #2 another control node, but we're going to do without for now.

Run the second 'kubeadm join' commadn from above (the first one adds control nodes!)

  1. Install the dashboard

Back on the master:

helm repo add kubernetes-dashboard https://kubernetes.github.io/dashboard/
helm install kubernetes-dashboard kubernetes-dashboard/kubernetes-dashboard --set protocolHttp=true,ingress.enabled=true,rbac.create=true,serviceAccount.create=true,service.externalPort=9090,networkPolicy.enabled=true,podLabels.app=dashboard
# CONCERN: It says the server URLs are, but doesn't list anything
kubectl create clusterrolebinding kubernetes-dashboard --clusterrole=cluster-admin --serviceaccount=default:kubernetes-dashboard
  1. Install MetalLB Load balancer

From https://metallb.universe.tf/installation/

kubectl edit configmap -n kube-system kube-proxy
# Chagne "mode" from "" to "ipvs"
# Change "strictARP" from "false" to "true"
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.6/manifests/namespace.yaml
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.6/manifests/metallb.yaml
# On first install only
kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"

On all nodes, set the interface to promiscuous:

sudo ifconfig eth0 promisc
cat <<EOF | sudo tee /etc/networkd-dispatcher/routable.d/50-ifup-promisc
#!/bin/sh
ip link set eth0 promisc on
EOF
sudo chmod 755 /etc/networkd-dispatcher/routable.d/50-ifup-promisc

Reboot the nodes and verify via ifconfig eth0

See where the dashboard is via:

kubectl get svc k8s-dashboard

  1. Configure the shared filesystems

    sudo mkdir /mnt/{shared,kubernetes}
sudo chown nobody:nogroup /mnt/{shared,kubernetes}

add the following to /etc/fstab:

10.42.42.10:/mnt/DroboFS/Shares/Public     /mnt/shared     nfs auto,nofail,noatime,nolock,intr,tcp,actimeo=1800 0 0
10.42.42.10:/mnt/DroboFS/Shares/Kubernetes /mnt/kubernetes nfs auto,nofail,noatime,nolock,intr,tcp,actimeo=1800 0 0

sudo mount -a
  1. Miscellany

Install traefik:

helm repo add traefik https://helm.traefik.io/traefik
helm repo update
helm install traefik traefik/traefik

Make avahi work?

sudo vim /etc/avahi/avahi-daemon.conf
# uncomment domain-name=local
sudo systemctl restart avahi-daemon

k9s?

wget https://github.com/derailed/k9s/releases/download/v0.24.7/k9s_Linux_arm64.tar.gz
tar xvzf k9s_Linux_arm64.tar.gz
sudo mv k9s /usr/local/bin/
sudo chmod 755 /usr/local/bin/k9s

Assign labels to the nodes:

# I assigned 'node:node1' and 'zone:zoneA' etc to the nodes, but didn't write down how...

Login into docker follow directions at https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/

Install kompose

wget https://github.com/kubernetes/kompose/releases/download/v1.22.0/kompose-linux-arm64
chmod +x kompose-linux-arm64
sudo mv kompose-linux-arm64 /usr/local/bin/kompose