Basically part and parcel from https://docs.ansible.com/ansible/latest/user_guide/sample_setup.html

```
production                # inventory file for production servers
staging                   # inventory file for staging environment

group_vars/
   group1.yml             # here we assign variables to particular groups
   group2.yml
host_vars/
   hostname1.yml          # here we assign variables to particular systems
   hostname2.yml

library/                  # if any custom modules, put them here (optional)
module_utils/             # if any custom module_utils to support modules, put them here (optional)
filter_plugins/           # if any custom filter plugins, put them here (optional)

site.yml                  # master playbook
webservers.yml            # playbook for webserver tier
dbservers.yml             # playbook for dbserver tier
tasks/                    # task files included from playbooks
    webservers-extra.yml  # <-- avoids confusing playbook with task files

roles/
    common/               # this hierarchy represents a "role"
        tasks/            #
            main.yml      #  <-- tasks file can include smaller files if warranted
        handlers/         #
            main.yml      #  <-- handlers file
        templates/        #  <-- files for use with the template resource
            ntp.conf.j2   #  <------- templates end in .j2
        files/            #
            bar.txt       #  <-- files for use with the copy resource
            foo.sh        #  <-- script files for use with the script resource
        vars/             #
            main.yml      #  <-- variables associated with this role
        defaults/         #
            main.yml      #  <-- default lower priority variables for this role
        meta/             #
            main.yml      #  <-- role dependencies
        library/          # roles can also include custom modules
        module_utils/     # roles can also include custom module_utils
        lookup_plugins/   # or other types of plugins, like lookup in this case

    webtier/              # same kind of structure as "common" was above, done for the webtier role
    monitoring/           # ""
    fooapp/               # ""
```

## Bootstrapping

1. Check out this repository
1. Create a file `~/.ansible_vault` with the vault password
1. Use raspberry pi imager to install Ubuntu LTS (20.04 at time of this writing) 64-bit.
1. Find IP (via DHCP lease, reservation, or console?). 
1. Login with username ubuntu, password ubuntu. Set a new password.
1. Add hostname to inventory.
1. Run `ssh-copy-id ubuntu@hostname` and enter your password.
1. Install galaxy modules with `ansible-galaxy install -r requirements.yml`
1. Run the `initial_user.yml` task:
```
ansible-playbook -u ubuntu tasks/initial_users.yml --limit=k8s3
```

## Bootstrapping the Cluster

Start with the first control node:
```
ansible-playbook microk8s.yml --limit=k8s1 --extra-vars="skip_git=True" # Leave off skip_git if github is up
```

It will pause at the bootstrap step, but you can go ahead and hit enter.

TODO: Automate this

After the playbook completes and you've received the notification to bootstrap, do the following:

```
k8s1:   sudo kubeadm init --pod-network-cidr 10.244.0.0/16 --control-plane-endpoint k8s.home.monkeybox.org --upload-certs --apiserver-advertise-address 10.42.42.201
local:  ansible-vault edit ~/monkeybox_kubernetes/Ansible/bootstrap_output.txt
        # record output
local:  ansible-vault edit ~/monkeybox_kubernetes/Ansible/group_vars/k8s/vault # record values from output
k8s1:   cat /etc/kubernetes/admin.conf
local:  ansible-vault edit ~/monkeybox_kubernetes/Ansible/roles/k8s/files/config # record the config
k8s1:   mkdir -p $HOME/.kube
k8s1:   sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
k8s1:   sudo chown $(id -u):$(id -g) $HOME/.kube/config
k8s1:   kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
```

Rerun the playbook:
```
ansible-playbook microk8s.yml --limit=k8s1 --extra-vars="skip_git=True" # Leave off skip_git if github is up
```

From here, you may proceed to "other nodes" or "deploying workloads"

## MetallB

local:
```
scp -r ~/monkeybox_kubernetes k8s1:
```

On k8s1:
```
kubectl edit configmap -n kube-system kube-proxy
# search for `ipvs`, and set `strictARP` to true. See note [1]
cd monkeybox_kubernetes/Workloads/metallb/
kubectl apply -f 001*
kubectl apply -f 002*
kubectl apply -f 003*
kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"
kubectl get pods -n metallb-system
```

note [1]: They helpfully give the following shell code to help with automation.
```
kubectl get configmap kube-proxy -n kube-system -o yaml | \
sed -e "s/strictARP: false/strictARP: true/" | \
kubectl apply -f - -n kube-system
```

## nfs-provisioning

On k8s1:
```
cd ~/monkeybox_kubernetes/Workloads/nfs-provisioning
kubectl apply -f 001*
kubectl apply -f 002*
kubectl apply -f 003*
```

## ingress-nginx

```
cd ~/monkeybox_kubernetes/Workloads/ingress-nginx
htpasswd -c auth fdamstra
kubectl create secret generic basic-auth --from-file=auth
kubectl apply -f ingress-nginx-controller-v0.45.0.yaml
```

## cert-manager

1. log into the aws console
2. iam->users->`letsencrypt-wildcard`->security credentials
3. 'create access key'
4. Copy the secret into a file called `password.txt` in `~/monkeybox_kubernetes/Workloads/cert-manager`
5. Copy teh access key id into `~/monkeybox_kubernetes/Workloads/cert-manager/wildcard*` 
```
cd ~/monkeybox_kubernetes/Workloads/cert-manager
kubectl create secret generic aws-route53-creds --from-file=password.txt -n default
kubectl apply -f cert-manager.yaml
# big pause here
kubectl apply -f staging_issuer.yaml
kubectl apply -f prod_issuer.yaml
kubectl apply -f wildcard_staging_issuer.yaml
kubectl apply -f wildcard_prod_issuer.yaml
```


## Other Nodes

Adding other nodes should be straightforward:

1. Login with username ubuntu, password ubuntu. Set a new password.
1. Add hostname to inventory.
1. Run `ssh-copy-id ubuntu@hostname` and enter your password.
1. Run the `initial_user.yml` task
```
ansible-playbook -u ubuntu tasks/initial_users.yml --limit=k8s3
```
1. Run the full playbook (may as well run on everything)
```
ansible-playbook site.yml
```

## Other workloasd

You just kinda work throught he
You just kinda work through them.

sudo kubeadm join 10.42.42.201:6443 --token x01udi.u7clxsm3004sne9p \
	--discovery-token-ca-cert-hash sha256:093b597fbd39eca8fdc0298339f1403792e4b06d50fabc9fdc226606e40197b9 \
	--control-plane --certificate-key 0f98113e1146805b2783e696f2df3f6760a12a223a393517036cc94b452406b1 \
  --apiserver-advertise-address IPHERE

etcd manifest original values:
```
    livenessProbe:
      initialDelaySeconds: 30
      periodSeconds: 30
      timeoutSeconds: 60
    startupProbe:
      initialDelaySeconds: 30
      periodSeconds: 30
      timeoutSeconds: 60
```

apiserver manifest original values:
```
      initialDelaySeconds: 30
      periodSeconds: 30
      timeoutSeconds: 60
    name: kube-apiserver
    readinessProbe:
      periodSeconds: 3
      timeoutSeconds: 45