microk8s ftw?

Ansible/README.md

@@ -180,7 +180,29 @@ ansible-playbook site.yml
 You just kinda work throught he
 You just kinda work through them.
 
-kubeadm join k8s.home.monkeybox.org:6443 --token x01udi.u7clxsm3004sne9p \
+sudo kubeadm join 10.42.42.201:6443 --token x01udi.u7clxsm3004sne9p \
 	--discovery-token-ca-cert-hash sha256:093b597fbd39eca8fdc0298339f1403792e4b06d50fabc9fdc226606e40197b9 \
 	--control-plane --certificate-key 0f98113e1146805b2783e696f2df3f6760a12a223a393517036cc94b452406b1 \
-  ---apiserver-advertise-address IPHERE
+  --apiserver-advertise-address IPHERE
+
+etcd manifest original values:
+```
+    livenessProbe:
+      initialDelaySeconds: 30
+      periodSeconds: 30
+      timeoutSeconds: 60
+    startupProbe:
+      initialDelaySeconds: 30
+      periodSeconds: 30
+      timeoutSeconds: 60
+```
+
+apiserver manifest original values:
+```
+      initialDelaySeconds: 30
+      periodSeconds: 30
+      timeoutSeconds: 60
+    name: kube-apiserver
+    readinessProbe:
+      periodSeconds: 3
+      timeoutSeconds: 45

Ansible/group_vars/microk8s/vars

@@ -0,0 +1,7 @@
+---
+# Adds a failsafe to prevent bootstrapping a provisioned cluster
+k8s_newinstall: True
+skip_git: True # not there when we start k8s
+
+# The following DNS should resolve to one or more of the control nodes
+k8s_control: k8s.home.monkeybox.org

Ansible/group_vars/microk8s/vault

@@ -0,0 +1,10 @@
+$ANSIBLE_VAULT;1.1;AES256
+64396331303163626261316434366266336366666239363934653330326138336634633865646236
+6362643335376332313233613935356433343833626236650a343763323365383261316266366462
+32303139626134356262336234366638343833666139643436356631663631343433616663646438
+3635383065663362640a306131376365373232373731316661643134646430336131616534653634
+61303534323939643034313834316464333636623533393066623964626533363737336666373939
+31353238303832333966636233633031303366306439616662643234343031366565373534626335
+34376165313062663035326230346466613434366131623961306430633038653636653034656338
+39656165366434616163316539333562666264313562633633646333336335643435323230666131
+34336533626331623466353030653230646334396636343465343262636138353164

Ansible/inventory.ini

@@ -4,13 +4,15 @@ k8s1 ansible_host=k8s1.home.monkeybox.org board=pi4
 k8s2 ansible_host=k8s2.home.monkeybox.org board=pi4
 k8s3 ansible_host=k8s3.home.monkeybox.org board=pi4
 
+[microk8s]
+k8s1
+k8s2
+k8s3
+
 [k8s:children]
 k8scontrol
 k8snode
 
 [k8scontrol]
-k8s1
-k8s2
-k8s3
 
 [k8snode]

Ansible/microk8s.yml

@@ -0,0 +1,7 @@
+---
+- hosts: microk8s
+  become: yes
+  become_user: root
+  roles:
+    - common
+    - microk8s

Ansible/roles/microk8s/files/config

@@ -0,0 +1,99 @@
+$ANSIBLE_VAULT;1.1;AES256
+65643263333935316366666334653865396530656230653438643034613935316365323235333363
+3639656239383339313439653166393738383839363334620a373530383162346630636133343231
+66623166393761303237646333653532323562346662653036636134313261323439386232613635
+3639356666393630390a306337346437386163646432636237303638396436336265653333363263
+66653337333661313830616630646263333638343532313535363065373362666632343936366430
+63393262663763643735353965663261306664663963326665303666613737613432633937363234
+35363466663234626638623163623439626463393362346330643133396165653837613962346135
+36373437656362373165336662663238393831323231616230626164383164663834303430303936
+33336333316164373331303432363738613765376134396235633836383262323235356430336334
+64366561386164386365623365303966373536323530316566373638363334356562396337633836
+63646661386663323930363631633235376664346465346135396562636462633032383433393236
+62313031633731373766386262636436636630373964336630393663383035393061666334623938
+63356364383061303930366539313663636362386466323963623663636438616663663034316234
+38613630346465646265663164613537333335393864313637356564613730373564376534306164
+38633139656464633766336537336164633538386163616438333439343061653936323833646163
+62626538363665336661336362303865346364646131643535383266303634316261333564356164
+35316539346236656263383539343636383538323233323263383963313036316264383162666662
+65363534643630616435623963626362306333643361313439353838366162643431653137633536
+62353631356135646662666131653131393961653433313538363830656235306232356465653935
+36356361613266636462323366386438353138313239613731346534643831343732636531396236
+36626464616262623436393131386437303733633236626437353335656463653864353338373864
+39633830623363396437373762313233383036356335656634663632656439363930383634333837
+62366633366237643636376132633139653563623963303761356635643737653736653966376433
+32633132356263376338653963373438643033343064376363666632633337636630366134393166
+30646265373136626232613864383937623732616433393931643134356266623530393238393531
+37643536336362613436653436643366653663306635336561316130646237623965656563613434
+65643732333738623165396339356239613139646566623738363734323732306236363531303633
+62323139653463633332306633373063623033643764623539313235633437303664343765303563
+31393461306531633634303934326332623932626236616436333465613539646134653936383232
+30613661343732626530666438353931383838366537383664643431306265353139646465343335
+61333839613865323038623335663831316239356366356431363164333938366533346563333764
+36393137323938336662316262353633363138653963363436613930336330363637396165656238
+33306139306237363230336535326464393136393363333537303762303639343537373437383236
+32366435333635386430356537316636613265666430303837393461326233643465333737313165
+36313636623935623330636339633431636438346163643533613639626635333665363464366661
+61363263303263623963643762373837386135656631386261346366613039333662373232646262
+30303564353837666263356239326439386537303933663837646164616561663065383836323962
+66386335626131383131303937303761613862613339653933653764353336393766623265386535
+62343931353864343832626630353436663462366165353035393238333933376432336563656630
+62313831323662383433626136386166663135333738353736636233366664343531303032353861
+64383865663063636666363863386236346534626631353765323234323062643336626536363961
+33333661303135386161373033653966313534623064343864366565646235373434613930393237
+62353332316533353334316436663434366430326239633436646566636635356534306165646639
+61363564656562336133636331333133393738363637316564393165643861636633643338323064
+33343731396534636339633663666538636466376433613561643636653638643363353639336561
+64373163393464316333383962343365366337613761383464366236393531386566323333646632
+66643166366136356463663936356663613763376265396266333738343031373435333431386162
+64326638326533333134393533643134366530663433303638666435393236336665326432623534
+65613363303034376661336238386134623736303432353930316536616631613331376534633235
+36613037326536656335656234653730363231613663363932643662386564613465346239366265
+35396534363139616233643662386631326433643233373232323362613732343939316531363164
+35386563373765396362303339666262353439373635393933353936386232396135613132323330
+36316462363031303766663863386466663839663230303935653039656130656539343562636562
+63383230326462393364353835323735613634666432663565633039656165626334393234623139
+35646266366461306463396663353331616364306234663634663234313662646362653230373363
+37653732393434393162656363303236353263376138313637376362383937373539376461393235
+64303039373830313562396435326438646464373834376532616231336634616235353764353931
+36616339643662613661643030343135373532353862636334306136346165336663613565306463
+30373734393737386536303637366163616464376639326566633865363062626331333764643962
+30633865316434356639303266383134333131376661623735316531613461646232383335626231
+38353533356432316333363631343330653566653238343963663563333737373264393334323933
+32316230643262663165366135653435663433633036313162396134333131386634636631666238
+36636137363865303933343431333537323463643734326361346437646163616237313965336462
+64323066323765383334373963326134663439313062376233613335626364653532646162666334
+37343835313736626661626237656235636634306266623036666261363762376133306366653166
+35633532623436656634373534343335653863323430373634373966373661363936663233666239
+30393536623835386465336638383836303562323066633030346564343362383635336630646661
+32363739636534396335343562343966636562323166333235666162353439666531636562366333
+33386463666538343862623863333330376330366166613831616233366166366236323932393965
+38613531386433333961653936333566626465303933393231636165376636633935616336653637
+37393964303464373434343238323739326636373334363838656532363536323632363866346434
+37663965333332626238363136343536626338383236396466346163313332333166323063613031
+32363839336332613133373065363636393932346537636237633330653435643034613564333434
+34636533623666663462613535323934656230383431396536396231663738656433666463616437
+30353662323038323234656336346432343465643762333936373566633437303461636562613133
+33666565656633656363353663643039323963636337316633353534383035626630393734636232
+38316262663132366265373832336130343863383130303837656234623030366561373063393766
+64633262653163326138303765306136616163653638333138303933363864396562366563363130
+66646165343130623632383163326635393732653062393864396563386166626361303163393939
+37663962623365363537336431636361373431353637303265653666396462616439343334666339
+63326438613863313061343565313134636237313736303339343663623439656662353637353738
+32653361323366393038633737646366623461353930386632656332653831613731316663316537
+63326365396430303661366362646633336135656361346165666664633535313566373162326661
+61383764303265303533353263383964353466353064623363613430333064623430633162383337
+37306234313961363661383838373939646438613564646535386361633565353932353362656264
+36656361613836363336386630663466373761396462353131343563623231323761646531323630
+35316666663265383565393630366461346334346334613963383438613031656431333366623166
+33363465346638363137316162356438653639663964303562663032313631653736356233313531
+66643136323039626234666335363930356331666365626264333738636636356431333431626431
+31396532356665386564643535323838336662643463393162386337616461643435646339656663
+35396134666135373138656563643466383561396335633363396234333435323837613033326466
+34373531376536653664666630303930326362336130646535623065303932356430653531623063
+65623137356364373239346235386564656639643335393664363837343937323633393062353831
+33366235643134346464313861333664663938373335353731636231653566363063383032363266
+37353435323131633538663634366131353162623961303366383534393562376264633632303038
+63626332646430663639643664383534623836623661643130353831376635393436666437313564
+35633734353134376634623138616233303236346434626537326633656333313230633134306136
+3036326239363231386437646262326661666435623462613734

Ansible/roles/microk8s/handlers/main.yml

@@ -0,0 +1,11 @@
+---
+- name: Reboot Server
+  throttle: 1
+  reboot:
+    msg: "Reboot Handler"
+    connect_timeout: 5
+    reboot_timeout: 300
+    pre_reboot_delay: 0
+    post_reboot_delay: 30
+    test_command: uptime
+  listen: "Reboot Server"

Ansible/roles/microk8s/tasks/bootstrap.yml

@@ -0,0 +1,36 @@
+---
+- name: Pull Bootstrap Images
+  shell: "kubeadm config images pull"
+  when: bootstrap_needed
+
+- name: Mention Bootstrapping
+  pause:
+    prompt: |
+      At this point, if you are bootstrapping, on ONE node only, run:
+        sudo kubeadm init --pod-network-cidr 10.244.0.0/16 --control-plane-endpoint k8s.home.monkeybox.org --upload-certs --apiserver-advertise-address 10.42.42.201
+      And if you need to do it again / reset it:
+         sudo kubeadm reset --kubeconfig /etc/kubernetes/admin.conf
+
+      Don't forget to copy /etc/kubernetes/admin.conf into this repository:
+         ansible-vault edit ~/monkeybox_kubernetes/Ansible/roles/k8s/files/config
+
+      AND A LOT MORE! Read /monkeybox_kubernetes/Ansible/roles/k8s/tasks/bootstrap.yml
+  when: bootstrap_needed
+
+# Here's the "a lot more":
+#
+# ```
+# sudo kubeadm init --pod-network-cidr 10.244.0.0/16 --control-plane-endpoint k8s.home.monkeybox.org --upload-certs
+# ```
+# 
+# Apply a good network
+# ```
+# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
+# ```
+#
+# Record output, especially
+# token:
+# discovery_token:
+# and record them in group_vars/k8s/vars
+# 
+# Also, copy /etc/kubernetes/config to ~/.kube/config

Ansible/roles/microk8s/tasks/cmdline.yml

@@ -0,0 +1,55 @@
+---
+## IPv6
+#- name: Check IPv6 Disabled
+#  shell: cat /boot/firmware/cmdline.txt | grep -c 'ipv6.disable' || true
+#  register: pi_ipv6_disabled
+#  changed_when: false # Do not mark as 'changed'
+#
+#- name: Add IPv6 Disabled
+#  replace: dest=/boot/firmware/cmdline.txt regexp='(\s*)$' replace=' ipv6.disable=1'
+#  when: pi_ipv6_disabled.stdout == "0"
+#  notify: "Reboot Server"
+
+# Quirks
+- name: Check SSD Quirk
+  shell: cat /boot/firmware/cmdline.txt | grep -c 'usb-storage.quirks=152d:1561:u' || true
+  register: pi_ssd_quirk
+  changed_when: false # Do not mark as 'changed'
+
+- name: Add SSD Quirk
+  replace: dest=/boot/firmware/cmdline.txt regexp='(\s*)$' replace=' usb-storage.quirks=152d:1561:u'
+  when: pi_ssd_quirk.stdout == "0"
+  notify: "Reboot Server"
+
+# CPUSet
+- name: Check cpuset cgroup
+  shell: cat /boot/firmware/cmdline.txt | grep -c 'cgroup_enable=cpuset' || true
+  register: pi_cgroup_cpuset
+  changed_when: false # Do not mark as 'changed'
+
+- name: Add cpuset cgroup
+  replace: dest=/boot/firmware/cmdline.txt regexp='(\s*)$' replace=' cgroup_enable=cpuset'
+  when: pi_cgroup_cpuset.stdout == "0"
+  notify: "Reboot Server"
+
+# Memory
+- name: Check memory cgroup enabled
+  shell: cat /boot/firmware/cmdline.txt | grep -c 'cgroup_enable=memory' || true
+  register: pi_cgroup_memory_enable
+  changed_when: false # Do not mark as 'changed'
+
+- name: Add memory cgroup enabled
+  replace: dest=/boot/firmware/cmdline.txt regexp='(\s*)$' replace=' cgroup_enable=memory'
+  when: pi_cgroup_memory_enable.stdout == "0"
+  notify: "Reboot Server"
+
+# Memory Group
+- name: Check memory cgroup
+  shell: cat /boot/firmware/cmdline.txt | grep -c 'cgroup_memory' || true
+  register: pi_cgroup_memory
+  changed_when: false # Do not mark as 'changed'
+
+- name: Add memory cgroup
+  replace: dest=/boot/firmware/cmdline.txt regexp='(\s*)$' replace=' cgroup_memory=1'
+  when: pi_cgroup_memory.stdout == "0"
+  notify: "Reboot Server"

Ansible/roles/microk8s/tasks/cri-o.yml

@@ -0,0 +1,41 @@
+---
+- name: libcontainers key
+  ansible.builtin.apt_key:
+    url: "https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/{{ crio_os }}/Release.key"
+    state: present
+
+- name: devel:kubic:libcontainers:stable.list
+  ansible.builtin.apt_repository:
+    repo: "deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/{{ crio_os }}/ /"
+    state: present
+    filename: devel:kubic:libcontainers:stable
+
+
+- name: cri-o key
+  ansible.builtin.apt_key:
+    url: "https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/{{ crio_version }}/{{ crio_os }}/Release.key"
+    state: present
+
+- name: "devel:kubic:libcontainers:stable:cri-o:{{ crio_version }}.list"
+  ansible.builtin.apt_repository:
+    repo: "deb http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/{{ crio_version }}/{{ crio_os }}/ /"
+    state: present
+    filename: "devel:kubic:libcontainers:stable:cri-o:{{ crio_version }}"
+
+
+- name: Install cri-o
+  apt:
+    pkg:
+    - cri-o
+    - cri-o-runc
+
+- name: Hold cri-o to prevent updates
+  dpkg_selections:
+    name: cri-o
+    selection: hold
+
+- name: Enable cri-o service
+  ansible.builtin.service:
+    name: crio
+    enabled: yes
+    state: started

Ansible/roles/microk8s/tasks/k9s.yml

@@ -0,0 +1,36 @@
+---
+#- name: Untaint the control nodes
+#  become_user: fdamstra
+#  shell: "kubectl taint nodes --all node-role.kubernetes.io/master-"
+#  ignore_errors: true # Errors out if already tainted
+
+- name: Download k9s
+  get_url:
+    url: https://github.com/derailed/k9s/releases/download/v0.24.7/k9s_Linux_arm64.tar.gz
+    dest: /usr/local/src/k9s_Linux_arm64.tar.gz
+  register: new_k9s
+  tags: wip
+
+- name: Extract k9s if Changed
+  ansible.builtin.unarchive:
+    src: /usr/local/src/k9s_Linux_arm64.tar.gz
+    dest: /usr/local/bin
+    creates: /usr/local/bin/k9s
+    exclude:
+      - LICENSE
+      - README
+    copy: no
+  when: new_k9s.changed
+  tags: wi
+
+# This is kind of a duplicate. We want to extract it if it doesn't exist, too.
+- name: Extract k9s if it doesnt exist
+  ansible.builtin.unarchive:
+    src: /usr/local/src/k9s_Linux_arm64.tar.gz
+    dest: /usr/local/bin
+    creates: /usr/local/bin/k9s
+    exclude:
+      - LICENSE
+      - README
+    copy: no
+  tags: wip

Ansible/roles/microk8s/tasks/main.yml

@@ -0,0 +1,56 @@
+---
+- name: Set the Hostname
+  ansible.builtin.hostname:
+    name: "{{ inventory_hostname }}"
+
+- name: Fix cmdline.txt
+  import_tasks: cmdline.yml
+
+- name: system.yml
+  import_tasks: system.yml
+
+- name: k9s.yml
+  import_tasks: k9s.yml
+
+- name: Flush handlers to Reboot if Required
+  meta: flush_handlers
+
+#- name: CRI-O Installation
+#  import_tasks: cri-o.yml
+#
+#- name: K8S Installation
+#  import_tasks: k8s_install.yml
+#
+#- name: Install Client Config
+#  import_tasks: k8s_kubeadm_client.yml
+
+## At this point, k8s is there, but we need to determine if a cluster has been created. If it has, then
+## the nodes will all have .kube/config under fdamstra, and be able to talk to the master, and ask if
+## there is a node marked 'k8s1', which is the node we bootstrap. If there is, then we skip all the 
+## bootstrapping. If not, we:
+##   a) Run the bootstrap on that node.
+##   b) Do not run anything else whatsoever.
+#- name: Check for k8s1
+#  become_user: fdamstra
+#  shell: kubectl get nodes | grep -c 'k8s1' || true
+#  register: k8s1_status
+#  changed_when: false # Do not mark as 'changed'
+#  tags: wip
+#
+#- name: Determine Whether Bootstrap is Needed/Desired
+#  set_fact:
+#    bootstrap_needed: "{{ True if ( k8s_newinstall==True and k8s1_status.stdout == '0' ) else False }}"
+#  tags: wip
+#
+#- name: Bootstrapping
+#  import_tasks: bootstrap.yml
+#  when: bootstrap_needed == True and inventory_hostname == "k8s1"
+#
+#- name: Join the cluster
+#  import_tasks: k8s_join.yml
+#  when: inventory_hostname != "k8s1" and not bootstrap_needed
+#
+#- name: Post joining tasks
+#  import_tasks: k8s_postjoin.yml
+#  when: not bootstrap_needed
+

Ansible/roles/microk8s/tasks/system.yml

@@ -0,0 +1,64 @@
+---
+- name: /etc/sysctl.d/99-kubernetes-cri.conf
+  copy:
+    dest: /etc/sysctl.d/99-kubernetes-cri.conf
+    content: |
+      net.bridge.bridge-nf-call-iptables  = 1
+      net.ipv4.ip_forward                 = 1
+      net.bridge.bridge-nf-call-ip6tables = 1
+    mode: 0644
+    owner: root
+    group: root
+  notify: "Reboot Server"
+
+- name: /etc/modules-load.d/netfilter.conf
+  copy:
+    dest: /etc/modules-load.d/netfilter.conf
+    content: |
+      overlay
+      br_netfilter
+    mode: 0644
+    owner: root
+    group: root
+  notify: "Reboot Server"
+
+- name: /etc/networkd-dispatcher/routable.d/50-ifup-promisc
+  copy:
+    dest: /etc/networkd-dispatcher/routable.d/50-ifup-promisc
+    content: |
+      #!/bin/sh
+      ip link set eth0 promisc on
+    mode: 0755
+    owner: root
+    group: root
+  notify: "Reboot Server"
+
+- name: Shared mount point
+  file:
+    path: /mnt/shared
+    state: directory
+    owner: nobody
+    group: nogroup
+
+- name: Public Share
+  ansible.posix.mount:
+    src: 10.42.42.10:/mnt/DroboFS/Shares/Public
+    path: /mnt/shared
+    opts: _netdev,auto,nofail,noatime,nolock,intr,tcp,actimeo=1800
+    state: mounted
+    fstype: nfs
+
+- name: Shared mount point
+  file:
+    path: /mnt/kubernetes
+    state: directory
+    owner: nobody
+    group: nogroup
+  
+- name: Kubernetes Share
+  ansible.posix.mount:
+    src: 10.42.42.10:/mnt/DroboFS/Shares/Kubernetes
+    path: /mnt/kubernetes
+    opts: _netdev,auto,nofail,noatime,nolock,intr,tcp,actimeo=1800
+    state: mounted
+    fstype: nfs

Ansible/site.yml

@@ -1,3 +1,4 @@
 ---
 - import_playbook: k8s.yml
+- import_playbook: microk8s.yml
 - import_playbook: io.yml

microk8s_notes.md

@@ -0,0 +1,130 @@
+udo Installation
+
+```
+sudo snap install microk8s --classic --channel=latest/stable
+sudo usermod -a -G microk8s fdamstra
+sudo chown -f -R fdamstra ~/.kube
+
+# Wait for ready
+microk8s status --wait-ready
+
+# Definitely want coredns:
+microk8s enable dns storage
+
+# may want:
+alias kubectl='microk8s kubectl'
+```
+
+# Adding nodes
+
+on the master:
+```
+microk8s add-node
+```
+
+on the remote, use the join command provided
+
+# Checking status
+
+run `microk8s status` and view "high-availability: yes"
+
+
+
+# To remove :
+
+Gracefully:
+```
+microk8s leave
+```
+
+Then from a remaining node:
+```
+microk8s remove-node <node>
+```
+
+Non-gracefully:
+```
+microk8s remove-node <node> --force
+```
+
+# Next Steps
+
+## MetalLB
+
+Using the status seems better
+
+```
+microk8s enable metallb
+# Enter IP range: 10.42.42.211-10.42.42.240
+```
+
+## NFS Provisioning
+```
+cd ~/monkeybox_kubernetes/Workloads/nfs-provisioning
+microk8s kubectl apply -f 001*
+microk8s kubectl apply -f 002*
+microk8s kubectl apply -f 003*
+```
+
+## ingress-nginx
+
+Not clear to me if the addon was the right ingress controller. My stuff uses 'ingress-nginx', and I think the microk8s addon is 'nginx-ingress', and I think these are different things. So I'm just installing 'ingress-nginx' with my stuff:
+
+```
+cd ~/monkeybox_kubernetes/Workloads/ingress-nginx
+htpasswd -c auth fdamstra # create my user as a seccreet for HTTP Basic Auth
+microk8s kubectl create secret generic basic-auth --from-file=auth
+microk8s kubectl apply -f ingress-nginx-controller-v0.45.0.yaml
+```
+
+## Cert Manager
+1. log into the aws console
+2. iam->users->`letsencrypt-wildcard`->security credentials
+3. 'create access key'
+4. Copy the secret into a file called `password.txt` in `~/monkeybox_kubernetes/Workloads/cert-manager`
+5. Copy teh access key id into `~/monkeybox_kubernetes/Workloads/cert-manager/wildcard*`
+```
+cd ~/monkeybox_kubernetes/Workloads/cert-manager
+microk8s kubectl create secret generic aws-route53-creds --from-file=password.txt -n default
+microk8s kubectl apply -f cert-manager.yaml
+sleep 60 # big pause here
+microk8s kubectl apply -f staging_issuer.yaml
+microk8s kubectl apply -f prod_issuer.yaml
+microk8s kubectl apply -f wildcard_staging_issuer.yaml
+microk8s kubectl apply -f wildcard_prod_issuer.yaml
+```
+
+## Generate my first workloads
+
+```
+cd ~/monkeybox_kubernetes/Workloads
+vim index.html
+# change issuer to letsencrypt-staging
+microk8s kubectl apply -f index.yaml
+# Validate that `io.monkeybox.org` works
+```
+
+Shit. It doesn't.
+
+## Fix CoreDNS?
+
+microk8s kubectl edit configmap -n kube-system coredns
+
+
+
+# Notes for future me:
+
+# addons
+
+These might be better than helm and/or better than installing by hand, methinks.
+
+Full list: https://microk8s.io/docs/addons#heading--list
+
+Add addons
+```
+microk8s enable dns storage
+```
+
+# If you want to update later
+sudo snap refresh microk8s --channel=latest/stable
+