Format Changes

ClamAV notes.md

@@ -18,7 +18,7 @@ Logging is horrible.  Clamd by default writes to a logfile, but doesn't apparent
 log when a scan actually runs or what its results were, unless that scan finds
 something.
 
-See `salt/fileroots/internal_splunk_forwarder/files/TA-clamav/default/inputs.conf` for the locations Splunk is looking for. 
+See `salt/fileroots/internal_splunk_forwarder/files/TA-clamav/default/inputs.conf` for the locations Splunk is looking for.
 
 ## Exceptions and False Positives
 

Patching Notes--CaaSP.md

@@ -119,6 +119,7 @@ watch "salt -L 'caasp-splunk-sh-dev,caasp-splunk-hf,caasp-splunk-cm' status.upti
 
 # Reboot the search head
 date; salt caasp-splunk-sh system.reboot
+
 # Wait for it ...
 watch "salt caasp-splunk-sh status.uptime --out=txt"
 

Patching Notes.md

@@ -124,7 +124,7 @@ salt 'customer-portal*' cmd.run 'systemctl restart docker'
 Portal Notes are here for further Troubleshooting if necessary: [Portal Notes](Portal%20Notes.md)
 
 #### Patch CaaSP
-See [Patch CaaSP instructions](Patching%20Notes--CaaSP.md)    
+See [Patch CaaSP instructions](Patching%20Notes--CaaSP.md)
 
 #### Troubleshooting
 
@@ -213,7 +213,7 @@ tsh --proxy=teleport.xdrtest.accenturefederalcyber.com login
 tsh ssh node=salt-master
 ```
 
-Start with `Sensu` and `Vault` 
+Start with `Sensu` and `Vault`
 ```
 # Vault-3 and Sensu
 salt -C 'vault-3* or sensu*' test.ping --out=txt
@@ -315,7 +315,7 @@ watch "salt -C 'resolver-govcloud-2.pvt.*com' test.ping --out=txt"
 
 ```
 
-Check uptime on the minions in GC to make sure you didn't miss any. 
+Check uptime on the minions in `GC Prod` to make sure you didn't miss any. 
 ```
 salt -C  '*accenturefederalcyber.com not ( afs* or nga* or ma-* or dc-c19* or la-c19* or dgi-* or moose-splunk-idx* or modelclient-splunk-idx* or bp-ot-demo* or bas-* or doed* or frtib* or ca-c19* or resolver* or vault-1*com or sensu*com )' cmd.run 'uptime | grep days'
 ```
@@ -325,12 +325,51 @@ Verify Portal is up: [Portal](https://portal.xdr.accenturefederalcyber.com/)
 Look in Sensu for any silent alerts.
 
 #### Reboot CaaSP
-See Patching Notes--CaaSP.md
+See Day 2 notes in [Patch CaaSP instructions](Patching%20Notes--CaaSP.md)
 
 
 ### Day 2 (Thursday), Step 2 of 4: Reboot Moose
 
-Don't forget `GC TEST`! Start there first. 
+`GovCloud (TEST)`
+
+Log in to Moose [Moose Splunk CM](https://moose-splunk-cm.pvt.xdrtest.accenturefederalcyber.com:8000/) and go to `settings->indexer clustering`.
+
+```
+salt 'moose-splunk-idx*' test.ping --out=txt
+
+# Do the first indexers
+salt 'moose-splunk-idx-63f.pvt.xdrtest.accenturefederalcyber.com' test.ping --out=txt
+date; salt moose-splunk-idx-63f.pvt.xdrtest.accenturefederalcyber.com system.reboot
+
+# Indexers take a while to restart
+watch "salt moose-splunk-idx-63f.pvt.xdrtest.accenturefederalcyber.com cmd.run 'uptime' --out=txt"
+salt 'moose-splunk-idx-63f.pvt.xdrtest.accenturefederalcyber.com' test.ping --out=txt
+```
+
+#### WAIT FOR SPLUNK CLUSTER TO HAVE 3 CHECKMARKS
+
+Repeat the above patching steps for the additional indexers, waiting for `3 green checks` in between each one.
+
+```
+# Do the second indexer
+salt moose-splunk-idx-d4f.pvt.xdrtest.accenturefederalcyber.com test.ping --out=txt
+date; salt moose-splunk-idx-d4f.pvt.xdrtest.accenturefederalcyber.com system.reboot
+
+# Indexers take a while to restart
+watch "salt moose-splunk-idx-d4f.pvt.xdrtest.accenturefederalcyber.com cmd.run 'uptime' --out=txt"
+
+# Do the third indexer
+salt moose-splunk-idx-273.pvt.xdrtest.accenturefederalcyber.com test.ping --out=txt
+date; salt moose-splunk-idx-273.pvt.xdrtest.accenturefederalcyber.com system.reboot
+
+# Indexers take a while to restart
+watch "salt moose-splunk-idx-273.pvt.xdrtest.accenturefederalcyber.com cmd.run 'uptime' --out=txt"
+
+# Verify all indexers patched:
+salt 'moose-splunk-idx*' cmd.run 'uptime' --out=txt
+```
+
+`GovCloud (PROD)`
 
 Log in to Moose [Moose Splunk CM](https://moose-splunk-cm.pvt.xdr.accenturefederalcyber.com:8000/) and go to `settings->indexer clustering`.
 
@@ -382,7 +421,7 @@ IF/WHEN an `Indexer` doesn't come back up follow these steps:
 - Look for "Please enter passphrase for disk splunkhot"
 ```
 
-In AWS console stop instance (which will remove ephemeral splunk data) then start it. 
+In AWS console stop instance (which will remove ephemeral splunk data) then start it.
 Then ensure the `/opt/splunkdata/hot` exists.
 ```
 salt -C 'moose-splunk-idx-422.pvt.xdr.accenturefederalcyber.com' cmd.run 'df -h'
@@ -474,7 +513,7 @@ salt -C '* not *.local not *.pvt.xdr.accenturefederalcyber.com' pkg.upgrade
 
 Error on `afs-splunk-ds-3: error: cannot open Packages database in /var/lib/rpm`
 
-Solution: 
+Solution:
 
 ```
 mkdir /root/backups.rpm/

RedHat Notes.md

@@ -27,9 +27,9 @@ salt/pillar/prod/rhel_subs.sls
 
 ## AWS Subscriptions/Repositories
 
-Oh no the aws redhat repos broke! Try this https://access.redhat.com/solutions/5009491 
+Oh no the aws redhat repos broke! Try this [Get an AWS RHUI Client Package Supporting IMDSv2](https://access.redhat.com/solutions/5009491)
 
-download the redhat repo client package and copy it over via salt then install it. 
+download the redhat repo client package and copy it over via salt then install it.
 
 ```
 yumdownloader rh-amazon-rhui-client # on salt master