|
@@ -179,11 +179,11 @@ Some silenced events will not unsilence and will need to be manually unsilenced.
|
|
|
salt -L 'vault-3.msoc.defpoint.local,sensu.msoc.defpoint.local' test.ping
|
|
|
date; salt -L 'vault-3.msoc.defpoint.local,sensu.msoc.defpoint.local' system.reboot
|
|
|
watch "salt -L 'vault-3.msoc.defpoint.local,sensu.msoc.defpoint.local' test.ping"
|
|
|
-salt -C '* not ( moose-splunk-indexer* or afs* or nga* or ma-* or mo-* or la-* or dc-* or vault-3* or sensu* or interconnect* or resolver* )' test.ping --out=txt
|
|
|
-date; salt -C '* not ( moose-splunk-indexer* or afs* or nga* or ma-* or mo-* or la-* or dc-* or vault-3* or sensu* or interconnect* or resolver* )' system.reboot
|
|
|
+salt -C '* not ( moose-splunk-indexer* or afs* or nga* or ma-* or mo-* or la-* or dc-* or vault-3* or sensu* or interconnect* or resolver* or nihor* )' test.ping --out=txt
|
|
|
+date; salt -C '* not ( moose-splunk-indexer* or afs* or nga* or ma-* or mo-* or la-* or dc-* or vault-3* or sensu* or interconnect* or resolver* or nihor* )' system.reboot
|
|
|
### You will lose connectivity to openvpn and salt master
|
|
|
### Log back in and verify they are back up
|
|
|
-watch "salt -C '* not ( moose-splunk-indexer* or afs* or nga* or ma-* or mo-* or la-* or dc-* or vault-3* or sensu* or interconnect* or resolver* )' cmd.run 'uptime' --out=txt"
|
|
|
+watch "salt -C '* not ( moose-splunk-indexer* or afs* or nga* or ma-* or mo-* or la-* or dc-* or vault-3* or sensu* or interconnect* or resolver* or nihor* )' cmd.run 'uptime' --out=txt"
|
|
|
|
|
|
# Take care of the interconencts/resolvers one at a time.
|
|
|
|
|
@@ -219,6 +219,8 @@ salt -C '* not ( afs* or saf* or nga* or ma-* or mo-* or dc-c19* or la-c19* or o
|
|
|
|
|
|
### Day 2 (Thursday), Step 2 of 4: Reboot Moose
|
|
|
|
|
|
+Log in to https://moose-splunk-cm.msoc.defpoint.local:8000/ and go to `settings->indexer clustering`.
|
|
|
+
|
|
|
```
|
|
|
salt -C 'moose-splunk-indexer*' test.ping --out=txt
|
|
|
|
|
@@ -228,13 +230,11 @@ date; salt -C 'moose-splunk-indexer-i-03ff4fb9915d5f7df.msoc.defpoint.local' sys
|
|
|
|
|
|
# Indexers take a while to restart
|
|
|
watch "salt -C 'moose-splunk-indexer-i-03ff4fb9915d5f7df.msoc.defpoint.local' cmd.run 'uptime' --out=txt"
|
|
|
-ping moose-splunk-indexer-1.msoc.defpoint.local
|
|
|
+ping moose-splunk-indexer-i-03ff4fb9915d5f7df.msoc.defpoint.local
|
|
|
```
|
|
|
|
|
|
#### WAIT FOR SPLUNK CLUSTER TO HAVE 3 CHECKMARKS
|
|
|
|
|
|
-Log in to https://moose-splunk-cm.msoc.defpoint.local:8000/ and go to `settings->indexer clustering`.
|
|
|
-
|
|
|
Repeat the above patching steps for the additional indexers, waiting for 3 green checks in between each one.
|
|
|
|
|
|
```
|
|
@@ -256,7 +256,7 @@ watch "salt -C 'moose-splunk-indexer-i-00ca1da87a2abcd56.msoc.defpoint.local' cm
|
|
|
salt 'moose-splunk-indexer*' cmd.run 'uptime' --out=txt
|
|
|
```
|
|
|
|
|
|
-#### If the indexer/checkmarks don't come back
|
|
|
+#### If the indexer/checkmarks don't come back ( legacy information )
|
|
|
|
|
|
If an indexer is not coming back up...look at screenshot in AWS... see this: `Probing EDD (edd=off to disable)... ok` then look at system log in AWS see this: `Please enter passphrase for disk splunkhot!`:
|
|
|
|