|
@@ -99,28 +99,30 @@ FYI, patching today.
|
|
|
Starting with Moose and Internal infra patching within `GC TEST`. Check disk space for potential issues. Return here to start on PROD after TEST is patched.
|
|
|
```
|
|
|
# Test connectivity between Salt Master and Minions
|
|
|
-salt -C '* not ( afs* or nga* or dc-c19* or la-c19* or bas-* or doed* or ca-c19* or frtib* or dgi* or threatq* or vmray* or resolver-vmray* or qcompliance* or openvpn* )' test.ping --out=txt
|
|
|
+salt -C '* not ( afs* or nga* or dc-c19* or la-c19* or bas-* or ca-c19* or frtib* or dgi* or threatq* or vmray* or resolver-vmray* or qcompliance* or openvpn* )' test.ping --out=txt
|
|
|
|
|
|
# Fred's update for df -h - checks for disk utilization at the 80-90% area
|
|
|
-salt -C '* not ( afs* or nga* or dc-c19* or la-c19* or bas-* or doed* or ca-c19* or frtib* or dgi* or threatq* or vmray* or resolver-vmray* or qcompliance* or openvpn* )' cmd.run 'df -h | egrep "[890][0-9]\%"'
|
|
|
+salt -C '* not ( afs* or nga* or dc-c19* or la-c19* or bas-* or ca-c19* or frtib* or dgi* or threatq* or vmray* or resolver-vmray* or qcompliance* or openvpn* )' cmd.run 'df -h | egrep "[890][0-9]\%"'
|
|
|
|
|
|
# Review packages that will be updated. Some packages are versionlocked (Collectd, Splunk, Teleport, etc.).
|
|
|
-salt -C '* not ( afs* or nga* or dc-c19* or la-c19* or bas-* or doed* or ca-c19* or frtib* or dgi* or threatq* or vmray* or resolver-vmray* or qcompliance* or openvpn* )' cmd.run 'yum check-update'
|
|
|
+salt -C '* not ( afs* or nga* or dc-c19* or la-c19* or bas-* or ca-c19* or frtib* or dgi* or threatq* or vmray* or resolver-vmray* or qcompliance* or openvpn* )' cmd.run 'yum check-update'
|
|
|
```
|
|
|
|
|
|
<!-- ```
|
|
|
Older commands that are still viable if Fred's one-liner has issues; feel free to skip and move to pkg.upgrade line
|
|
|
-salt -C '* not ( afs* or nga* or dc-c19* or la-c19* or bp-ot-demo* or bas-* or doed* or ca-c19* or frtib* or dgi* or threatq* )' cmd.run 'df -h /boot'
|
|
|
-salt -C '* not ( afs* or nga* or dc-c19* or la-c19* or bp-ot-demo* or bas-* or doed* or ca-c19* or frtib* or dgi* or threatq* )' cmd.run 'df -h /var/log' # some at 63%
|
|
|
-salt -C '* not ( afs* or nga* or dc-c19* or la-c19* or bp-ot-demo* or bas-* or doed* or ca-c19* or frtib* or dgi* or threatq* )' cmd.run 'df -h /var' # one at 74%
|
|
|
-salt -C '* not ( afs* or nga* or dc-c19* or la-c19* or bp-ot-demo* or bas-* or doed* or ca-c19* or frtib* or dgi* or threatq* )' cmd.run 'df -h'
|
|
|
+salt -C '* not ( afs* or nga* or dc-c19* or la-c19* or bas-* or ca-c19* or frtib* or dgi* or threatq* )' cmd.run 'df -h /boot'
|
|
|
+salt -C '* not ( afs* or nga* or dc-c19* or la-c19* or bas-* or ca-c19* or frtib* or dgi* or threatq* )' cmd.run 'df -h /var/log' # some at 63%
|
|
|
+salt -C '* not ( afs* or nga* or dc-c19* or la-c19* or bas-* or ca-c19* or frtib* or dgi* or threatq* )' cmd.run 'df -h /var' # one at 74%
|
|
|
+salt -C '* not ( afs* or nga* or dc-c19* or la-c19* or bas-* or ca-c19* or frtib* or dgi* or threatq* )' cmd.run 'df -h'
|
|
|
|
|
|
``` -->
|
|
|
> :warning: **OpenVPN decommissioned on March 25, 2022; replaced with AWS VPN. Omit OpenVPN Commands.**
|
|
|
|
|
|
+See [AWS VPN Notes](AWS%20VPN%20NOTES.md)
|
|
|
+
|
|
|
### Also, the phantom_repo pkg wants to upgrade, but we are not ready. Let's exclude that.
|
|
|
```
|
|
|
-salt -C '* not ( afs* or nga* or dc-c19* or la-c19* or bas-* or doed* or ca-c19* or frtib* or dgi* or threatq* or openvpn* or vmray* or resolver-vmray* or phantom-0* or qcompliance* )' pkg.upgrade
|
|
|
+salt -C '* not ( afs* or nga* or dc-c19* or la-c19* or bas-* or ca-c19* or frtib* or dgi* or threatq* or openvpn* or vmray* or resolver-vmray* or phantom-0* or qcompliance* )' pkg.upgrade
|
|
|
|
|
|
# update phantom, but exclude the phantom repo.
|
|
|
salt -C 'phantom-0*' pkg.upgrade disablerepo='["phantom-base",]'
|
|
@@ -159,7 +161,7 @@ salt vmray* cmd.run 'systemctl start vmray-server vmray-worker'
|
|
|
|
|
|
### Run it again to make sure nothing got missed.
|
|
|
```
|
|
|
-salt -C '* not ( afs* or nga* or dc-c19* or la-c19* or bas-* or doed* or ca-c19* or frtib* or dgi* or threatq* or vmray* or resolver-vmray* or phantom-0* or qcompliance* )' pkg.upgrade
|
|
|
+salt -C '* not ( afs* or nga* or dc-c19* or la-c19* or bas-* or ca-c19* or frtib* or dgi* or threatq* or vmray* or resolver-vmray* or phantom-0* or qcompliance* )' pkg.upgrade
|
|
|
```
|
|
|
|
|
|
> :warning: After upgrades check on Portal to make sure it is still up.
|
|
@@ -356,16 +358,16 @@ watch "salt -C 'vault-1*com or sensu*com' test.ping --out=txt"
|
|
|
|
|
|
Reboot majority of servers in GC.
|
|
|
```
|
|
|
-salt -C '*com not ( afs* or nga* or dc-c19* or la-c19* or dgi-* or moose-splunk-idx* or modelclient-splunk-idx* or bas-* or doed* or frtib* or ca-c19* or resolver* or vault-1*com or sensu*com or qcompliance* or vmray-worker* or openvpn* )' test.ping --out=txt
|
|
|
+salt -C '*com not ( afs* or nga* or dc-c19* or la-c19* or dgi-* or moose-splunk-idx* or modelclient-splunk-idx* or bas-* or frtib* or ca-c19* or resolver* or vault-1*com or sensu*com or qcompliance* or vmray-worker* or openvpn* )' test.ping --out=txt
|
|
|
|
|
|
-date; salt -C '*com not ( afs* or nga* or dc-c19* or la-c19* or dgi-* or moose-splunk-idx* or modelclient-splunk-idx* or bas-* or doed* or frtib* or ca-c19* or resolver* or vault-1*com or sensu*com or qcompliance* or vmray-worker* or openvpn* )' system.reboot --async
|
|
|
+date; salt -C '*com not ( afs* or nga* or dc-c19* or la-c19* or dgi-* or moose-splunk-idx* or modelclient-splunk-idx* or bas-* or frtib* or ca-c19* or resolver* or vault-1*com or sensu*com or qcompliance* or vmray-worker* or openvpn* )' system.reboot --async
|
|
|
```
|
|
|
> :warning:
|
|
|
### You will lose connectivity to Salt master
|
|
|
### Log back in and verify they are back up
|
|
|
|
|
|
```
|
|
|
-watch "salt -C '*accenturefederalcyber.com not ( afs* or nga* or dc-c19* or la-c19* or dgi-* or moose-splunk-idx* or modelclient-splunk-idx* or bas-* or doed* or frtib* or ca-c19* or resolver* or vault-1*com or sensu*com or qcompliance* )' cmd.run 'uptime' --out=txt"
|
|
|
+watch "salt -C '*accenturefederalcyber.com not ( afs* or nga* or dc-c19* or la-c19* or dgi-* or moose-splunk-idx* or modelclient-splunk-idx* or bas-* or frtib* or ca-c19* or resolver* or vault-1*com or sensu*com or qcompliance* )' cmd.run 'uptime' --out=txt"
|
|
|
```
|
|
|
|
|
|
Take care of the resolvers one at a time and with the `GC Prod Salt Master`. Reboot one of each at the same time.
|
|
@@ -388,7 +390,7 @@ watch "salt -C 'vmray-worker*com' test.ping --out=txt"
|
|
|
|
|
|
Check uptime on the minions in `GC Prod` to make sure you didn't miss any.
|
|
|
```
|
|
|
-salt -C '*accenturefederalcyber.com not ( afs* or nga* or dc-c19* or la-c19* or dgi-* or moose-splunk-idx* or modelclient-splunk-idx* or bas-* or doed* or frtib* or ca-c19* or resolver* or vault-1*com or sensu*com or qcompliance* )' cmd.run 'uptime | grep days'
|
|
|
+salt -C '*accenturefederalcyber.com not ( afs* or nga* or dc-c19* or la-c19* or dgi-* or moose-splunk-idx* or modelclient-splunk-idx* or bas-* or frtib* or ca-c19* or resolver* or vault-1*com or sensu*com or qcompliance* )' cmd.run 'uptime | grep days'
|
|
|
```
|
|
|
|
|
|
Verify Portal is up: [Portal](https://portal.xdr.accenturefederalcyber.com/)
|
|
@@ -867,17 +869,17 @@ Today's patching is all XDR customer environments. Indexers and Searchheads will
|
|
|
Run these commands on `GC Prod Salt Master`. These notes should patch all Splunk instances.
|
|
|
|
|
|
```
|
|
|
-salt -C 'afs*local or afs*com or la-*com or nga*com or nga*local or dc*com or bas-*com or doed*com or frtib*com or ca-c19*com or dgi*com' test.ping --out=txt
|
|
|
+salt -C 'afs*local or afs*com or la-*com or nga*com or nga*local or dc*com or bas-*com or frtib*com or ca-c19*com or dgi*com' test.ping --out=txt
|
|
|
|
|
|
-salt -C 'afs*local or afs*com or la-*com or nga*com or nga*local or dc*com or bas-*com or doed*com or frtib*com or ca-c19*com or dgi*com' cmd.run 'uptime'
|
|
|
+salt -C 'afs*local or afs*com or la-*com or nga*com or nga*local or dc*com or bas-*com or frtib*com or ca-c19*com or dgi*com' cmd.run 'uptime'
|
|
|
|
|
|
# Fred's update for df -h:
|
|
|
-salt -C 'afs*local or afs*com or la-*com or nga*com or nga*local or dc*com or bas-*com or doed*com or frtib*com or ca-c19*com or dgi*com' cmd.run 'df -h | egrep "[890][0-9]\%"'
|
|
|
+salt -C 'afs*local or afs*com or la-*com or nga*com or nga*local or dc*com or bas-*com or frtib*com or ca-c19*com or dgi*com' cmd.run 'df -h | egrep "[890][0-9]\%"'
|
|
|
|
|
|
# SKIP this one as long as Fred's kung fu works
|
|
|
-salt -C 'afs*local or afs*com or la-*com or nga*com or nga*local or dc*com or bas-*com or doed*com or frtib*com or ca-c19*com or dgi*com' cmd.run 'df -h'
|
|
|
+salt -C 'afs*local or afs*com or la-*com or nga*com or nga*local or dc*com or bas-*com or frtib*com or ca-c19*com or dgi*com' cmd.run 'df -h'
|
|
|
|
|
|
-salt -C 'afs*local or afs*com or la-*com or nga*com or nga*local or dc*com or bas-*com or doed*com or frtib*com or ca-c19*com or dgi*com' pkg.upgrade
|
|
|
+salt -C 'afs*local or afs*com or la-*com or nga*com or nga*local or dc*com or bas-*com or frtib*com or ca-c19*com or dgi*com' pkg.upgrade
|
|
|
|
|
|
```
|
|
|
|