|
@@ -436,12 +436,11 @@ Resuming today's patching with the reboots of customer LCPs.
|
|
|
|
|
|
Remeber to silence Sensu alerts before restarting servers.
|
|
|
|
|
|
-NOTE: Restart LCPs one server at a time in order to minimize risk of concurrent outages.
|
|
|
+NOTE: Restart LCPs one server at a time at a location in order to minimize risk of concurrent outages.
|
|
|
|
|
|
#### First syslog servers
|
|
|
|
|
|
-*** Try to reboot syslog servers 1,3,and 7 at the same time!!! see below!!! ***
|
|
|
-
|
|
|
+Restart the first syslog server by itself to check for reboot issues.
|
|
|
```
|
|
|
salt -C '*syslog-1* not *.local' cmd.run 'uptime'
|
|
|
date; salt -C '*syslog-1* not *.local' system.reboot
|
|
@@ -450,19 +449,7 @@ watch "salt -C '*syslog-1* not *.local' cmd.run 'ps -ef | grep syslog-ng | grep
|
|
|
# Look for /usr/sbin/syslog-ng -F -p /var/run/syslogd.pid
|
|
|
```
|
|
|
|
|
|
-Troubleshooting
|
|
|
-
|
|
|
-##### Possible issue: if syslog-ng doesn't start, it might need the setenforce 0 command run ( left here for legacy reasons )
|
|
|
|
|
|
-2020-06-11 - had to do this for afs-syslog-5 through 8
|
|
|
-
|
|
|
-```
|
|
|
-salt saf-splunk-syslog-1 cmd.run 'setenforce 0'
|
|
|
-salt saf-splunk-syslog-1 cmd.run 'systemctl stop rsyslog'
|
|
|
-salt saf-splunk-syslog-1 cmd.run 'systemctl start syslog-ng'
|
|
|
-
|
|
|
-watch "salt -C '*syslog-1* not *.local' test.ping"
|
|
|
-```
|
|
|
|
|
|
#### Second syslog servers
|
|
|
|
|
@@ -512,11 +499,22 @@ date; salt -C '*splunk-syslog-2 or *splunk-syslog-4 or *splunk-syslog-6 or *splu
|
|
|
watch "salt -C '*splunk-syslog-2 or *splunk-syslog-4 or *splunk-syslog-6 or *splunk-syslog-8' test.ping"
|
|
|
salt -C '*splunk-syslog-2 or *splunk-syslog-4 or *splunk-syslog-6 or *splunk-syslog-8' cmd.run 'ps -ef | grep syslog-ng | grep -v grep'
|
|
|
|
|
|
-
|
|
|
```
|
|
|
|
|
|
#### Troubleshooting
|
|
|
|
|
|
+##### Possible issue: if syslog-ng doesn't start, it might need the setenforce 0 command run ( left here for legacy reasons )
|
|
|
+
|
|
|
+2020-06-11 - had to do this for afs-syslog-5 through 8
|
|
|
+
|
|
|
+```
|
|
|
+salt saf-splunk-syslog-1 cmd.run 'setenforce 0'
|
|
|
+salt saf-splunk-syslog-1 cmd.run 'systemctl stop rsyslog'
|
|
|
+salt saf-splunk-syslog-1 cmd.run 'systemctl start syslog-ng'
|
|
|
+
|
|
|
+watch "salt -C '*syslog-1* not *.local' test.ping"
|
|
|
+```
|
|
|
+
|
|
|
If the syslog-ng service doesn't start, check the syslog-ng file for oms agent added configurations.
|
|
|
|
|
|
#### Verify logs are flowing
|
|
@@ -548,7 +546,7 @@ NOTICE: NGA sourcetype="zscaler:web" logs are handled by fluentd and can lag beh
|
|
|
### POP DS (could these be restarted at the same time? Or in 2 batches?)
|
|
|
|
|
|
|
|
|
-```
|
|
|
+<!-- ```
|
|
|
salt -C '*splunk-ds-1* not *.local' cmd.run 'uptime'
|
|
|
date; salt -C '*splunk-ds-1* not *.local' system.reboot
|
|
|
watch "salt -C '*splunk-ds-1* not *.local' cmd.run 'uptime'"
|
|
@@ -560,7 +558,7 @@ watch "salt -C '*splunk-ds-2* not *.local' cmd.run 'uptime'"
|
|
|
salt afs-splunk-ds-4 cmd.run 'uptime'
|
|
|
date; salt afs-splunk-ds-4 system.reboot
|
|
|
watch "salt -C '*splunk-ds-4* not *.local' cmd.run 'uptime'"
|
|
|
-```
|
|
|
+``` -->
|
|
|
|
|
|
Don't forget DS-4
|
|
|
|