patching, salt, splunk

DNSSEC Notes.md

@@ -11,6 +11,28 @@ gc-prod-resolver-govcloud
 If DNS resolution stops working, restart the unbound service.
 `systemctl status unbound`
 
+### Troubleshooting
+
+```
+Mar 06 03:25:01 oscontext-unbound.msoc.defpoint.local unbound[372]: /etc/unbound/conf.d/xdr.conf:1: error: cannot open include file '/etc/unbound/conf.... denied
+Mar 06 03:25:01 oscontext-unbound.msoc.defpoint.local unbound[372]: read /etc/unbound/unbound.conf failed: 1 errors in configuration file
+Mar 06 03:25:01 oscontext-unbound.msoc.defpoint.local systemd[1]: unbound.service: main process exited, code=exited, status=1/FAILURE
+Mar 06 03:25:01 oscontext-unbound.msoc.defpoint.local systemd[1]: Unit unbound.service entered failed state.
+Mar 06 03:25:01 oscontext-unbound.msoc.defpoint.local systemd[1]: unbound.service failed.
+```
+
+Fixed by...
+```
+  452  03/06/21 03:50:22 +0000 chown root:unbound /etc/unbound/conf.d/oscontext.conf
+  453  03/06/21 03:50:36 +0000 chmod 640 /etc/unbound/conf.d/oscontext.conf
+  454  03/06/21 03:50:38 +0000 ls -l /etc/unbound/conf.d
+  455  03/06/21 03:50:43 +0000 systemctl restart unbound
+```
+
+
+
+### Setup
+
 AWS resolvers can't play any part whatsoever in DNSSEC. They just break it.
 
 So unbound servers need external DNS.

Patching Notes--CaaSP.md

@@ -53,9 +53,6 @@ salt -C 'not ( vic-* or caasp-exp* or VIC-* )' cmd.run 'df -h | egrep "[890][0-9
 # Review packages that will be updated for CentOS.
 salt -C 'not ( vic-* or caasp-exp* or VIC-* ) and ( G@os:CentOS or G@os:Amazon )' cmd.run 'yum check-update' 
 
-# Update Ubuntu's package cache
-salt caasp-vault cmd.run 'apt-get update'
-
 # Upgrade packages
 salt -C 'not ( vic-* or caasp-exp* or VIC-* )' pkg.upgrade
 ```
@@ -76,9 +73,9 @@ cd jenkins_docker/
 
 #### Step 4 (Day 1): Reboot Victims 
 
-Post to Slack:
+Post to Slack xdr-patching channel:
 ```
- Rebooting caasp victims now.
+Rebooting CaaSP victims now.
 ```
 
 ```
@@ -99,7 +96,7 @@ salt -C 'vic-* or caasp-exp* or VIC-*' status.uptime --out=txt
 
 Post to Slack:
 ```
- Rebooting caasp splunk now.
+Rebooting caasp splunk now.
 ```
 
 ```

Patching Notes.md

@@ -21,7 +21,7 @@ Each month the environment must be patched to comply with FedRAMP requirements.
 
 Email Template that needs to be sent out prior to patching and email addresses of individuals who should get the email. 
 ```
-Leonard, Wesley A. <wesley.a.leonard@accenturefederal.com>; Waddle, Duane E. <duane.e.waddle@accenturefederal.com>; Nair, Asha A. <asha.a.nair@accenturefederal.com>; Middleton, S. <s.middleton@accenturefederal.com>; Crawley, Angelita <angelita.crawley@accenturefederal.com>; Rivas, Gregory A. <gregory.a.rivas@accenturefederal.com>; Damstra, Frederick T. <frederick.t.damstra@accenturefederal.com>; Poulton, Brad <brad.poulton@accenturefederal.com>; Williams, Colby <colby.williams@accenturefederal.com>; Mahmood, Shahid <shahid.mahmood@accenturefederal.com>; Naughton, Brandon <brandon.naughton@accenturefederal.com>
+Leonard, Wesley A. <wesley.a.leonard@accenturefederal.com>; Waddle, Duane E. <duane.e.waddle@accenturefederal.com>; Nair, Asha A. <asha.a.nair@accenturefederal.com>; Middleton, S. <s.middleton@accenturefederal.com>; Crawley, Angelita <angelita.crawley@accenturefederal.com>; Rivas, Gregory A. <gregory.a.rivas@accenturefederal.com>; Damstra, Frederick T. <frederick.t.damstra@accenturefederal.com>; Poulton, Brad <brad.poulton@accenturefederal.com>; Williams, Colby <colby.williams@accenturefederal.com>; Mahmood, Shahid <shahid.mahmood@accenturefederal.com>; Naughton, Brandon <brandon.naughton@accenturefederal.com>; Cooper, Jeremy <jeremy.cooper@accenturefederal.com>;
 ```
 
 ```
@@ -82,16 +82,16 @@ FYI, patching today.
 Starting with moose and internal infra patching. Check disk space for potential issues. 
 ```
 salt -C '* not ( afs* or saf* or nga* or ma-* or mo-* or dc-c19* or la-c19* or nihor* or bp-ot-demo* )' test.ping --out=txt
-salt -C '* not ( afs* or saf* or nga* or ma-* or mo-* or dc-c19* or la-c19* or nihor* or bp-ot-demo*)' cmd.run 'df -h /boot'  
-salt -C '* not ( afs* or saf* or nga* or ma-* or mo-* or dc-c19* or la-c19* or nihor* or bp-ot-demo*)' cmd.run 'df -h /var/log'   # some at 63%
-salt -C '* not ( afs* or saf* or nga* or ma-* or mo-* or dc-c19* or la-c19* or nihor* or bp-ot-demo*)' cmd.run 'df -h /var'        # one at 74%
-salt -C '* not ( afs* or saf* or nga* or ma-* or mo-* or dc-c19* or la-c19* or nihor* or bp-ot-demo*)' cmd.run 'df -h'
+salt -C '* not ( afs* or saf* or nga* or ma-* or mo-* or dc-c19* or la-c19* or nihor* or bp-ot-demo* )' cmd.run 'df -h /boot'  
+salt -C '* not ( afs* or saf* or nga* or ma-* or mo-* or dc-c19* or la-c19* or nihor* or bp-ot-demo* )' cmd.run 'df -h /var/log'   # some at 63%
+salt -C '* not ( afs* or saf* or nga* or ma-* or mo-* or dc-c19* or la-c19* or nihor* or bp-ot-demo* )' cmd.run 'df -h /var'        # one at 74%
+salt -C '* not ( afs* or saf* or nga* or ma-* or mo-* or dc-c19* or la-c19* or nihor* or bp-ot-demo* )' cmd.run 'df -h'
 
 # Fred's update for df -h:
-salt -C '* not ( afs* or saf* or nga* or ma-* or mo-* or dc-c19* or la-c19* or nihor* or bp-ot-demo*)' cmd.run 'df -h | egrep "[890][0-9]\%"'
+salt -C '* not ( afs* or saf* or nga* or ma-* or mo-* or dc-c19* or la-c19* or nihor* or bp-ot-demo* )' cmd.run 'df -h | egrep "[890][0-9]\%"'
 
 # Review packages that will be updated. some packages are versionlocked (Collectd, Splunk,etc.).
-salt -C '* not ( afs* or saf* or nga* or ma-* or mo-* or dc-c19* or la-c19* or nihor* or bp-ot-demo*)' cmd.run 'yum check-update' 
+salt -C '* not ( afs* or saf* or nga* or ma-* or mo-* or dc-c19* or la-c19* or nihor* or bp-ot-demo* )' cmd.run 'yum check-update' 
 
 ### OpenVPN sometimes goes down with patching and needs a restart of the service. 
 ### Let's patch the VPN after everthing else. I am not sure which package is causing the issue. Kernal? bind-utils? 
@@ -101,13 +101,13 @@ salt -C '* not ( afs* or saf* or nga* or ma-* or mo-* or dc-c19* or la-c19* or n
 salt -C 'openvpn*' pkg.upgrade
 
 # Just to be sure, run it again to make sure nothing got missed. 
-salt -C '* not ( afs* or saf* or nga* or ma-* or mo-* or dc-c19* or la-c19* or nihor* or bp-ot-demo*)' pkg.upgrade exclude='phantom_repo'
+salt -C '* not ( afs* or saf* or nga* or ma-* or mo-* or dc-c19* or la-c19* or nihor* or bp-ot-demo* )' pkg.upgrade exclude='phantom_repo'
 
 #patch GC ( from the GC salt master )
-salt -C  '*accenturefederalcyber.com not ( nihor* or bp-ot-demo* or bas-* )' test.ping
-salt -C  '*accenturefederalcyber.com not ( nihor* or bp-ot-demo* or bas-* )' cmd.run 'df -h | egrep "[890][0-9]\%"'
-salt -C  '*accenturefederalcyber.com not ( nihor* or bp-ot-demo* or bas-* )' cmd.run 'yum check-update'
-salt -C  '*accenturefederalcyber.com not ( nihor* or bp-ot-demo* or bas-* )' pkg.upgrade
+salt -C  '*accenturefederalcyber.com not ( nihor* or bp-ot-demo* or bas-* or doed* )' test.ping
+salt -C  '*accenturefederalcyber.com not ( nihor* or bp-ot-demo* or bas-* or doed* )' cmd.run 'df -h | egrep "[890][0-9]\%"'
+salt -C  '*accenturefederalcyber.com not ( nihor* or bp-ot-demo* or bas-* or doed* )' cmd.run 'yum check-update'
+salt -C  '*accenturefederalcyber.com not ( nihor* or bp-ot-demo* or bas-* or doed* )' pkg.upgrade
 ```
 
 > :warning: After upgrades check on Portal to make sure it is still up. 

Salt Upgrade 3001.2 -> 3001.6 Notes.md

@@ -51,10 +51,14 @@ salt salt* cmd.run 'yum check-update | grep salt'
 cmd.run_bg 'systemd-run --scope yum update salt-minion -y && sleep 240 && systemctl daemon-reload && sleep 20 && systemctl start salt-minion'
 ```
 
+
+
 Did you miss any?
 `salt -G saltversion:3001.3 test.ping`
 
 
+BAD DNS for Splunk returner
+requests.packages.urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='moose-hec.xdr.accenturefederalcyber.com', port=8088): Max retries exceeded with url: /services/collector/event (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fb058f0deb8>: Failed to establish a new connection: [Errno 110] Connection timed out',))
 
 
 

Splunk Notes.md

@@ -7,7 +7,6 @@ Change user to Splunk
 sudo -iu splunk
 
 
-
 ---
 How to apply the git changes to the CM or customer DS. Be patient, it is splunk. Review logs in salt
 
@@ -35,7 +34,8 @@ Splunk CM is the license master and the salt master is used to push out a new li
 
 Update the license file at salt/fileroots/splunk/files/licenses/<customer>/
 
-`salt *cm* state.sls splunk.license_master`
+`salt-run 
+`salt *cm* state.sls splunk.license_master --output-diff`