Patching Notes--CaaSP.md 6.9 KB
Patching Notes for the CaaSP Environment
Patch/Upgrade Jenkins Container
Timeline
- When test or production (Commercial or Govcloud) is patched
Patching Process
There isn't typically a need to inform anyone of patching as CaaSP is not considered "production" right now.
Detailed Steps
Day 1
Step 1: Victim Instances
Connect to the CaaSP Salt Master and run the following commands:
### There is also the grain 'role:caasp-victim' that can be used instead of 'vic-*' or 'vic-* or VIC-*'.
### Linux Victims
sudo -i
salt -C 'vic-* and G@kernel:Linux' test.ping --out=txt
salt -C 'vic-* and G@kernel:Linux' cmd.run 'df -h | egrep "[890][0-9]\%"'
# Review packages that will be updated (RHEL family only).
salt -C 'vic-* and G@os_family:RedHat' cmd.run 'yum check-update'
# Upgrade packages
salt -C 'vic-* and G@kernel:Linux' pkg.upgrade
### Windows Victims
# Check for full disk(s). Anything under 95% is ok. The victims can have very full hard drives.
salt -G 'os:Windows' status.diskusage --out=txt
#### Upgrade only the packages installed by chocolatey. This does NOT patch the operating system.
#### A failure here may be noted but is not a show stopper.
salt -G 'os:Windows' chocolatey.upgrade all
#### (OPTIONAL) Apply Windows OS updates/patches (it may take quite some time)
salt -t 60 -G 'os:Windows' win_wua.list summary=True skip_installed=True install=True
Step 2 (Day 1): Splunk, Kali, Bastion, etc. Instances
:warning: This may upgrade Salt!
NOTE: Upgrading Docker will stop or restart the Jenkins and Phoenix containers.
salt -C 'not ( vic-* or VIC-* )' test.ping --out=txt
salt -C 'not ( vic-* or VIC-* )' cmd.run 'df -h | egrep "[890][0-9]\%"'
# Review packages that will be updated for RedHat family OSs.
salt -C 'not ( vic-* or VIC-* ) and ( G@os_family:RedHat )' cmd.run 'yum check-update'
# Upgrade packages
salt -C 'not ( vic-* or VIC-* )' pkg.upgrade
Step 3 (Day 1): Post Patching
Ensure the Docker container for Jenkins is still running:
- Jenkins Dashboard
- OR
sshtocaasp-build-01and runsudo -u docker docker container ls
If it is not running, ssh to caasp-build-01 and run:
sudo -iu docker
cd jenkins_docker/
../bin/docker-compose up -d
Ensure the Phoenix container for Jenkins is still running:
Step 4 (Day 1): Reboot Victims
Post to Slack #xdr-patching:
Rebooting CaaSP victims now.
salt -C 'vic-* or VIC-* or G@role:caasp-victim' test.ping --out=txt
salt -C '( vic-* or VIC-* ) and G@kernel:Linux' system.reboot
salt -G 'os:Windows' system.reboot timeout=30 in_seconds=True
#### This may take a long time--especially for the Windows victims
#### Keep waiting ...
watch "salt -C 'vic-* or VIC-*' test.ping --out=txt"
#### Check uptime. Look for values/seconds less than 1,000.
salt -C 'vic-* or VIC-*' status.uptime --out=txt
Day 2
Step 1: Reboot Splunk Instances
Post to Slack #xdr-patching Channel:
Rebooting CaaSP Splunk now.
# Reboot the dev search head, hf, and cm
date; salt -L 'caasp-splunk-sh-dev,caasp-splunk-hf,caasp-splunk-cm' system.reboot
# Wait for them ...
watch "salt -L 'caasp-splunk-sh-dev,caasp-splunk-hf,caasp-splunk-cm' status.uptime --out=txt"
# Reboot the search head
date; salt caasp-splunk-sh system.reboot
# Wait for it ...
watch "salt caasp-splunk-sh status.uptime --out=txt"
# Reboot one indexer at a time (ping optional)
salt caasp-splunk-idx-i-0b66b9 test.ping --out=txt
date; salt caasp-splunk-idx-i-0b66b9 system.reboot
# Indexers take a while to restart
watch "salt caasp-splunk-idx-i-0b66b9 status.uptime --out=txt"
Wait for the Splunk indexing cluster to have four green checkmarks
Log in to the CaaSP Splunk platforms below and go to Settings->Indexer clustering:
You can also go to the Cluster Master but you must create an SSH tunnel to do so.
Repeat the above patching steps for the additional indexers, waiting for four green checks in between each one.
# Do the second indexer
salt caasp-splunk-idx-i-0babc3 test.ping --out=txt
date; salt caasp-splunk-idx-i-0babc3 system.reboot
# Indexers take a while to restart
watch "salt caasp-splunk-idx-i-0babc3 status.uptime --out=txt"
Wait for the Splunk cluster to have four green checkmarks
# Do the third indexer
salt caasp-splunk-idx-i-04665e test.ping --out=txt
date; salt caasp-splunk-idx-i-04665e system.reboot
# Indexers take a while to restart
watch "salt caasp-splunk-idx-i-04665e status.uptime --out=txt"
# Verify all indexers rebooted (check for seconds less than a few thousand):
salt 'caasp-splunk-idx-i-*' status.uptime --out=txt
Ensure all Splunk instances have been rebooted
salt 'caasp-splunk-*' status.uptime --out=txt
Step 2 (Day 2): Reboot Kali, Jenkins, the Bastion, OSCDNS, Phoenix, and Salt Master
Post to Slack #xdr-patching:
Rebooting CaaSP support infrastructure (Jenkins, Phoenix, etc.) now.
salt -L 'caasp-kali,caasp-build-01,caasp-bastion,caasp-oscdns,caasp-phoenix-01,caasp-salt-master' test.ping --out=txt
date; salt -L 'caasp-kali,caasp-build-01,caasp-bastion,caasp-oscdns,caasp-phoenix-01,caasp-salt-master' system.reboot
#### Rebooting will disconnect you from the Salt Master. Once you are able to ssh back in ...
salt -L 'caasp-kali,caasp-build-01,caasp-bastion,caasp-oscdns,caasp-phoenix-01,caasp-salt-master' status.uptime --out=txt
Patching or Upgrading the Jenkins Container
The CaaSP Jenkins container uses the jenkins/jenkins-lts image published on Docker Hub. When you log in to Jenkins and see that there's a new LTS release, it is recommended to upgrade the container. The way to see if a new update is available is to look at the notifications bell in the upper right hand part of the screen. The notification will state, "New version of Jenkins (2.277.1) is available for download."
If needed, post to Slack to notify of the Jenkins Upgrade in the #xdr-patching channel:
Rebooting CaaSP Jenkins for updates.
SSH to caasp-build-01.
The process is as follows:
- Stop the container
- Pull down the updated image
Start the container
sudo -iu docker cd jenkins_docker/ ../bin/docker-compose down ../bin/docker-compose pull ../bin/docker-compose up -d
Use docker logs -f jenkins to watch the container's logs for INFO hudson.WebAppMain$3#run: Jenkins is fully up and running.