infrastructure-notes/Patching Notes--CaaSP.md

Patching Notes for the CaaSP Environment

Day 1

Day 2

Patch/Upgrade Jenkins Container

Timeline

• When test or production (commercial or govcloud) is patched

Patching Process

There isn't typically a need to inform anyone of patching as CaaSP is not considered "production" right now.

Detailed Steps

Day 1

Step 1: Victim Instances

Connect to the CaaSP Salt Master and run the following commands:

### There is also the grain 'role:caasp-victim' but it isn't present on every victim yet

### CentOS Victims
salt -C '( vic-* or caasp-exp* ) and G@os:CentOS' test.ping --out=txt
salt -C '( vic-* or caasp-exp* ) and G@os:CentOS' cmd.run 'df -h | egrep "[890][0-9]\%"'

# Review packages that will be updated.
salt -C '( vic-* or caasp-exp* ) and G@os:CentOS' cmd.run 'yum check-update' 

# Upgrade packages
salt -C '( vic-* or caasp-exp* ) and G@os:CentOS' pkg.upgrade


### Windows Victims
#### NOTE: This will only upgrade packages installed by chocolatey. This does NOT patch the operating system.
salt -G 'os:Windows' chocolatey.upgrade all

Step 2 (Day 1): Splunk, Kali, Bastion, etc. Instances

WARNING: This may upgrade Salt!

NOTE: Upgrading Docker will stop or restart the Jenkins container.

salt -C 'not ( vic-* or caasp-exp* or VIC-* )' test.ping --out=txt
salt -C 'not ( vic-* or caasp-exp* or VIC-* )' cmd.run 'df -h | egrep "[890][0-9]\%"'

# Review packages that will be updated for CentOS.
salt -C 'not ( vic-* or caasp-exp* or VIC-* ) and ( G@os:CentOS or G@os:Amazon )' cmd.run 'yum check-update' 

# Review packages that will be upgraded for Ubuntu
salt caasp-vault cmd.run 'apt-get update'

# Upgrade packages
salt -C 'not ( vic-* or caasp-exp* or VIC-* )' pkg.upgrade

Step 3 (Day 1): Post Patching

Ensure the Docker container for Jenkins is still running:

• https://build.caasp.accenturefederalcyber.com/
• OR ssh to caasp-build-01 and run sudo -u docker docker container ls

If it is not running, ssh to caasp-build-01 and run:

sudo -iu docker
cd jenkins_docker/
../bin/docker-compose up -d

Step 4 (Day 1): Reboot Victims

salt -C 'vic-* or caasp-exp* or VIC-*' test.ping --out=txt
salt -C 'vic-* or caasp-exp* or VIC-*' system.reboot

#### This will take a long time--especially for the Windows victims

#### Keep waiting ...
watch "salt -C 'vic-* or caasp-exp* or VIC-*' test.ping --out=txt"

#### Check uptime. Look for values/seconds less than 1,000.
salt -C 'vic-* or caasp-exp* or VIC-*' status.uptime --out=txt

Day 2

Step 1: Reboot Splunk Instances

salt 'caasp-splunk-*' test.ping --out=txt

# Reboot the cluster master
salt caasp-splunk-cm system.reboot
# Wait for it ...
watch "salt caasp-splunk-cm status.uptime --out=txt"

# Reboot the heavy forwarder
salt caasp-splunk-hf system.reboot
# Wait for it ...
watch "salt caasp-splunk-hf status.uptime --out=txt"

# Reboot the search head
salt caasp-splunk-sh system.reboot
# Wait for it ...
watch "salt caasp-splunk-sh status.uptime --out=txt"

# Reboot one indexer at a time (ping optional)
salt caasp-splunk-idx-i-0b66b9 test.ping --out=txt
date; salt caasp-splunk-idx-i-0b66b9 system.reboot

# Indexers take a while to restart
watch "salt caasp-splunk-idx-i-0b66b9 status.uptime --out=txt"

Wait for the Splunk indexing cluster to have four green checkmarks

Log in to https://splunk.caasp.accenturefederalcyber.com and go to Settings->Indexer clustering.

You can also go to the Cluster Master but you must create an SSH tunnel to do so.

Repeat the above patching steps for the additional indexers, waiting for four green checks in between each one.

# Do the second indexer
salt caasp-splunk-idx-i-0babc3 test.ping --out=txt
date; salt caasp-splunk-idx-i-0babc3 system.reboot

# Indexers take a while to restart
watch "salt caasp-splunk-idx-i-0babc3 status.uptime --out=txt"

Wait for the Splunk cluster to have four green checkmarks

# Do the third indexer
salt caasp-splunk-idx-i-04665e test.ping --out=txt
date; salt caasp-splunk-idx-i-04665e system.reboot

# Indexers take a while to restart
watch "salt caasp-splunk-idx-i-04665e status.uptime --out=txt"

# Verify all indexers rebooted (check for seconds less than a few thousand):
salt 'caasp-splunk-idx-i-*' status.uptime --out=txt

Ensure all Splunk instances have been rebooted

salt 'caasp-splunk-*' status.uptime --out=txt

Step 2 (Day 2): Reboot Kali, Jenkins, Vault, the Bastion, OSCDNS, and Salt Master

salt -L 'caasp-kali,caasp-build-01,caasp-vault,caasp-bastion,caasp-oscdns,caasp-salt-master' test.ping --out=txt
salt -L 'caasp-kali,caasp-build-01,caasp-vault,caasp-bastion,caasp-oscdns,caasp-salt-master' system.reboot

#### This will disconnect you from the Salt Master. Once you are able to ssh back in ...
salt -L 'caasp-kali,caasp-build-01,caasp-vault,caasp-bastion,caasp-oscdns,caasp-salt-master' status.uptime --out=txt

Patching or Upgrading the Jenkins Container

The CaaSP Jenkins container uses the jenkins/jenkins-lts image published on Docker Hub. When you log in to Jenkins and see that there's a new LTS release, it is recommended to upgrade the container.

SSH to caasp-build-01.

The process is as follows:

• Stop the container
• Pull down the updated image
• Start the container

This can be accomplished it two different ways:

sudo -iu docker
cd jenkins_docker/
../bin/docker-compose down
../bin/docker-compose pull
../bin/docker-compose up -d

OR

sudo -iu docker
cd jenkins_docker/
../bin/docker-compose --force-recreate up -d