Patching Notes for the CaaSP Environment

Timeline

When test or production (Commercial or Govcloud) is patched

Patching Process

There isn't typically a need to inform anyone of patching as CaaSP is not considered "production" right now.

:warning: Not all CaaSP instances may be running. Use the xdrtest tool to check the status of all CaaSP EC2 instances and start any that are stopped.

xdrtest --profile cyber-range --tagvalue CaaSP status

### Followed by

xdrtest --profile cyber-range --tagvalue CaaSP-OnDemand status

Detailed Steps

Day 1

Step 1: Victim Instances

Connect to the CaaSP Salt Server and run the following commands:

### There is also the grain 'role:caasp-victim' that can be used instead of 'vic-*' or 'vic-* or VIC-*'.

### Linux Victims
sudo -iu root
salt -C 'vic-* or v-* and G@kernel:Linux' test.ping --out=txt
salt -C 'vic-* or v-* and G@kernel:Linux' cmd.run 'df -h | egrep "[890][0-9]\%"'

# Review packages that will be updated (RHEL family only).
salt -C 'vic-* or v-* and G@os_family:RedHat' cmd.run 'yum check-update'

# Upgrade the Salt minion (if a new version is available)
salt -C 'vic-* or v-* and G@kernel:Linux' state.sls caasp.salt-minion

# Upgrade packages
salt -C 'vic-* or v-* and G@kernel:Linux' pkg.upgrade


### Windows Victims
# Check for full disk(s). Anything under 95% is ok. The victims can have very full hard drives. 
salt -G 'os:Windows' status.diskusage --out=txt

#### Upgrade only the packages installed by chocolatey. This does NOT patch the operating system;
#### however, it _may_ upgrade the Salt minion.
#### A failure here may be noted but is not a show stopper.

# NOTE: To upgrade only the Salt minion, use `salt -G 'os:Windows' state.sls caasp.salt-minion` and wait
# a few minutes for the new version to start.
salt -G 'os:Windows' chocolatey.upgrade all

#### (OPTIONAL) Apply Windows OS updates/patches (it may take quite some time)
salt -t 60 -G 'os:Windows' win_wua.list summary=True skip_installed=True install=True

Step 2 (Day 1): Splunk, Kali, Bastion, etc. Instances

:warning: This may upgrade Salt! There is a Salt state to upgrade the Salt minions: salt \* state.sls caasp.salt-minion

NOTE: Upgrading Docker may stop or restart the Jenkins and Phoenix containers.

salt -C 'not ( vic-* or VIC-* or v-* )' test.ping --out=txt
salt -C 'not ( vic-* or VIC-* or v-* )' cmd.run 'df -h | egrep "[890][0-9]\%"'

# Review packages that will be updated for RedHat family OSs.
salt -C 'not ( vic-* or VIC-* or v-* ) and ( G@os_family:RedHat )' cmd.run 'yum check-update'

# Upgrade packages
date; salt -C 'not ( vic-* or VIC-* or v-* )' pkg.upgrade

Step 3 (Day 1): Post Patching

Ensure the Docker container for Jenkins is still running:

Jenkins Dashboard
OR tsh ssh to caasp-jenkins and run sudo -u docker docker container ls

If it is not running, tsh ssh to caasp-jenkins and run:

sudo -iu docker
cd jenkins_docker/
../bin/docker-compose up -d

Ensure the Phoenix container for Jenkins is still running:

Phoenix Dashboard

Step 4 (Day 1): Reboot Victims

Post to Slack #xdr-patching:

Rebooting CaaSP victims now.

salt -C 'vic-* or VIC-* or v-* or G@role:caasp-victim' test.ping --out=txt
date; salt -C '( vic-* or VIC-* or v-* ) and G@kernel:Linux' system.reboot --async
date; salt -G 'os:Windows' system.reboot timeout=30 in_seconds=True

#### This may take a long time--especially for the Windows victims

#### Keep waiting ...
watch "salt -C 'vic-* or VIC-* or v-*' test.ping --out=txt"

#### Check uptime. Look for values/seconds less than 1,000.
salt -C 'vic-* or VIC-* or v-*' status.uptime --out=txt

Day 2

Step 1: Reboot Splunk Instances

Post to Slack #xdr-patching Channel:

 Rebooting CaaSP Splunk and CaaSP Phantom now.

date; salt -L 'caasp-splunk-sh-dev,caasp-splunk-hf,caasp-splunk-cm,caasp-phantom' test.ping -t 5

# Check for disk usage
salt -L 'caasp-splunk-sh-dev,caasp-splunk-hf,caasp-splunk-cm,caasp-phantom' cmd.run 'df -h | egrep "[890][0-9]\%"'

# Reboot the dev search head, HF, CM, and Phantom
date; salt -L 'caasp-splunk-sh-dev,caasp-splunk-hf,caasp-splunk-cm,caasp-phantom' system.reboot --async

# Wait for them ...
watch "salt -L 'caasp-splunk-sh-dev,caasp-splunk-hf,caasp-splunk-cm,caasp-phantom' status.uptime --out=txt"

# Verify Splunk Service is Active
salt -L 'caasp-splunk-sh-dev,caasp-splunk-hf,caasp-splunk-cm' cmd.run 'systemctl status splunk | grep Active'

# Reboot the search head
salt caasp-splunk-sh test.ping --out=txt
date; salt caasp-splunk-sh system.reboot --async

# Wait for it ...
watch "salt caasp-splunk-sh status.uptime --out=txt"

# Reboot one indexer at a time (ping optional)
salt 'caasp-splunk-idx-i-<IDHERE>' test.ping --out=txt
salt 'caasp-splunk-idx-i-<IDHERE>' cmd.run 'df -h | egrep "[890][0-9]\%"'
date; 'salt caasp-splunk-idx-i-<IDHERE>' system.reboot --async

# Indexers take a while to restart
watch "salt 'caasp-splunk-idx-i-<IDHERE>' status.uptime --out=txt"

Wait for the Splunk indexing cluster to have four green checkmarks

You can also go to the Cluster Master but you must create an SSH tunnel to do so.

Repeat the above patching steps for the additional indexers, waiting for four green checks in between each one.

# Do the second indexer
salt 'caasp-splunk-idx-i-<IDHERE>' test.ping --out=txt
salt 'caasp-splunk-idx-i-<IDHERE>' cmd.run 'df -h | egrep "[890][0-9]\%"'
date; salt 'caasp-splunk-idx-i-<IDHERE>' system.reboot --async

# Indexers take a while to restart
watch "salt 'caasp-splunk-idx-i-<IDHERE>' status.uptime --out=txt"

Wait for the Splunk cluster to have four green checkmarks

# Do the third indexer
salt 'caasp-splunk-idx-i-<IDHERE>' test.ping --out=txt
salt 'caasp-splunk-idx-i-<IDHERE>' cmd.run 'df -h | egrep "[890][0-9]\%"'
date; salt 'caasp-splunk-idx-i-<IDHERE>' system.reboot --async

# Indexers take a while to restart
watch "salt 'caasp-splunk-idx-i-<IDHERE>' status.uptime --out=txt"

# Verify all indexers rebooted (check for seconds less than a few thousand):
salt 'caasp-splunk-idx-i-*' status.uptime --out=txt
salt 'caasp-splunk-idx-i-*' cmd.run 'systemctl status splunk | grep Active'

Ensure all Splunk instances have been rebooted

salt 'caasp-splunk-*' status.uptime --out=txt
salt 'caasp-splunk-*' cmd.run 'systemctl status splunk | grep Active'

Step 2 (Day 2): Reboot Kali, Jenkins, the Bastion, OSCDNS, Phoenix, Cribl, and Salt Master

Post to Slack #xdr-patching:

Rebooting CaaSP support infrastructure (Jenkins, Phoenix, Cribl, etc.) now.

salt -G 'role:caasp-cnc' test.ping --out=text
salt -G 'role:caasp-cnc' cmd.run 'df -h | egrep "[890][0-9]\%"'
date; salt -G 'role:caasp-cnc' system.reboot --async

#### Rebooting will disconnect you from the Salt Master. Once you are able to ssh back in ...
salt -G 'role:caasp-cnc' status.uptime --out=txt

Patching or Upgrading the Jenkins Container

The CaaSP Jenkins container uses the jenkins/jenkins-lts image published on Docker Hub. When you log in to Jenkins and see that there's a new LTS release, it is recommended to upgrade the container. The way to see if a new update is available is to look at the notifications bell in the upper right hand part of the screen. The notification will state, "New version of Jenkins (x.yyy.z) is available for download."

If needed, post to Slack to notify of the Jenkins Upgrade in the #xdr-patching channel:

Rebooting CaaSP Jenkins for updates.

SSH to caasp-jenkins.

The process is as follows:

Stop the container
Pull down the updated image

Start the container

sudo -iu docker
cd jenkins_docker/
../bin/docker-compose down
../bin/docker-compose pull
../bin/docker-compose up -d

Use docker logs -f jenkins to watch the container's logs for INFO hudson.WebAppMain$3#run: Jenkins is fully up and running.

Patching Notes--CaaSP.md 8.5 KB Histórico Em bruto