# Patching Notes for the CaaSP Environment

[Day 1](#Day-1)

[Day 2](#Day-2)

[Patch/Upgrade Jenkins Container](#Patching-or-Upgrading-the-Jenkins-Container)

## Timeline
* When test or production (commercial or govcloud) is patched

## Patching Process

There isn't typically a need to inform anyone of patching as CaaSP is not considered "production" right now.

## Detailed Steps

### Day 1
#### Step 1: Victim Instances

Connect to the CaaSP Salt Master and run the following commands:

```
### There is also the grain 'role:caasp-victim' that can be used instead of 'vic-*' or 'vic-* or VIC-*'.

### Linux Victims
sudo -i
salt -C 'vic-* and G@kernel:Linux' test.ping --out=txt
salt -C 'vic-* and G@kernel:Linux' cmd.run 'df -h | egrep "[890][0-9]\%"'

# Review packages that will be updated (RHEL family only).
salt -C 'vic-* and G@os_family:RedHat' cmd.run 'yum check-update' 

# Upgrade packages
salt -C 'vic-* and G@kernel:Linux' pkg.upgrade


### Windows Victims
# Check for full disk(s). Anything under 95% is ok. The victims can have very full hard drives. 
salt -G 'os:Windows' status.diskusage --out=txt

#### Upgrade only the packages installed by chocolatey. This does NOT patch the operating system.
#### A failure here may be noted but is not a show stopper.
salt -G 'os:Windows' chocolatey.upgrade all

#### (OPTIONAL) Apply Windows OS updates/patches (it may take quite some time)
salt -t 60 -G 'os:Windows' win_wua.list summary=True skip_installed=True install=True
```

#### Step 2 (Day 1): Splunk, Kali, Bastion, etc. Instances

WARNING: This may upgrade Salt!

NOTE: Upgrading Docker will stop or restart the Jenkins and Phoenix containers.

```
salt -C 'not ( vic-* or VIC-* )' test.ping --out=txt
salt -C 'not ( vic-* or VIC-* )' cmd.run 'df -h | egrep "[890][0-9]\%"'

# Review packages that will be updated for RedHat family OSs.
salt -C 'not ( vic-* or VIC-* ) and ( G@os_family:RedHat )' cmd.run 'yum check-update' 

# Upgrade packages
salt -C 'not ( vic-* or VIC-* )' pkg.upgrade
```

#### Step 3 (Day 1): Post Patching

Ensure the Docker container for Jenkins is still running:
- https://build.caasp.accenturefederalcyber.com/
- OR `ssh` to `caasp-build-01` and run `sudo -u docker docker container ls`

If it is not running, `ssh` to `caasp-build-01` and run:

```
sudo -iu docker
cd jenkins_docker/
../bin/docker-compose up -d
```

Ensure the Phoenix container for Jenkins is still running:
- https://phoenix.caasp.accenturefederalcyber.com/

#### Step 4 (Day 1): Reboot Victims 

Post to Slack xdr-patching channel:
```
 Rebooting CaaSP victims now.
```

```
salt -C 'vic-* or VIC-* or G@role:caasp-victim' test.ping --out=txt
salt -C '( vic-* or VIC-* ) and G@kernel:Linux' system.reboot
salt -C '( vic-* or VIC-* ) and G@os:Windows' system.reboot timeout=30 in_seconds=True

#### This may take a long time--especially for the Windows victims

#### Keep waiting ...
watch "salt -C 'vic-* or VIC-*' test.ping --out=txt"

#### Check uptime. Look for values/seconds less than 1,000.
salt -C 'vic-* or VIC-*' status.uptime --out=txt
```

### Day 2
#### Step 1: Reboot Splunk Instances

Post to Slack:
```
 Rebooting CaaSP Splunk now.
```

```
# Reboot the dev search head, hf, and cm
salt -L 'caasp-splunk-sh-dev,caasp-splunk-hf,caasp-splunk-cm' system.reboot
# Wait for them ...
watch "salt -L 'caasp-splunk-sh-dev,caasp-splunk-hf,caasp-splunk-cm' status.uptime --out=txt"

# Reboot the search head
salt caasp-splunk-sh system.reboot
# Wait for it ...
watch "salt caasp-splunk-sh status.uptime --out=txt"

# Reboot one indexer at a time (ping optional)
salt caasp-splunk-idx-i-0b66b9 test.ping --out=txt
date; salt caasp-splunk-idx-i-0b66b9 system.reboot

# Indexers take a while to restart
watch "salt caasp-splunk-idx-i-0b66b9 status.uptime --out=txt"
```

#### Wait for the Splunk indexing cluster to have four green checkmarks

Log in to https://splunk.caasp.accenturefederalcyber.com or https://sh-dev.caasp.accenturefederalcyber.com and go to `Settings->Indexer clustering`.

You can also go to the Cluster Master but you must [create an SSH tunnel](https://github.xdr.accenturefederalcyber.com/content-delivery/afs_cyber_range_infrastructure/wiki/SSH-Tunneling) to do so.

Repeat the above patching steps for the additional indexers, waiting for four green checks in between each one.

```
# Do the second indexer
salt caasp-splunk-idx-i-0babc3 test.ping --out=txt
date; salt caasp-splunk-idx-i-0babc3 system.reboot

# Indexers take a while to restart
watch "salt caasp-splunk-idx-i-0babc3 status.uptime --out=txt"
```

#### Wait for the Splunk cluster to have four green checkmarks

```
# Do the third indexer
salt caasp-splunk-idx-i-04665e test.ping --out=txt
date; salt caasp-splunk-idx-i-04665e system.reboot

# Indexers take a while to restart
watch "salt caasp-splunk-idx-i-04665e status.uptime --out=txt"

# Verify all indexers rebooted (check for seconds less than a few thousand):
salt 'caasp-splunk-idx-i-*' status.uptime --out=txt
```

#### Ensure all Splunk instances have been rebooted

```
salt 'caasp-splunk-*' status.uptime --out=txt
```

#### Step 2 (Day 2): Reboot Kali, Jenkins, the Bastion, OSCDNS, Phoenix, and Salt Master

Post to Slack:
```
Rebooting CaaSP support infrastructure (Jenkins, Phoenix, etc.) now.
```

```
salt -L 'caasp-kali,caasp-build-01,caasp-bastion,caasp-oscdns,caasp-phoenix-01,caasp-salt-master' test.ping --out=txt
salt -L 'caasp-kali,caasp-build-01,caasp-bastion,caasp-oscdns,caasp-phoenix-01,caasp-salt-master' system.reboot

#### Rebooting will disconnect you from the Salt Master. Once you are able to ssh back in ...
salt -L 'caasp-kali,caasp-build-01,caasp-bastion,caasp-oscdns,caasp-phoenix-01,caasp-salt-master' status.uptime --out=txt
```

## Patching or Upgrading the Jenkins Container


The CaaSP Jenkins container uses the jenkins/jenkins-lts image published on Docker Hub. When you log in to Jenkins and see that there's a new LTS release, it is recommended to upgrade the container. The way to see if a new update is available is to look at the notifications bell in the upper right hand part of the screen. The notification will state, "New version of Jenkins (2.277.1) is available for download."

If needed, post to Slack to notify of the Jenkins Upgrade in the xdr-patching channel:
```
Rebooting CaaSP Jenkins for updates. 
```

SSH to `caasp-build-01`.

The process is as follows:
* Stop the container
* Pull down the updated image
* Start the container

```
sudo -iu docker
cd jenkins_docker/
../bin/docker-compose down
../bin/docker-compose pull
../bin/docker-compose up -d
```

Use `docker logs -f jenkins` to watch the container's logs for `INFO    hudson.WebAppMain$3#run: Jenkins is fully up and running`.