# Patching Notes for the CaaSP Environment

[Day 1](#Day-1)

[Day 2](#Day-2)

[Patch/Upgrade Jenkins Container](#Patching-or-Upgrading-the-Jenkins-Container)

## Timeline
* When test or production (Commercial or Govcloud) is patched

## Patching Process

There isn't typically a need to inform anyone of patching as CaaSP is not considered "production" right now.

## Detailed Steps

### Day 1
#### Step 1: Victim Instances

Connect to the `CaaSP Salt Master` and run the following commands:

```
### There is also the grain 'role:caasp-victim' that can be used instead of 'vic-*' or 'vic-* or VIC-*'.

### Linux Victims
sudo -i
salt -C 'vic-* and G@kernel:Linux' test.ping --out=txt
salt -C 'vic-* and G@kernel:Linux' cmd.run 'df -h | egrep "[890][0-9]\%"'

# Review packages that will be updated (RHEL family only).
salt -C 'vic-* and G@os_family:RedHat' cmd.run 'yum check-update' 

# Upgrade packages
salt -C 'vic-* and G@kernel:Linux' pkg.upgrade


### Windows Victims
# Check for full disk(s). Anything under 95% is ok. The victims can have very full hard drives. 
salt -G 'os:Windows' status.diskusage --out=txt

#### Upgrade only the packages installed by chocolatey. This does NOT patch the operating system.
#### A failure here may be noted but is not a show stopper.
salt -G 'os:Windows' chocolatey.upgrade all

#### (OPTIONAL) Apply Windows OS updates/patches (it may take quite some time)
salt -t 60 -G 'os:Windows' win_wua.list summary=True skip_installed=True install=True
```

#### Step 2 (Day 1): Splunk, Kali, Bastion, etc. Instances

> :warning: This may upgrade Salt!

NOTE: Upgrading Docker will stop or restart the Jenkins and Phoenix containers.

```
salt -C 'not ( vic-* or VIC-* )' test.ping --out=txt
salt -C 'not ( vic-* or VIC-* )' cmd.run 'df -h | egrep "[890][0-9]\%"'

# Review packages that will be updated for RedHat family OSs.
salt -C 'not ( vic-* or VIC-* ) and ( G@os_family:RedHat )' cmd.run 'yum check-update' 

# Upgrade packages
salt -C 'not ( vic-* or VIC-* )' pkg.upgrade
```

#### Step 3 (Day 1): Post Patching

Ensure the Docker container for Jenkins is still running:
- [Jenkins Dashboard](https://build.caasp.accenturefederalcyber.com/)
- OR `ssh` to `caasp-build-01` and run `sudo -u docker docker container ls`

If it is not running, `ssh` to `caasp-build-01` and run:

```
sudo -iu docker
cd jenkins_docker/
../bin/docker-compose up -d
```

Ensure the Phoenix container for Jenkins is still running:
- [Phoenix Dashboard](https://phoenix.caasp.accenturefederalcyber.com/)

#### Step 4 (Day 1): Reboot Victims 

Post to Slack [#xdr-patching](https://afscyber.slack.com/archives/CJ462RRBM):
```
 Rebooting CaaSP victims now.
```

```
salt -C 'vic-* or VIC-* or G@role:caasp-victim' test.ping --out=txt
salt -C '( vic-* or VIC-* ) and G@kernel:Linux' system.reboot
salt -G 'os:Windows' system.reboot timeout=30 in_seconds=True

#### This may take a long time--especially for the Windows victims

#### Keep waiting ...
watch "salt -C 'vic-* or VIC-*' test.ping --out=txt"

#### Check uptime. Look for values/seconds less than 1,000.
salt -C 'vic-* or VIC-*' status.uptime --out=txt
```

### Day 2
#### Step 1: Reboot Splunk Instances

Post to Slack [#xdr-patching Channel](https://afscyber.slack.com/archives/CJ462RRBM):
```
 Rebooting CaaSP Splunk now.
```

```
# Reboot the dev search head, hf, and cm
date; salt -L 'caasp-splunk-sh-dev,caasp-splunk-hf,caasp-splunk-cm' system.reboot

# Wait for them ...
watch "salt -L 'caasp-splunk-sh-dev,caasp-splunk-hf,caasp-splunk-cm' status.uptime --out=txt"

# Reboot the search head
date; salt caasp-splunk-sh system.reboot
# Wait for it ...
watch "salt caasp-splunk-sh status.uptime --out=txt"

# Reboot one indexer at a time (ping optional)
salt caasp-splunk-idx-i-0b66b9 test.ping --out=txt
date; salt caasp-splunk-idx-i-0b66b9 system.reboot

# Indexers take a while to restart
watch "salt caasp-splunk-idx-i-0b66b9 status.uptime --out=txt"
```

#### Wait for the Splunk indexing cluster to have four green checkmarks

Log in to the CaaSP Splunk platforms below and go to `Settings->Indexer clustering`:
   * [CaaSP Splunk](https://splunk.caasp.accenturefederalcyber.com) or 
   * [CaaSP Dev Search Head](https://sh-dev.caasp.accenturefederalcyber.com) 

You can also go to the Cluster Master but you must [create an SSH tunnel](https://github.xdr.accenturefederalcyber.com/content-delivery/afs_cyber_range_infrastructure/wiki/SSH-Tunneling) to do so.

Repeat the above patching steps for the additional indexers, waiting for `four` green checks in between each one.

```
# Do the second indexer
salt caasp-splunk-idx-i-0babc3 test.ping --out=txt
date; salt caasp-splunk-idx-i-0babc3 system.reboot

# Indexers take a while to restart
watch "salt caasp-splunk-idx-i-0babc3 status.uptime --out=txt"
```

#### Wait for the Splunk cluster to have four green checkmarks

```
# Do the third indexer
salt caasp-splunk-idx-i-04665e test.ping --out=txt
date; salt caasp-splunk-idx-i-04665e system.reboot

# Indexers take a while to restart
watch "salt caasp-splunk-idx-i-04665e status.uptime --out=txt"

# Verify all indexers rebooted (check for seconds less than a few thousand):
salt 'caasp-splunk-idx-i-*' status.uptime --out=txt
```

#### Ensure all Splunk instances have been rebooted

```
salt 'caasp-splunk-*' status.uptime --out=txt
```

#### Step 2 (Day 2): Reboot Kali, Jenkins, the Bastion, OSCDNS, Phoenix, and Salt Master

Post to Slack [#xdr-patching](https://afscyber.slack.com/archives/CJ462RRBM):
```
Rebooting CaaSP support infrastructure (Jenkins, Phoenix, etc.) now.
```

```
salt -L 'caasp-kali,caasp-build-01,caasp-bastion,caasp-oscdns,caasp-phoenix-01,caasp-salt-master' test.ping --out=txt
date; salt -L 'caasp-kali,caasp-build-01,caasp-bastion,caasp-oscdns,caasp-phoenix-01,caasp-salt-master' system.reboot

#### Rebooting will disconnect you from the Salt Master. Once you are able to ssh back in ...
salt -L 'caasp-kali,caasp-build-01,caasp-bastion,caasp-oscdns,caasp-phoenix-01,caasp-salt-master' status.uptime --out=txt
```

## Patching or Upgrading the Jenkins Container


The CaaSP Jenkins container uses the `jenkins/jenkins-lts` image published on Docker Hub. When you log in to Jenkins and see that there's a new LTS release, it is recommended to upgrade the container. The way to see if a new update is available is to look at the notifications bell in the upper right hand part of the screen. The notification will state, `"New version of Jenkins (2.277.1) is available for download."`

If needed, post to Slack to notify of the Jenkins Upgrade in the [#xdr-patching](https://afscyber.slack.com/archives/CJ462RRBM) channel:
```
Rebooting CaaSP Jenkins for updates. 
```

SSH to `caasp-build-01`.

The process is as follows:
* Stop the container
* Pull down the updated image
* Start the container

```
sudo -iu docker
cd jenkins_docker/
../bin/docker-compose down
../bin/docker-compose pull
../bin/docker-compose up -d
```

Use `docker logs -f jenkins` to watch the container's logs for `INFO    hudson.WebAppMain$3#run: Jenkins is fully up and running`.