# Patching Notes for the CaaSP Environment

:warning:
Try to _*not*_ upgrade Salt (minion or master) to 3005. The Linux boxes (not victims) should be version locked to 3004.2. I do not know if the Chocolatey install will upgrade or not for Windows.
:warning:

[Day 1](#Day-1)

[Day 2](#Day-2)

[Patch/Upgrade Jenkins Container](#Patching-or-Upgrading-the-Jenkins-Container)

## Timeline
* When test or production (Commercial or Govcloud) is patched

## Patching Process

There isn't typically a need to inform anyone of patching as CaaSP is not considered "production" right now.

:warning: Not all CaaSP instances may be running. Use the `xdrtest` tool to check the status of all CaaSP EC2
instances and start any that are stopped.

```shell
xdrtest --profile cyber-range --tagvalue CaaSP status

### Followed by

xdrtest --profile cyber-range --tagvalue CaaSP-OnDemand status
```

## Detailed Steps

### Day 1
#### Step 1: Victim Instances

Connect to the CaaSP Salt Server and run the following commands:

```shell
### There is also the grain 'role:caasp-victim' that can be used instead of 'vic-*' or 'vic-* or VIC-*'.

### Linux Victims
sudo -iu root
salt -C 'vic-* or v-* and G@kernel:Linux' test.ping --out=txt
salt -C 'vic-* or v-* and G@kernel:Linux' cmd.run 'df -h | egrep "[890][0-9]\%"'

# Review packages that will be updated (RHEL family only).
salt -C 'vic-* or v-* and G@os_family:RedHat' cmd.run 'yum check-update'

# Upgrade the Salt minion (if a new version is available)
salt -C 'vic-* or v-* and G@kernel:Linux' state.sls caasp.salt-minion

# Upgrade packages
salt -C 'vic-* or v-* and G@kernel:Linux' pkg.upgrade


### Windows Victims
# Check for full disk(s). Anything under 95% is ok. The victims can have very full hard drives. 
salt -G 'os:Windows' status.diskusage --out=txt

#### Upgrade only the packages installed by Chocolatey. This does NOT patch the operating system;
#### however, it _may_ upgrade the Salt minion.
#### A failure here may be noted but is not a show stopper.

# NOTE: To upgrade only the Salt minion, use `salt -G 'os:Windows' state.sls caasp.salt-minion` and wait
# a few minutes for the new version to start.
:warning: Do not perform this step until the Salt master is upgraded to the 3005 release.
salt -G 'os:Windows' chocolatey.upgrade all

#### (OPTIONAL) Apply Windows OS updates/patches (it may take quite some time)
salt -t 60 -G 'os:Windows' win_wua.list summary=True skip_installed=True install=True
```

#### Step 2 (Day 1): Splunk, Kali, Bastion, etc. Instances

> :warning: This may upgrade Salt! There is a Salt state to upgrade the Salt minions: `salt \* state.sls caasp.salt-minion`

NOTE: Upgrading Docker may stop or restart the Jenkins and Phoenix containers.

```shell
salt -C 'not ( vic-* or VIC-* or v-* )' test.ping --out=txt
salt -C 'not ( vic-* or VIC-* or v-* )' cmd.run 'df -h | egrep "[890][0-9]\%"'

# Review packages that will be updated for RedHat family OSs.
salt -C 'not ( vic-* or VIC-* or v-* ) and ( G@os_family:RedHat )' cmd.run 'yum check-update'

# Upgrade packages
date; salt -C 'not ( vic-* or VIC-* or v-* )' pkg.upgrade
```

#### Step 3 (Day 1): Post Patching

Ensure the Docker container for Jenkins is still running:
- [Jenkins Dashboard](https://build.caasp.accenturefederalcyber.com/)
- OR `tsh ssh` to `caasp-jenkins` and run `sudo -u docker docker container ls`

If it is not running, `tsh ssh` to `caasp-jenkins` and run:

```shell
sudo -iu docker
cd jenkins_docker/
../bin/docker-compose up -d
```

Ensure the Phoenix container for Jenkins is still running:
- [Phoenix Dashboard](https://phoenix.caasp.accenturefederalcyber.com/)

#### Step 4 (Day 1): Reboot Victims 

Post to Slack [#xdr-patching](https://afscyber.slack.com/archives/CJ462RRBM):
```
Rebooting CaaSP victims now.
```

```shell
salt -C 'vic-* or VIC-* or v-* or G@role:caasp-victim' test.ping --out=txt
date; salt -C '( vic-* or VIC-* or v-* ) and G@kernel:Linux' system.reboot --async
date; salt -G 'os:Windows' system.reboot timeout=30 in_seconds=True

#### This may take a long time--especially for the Windows victims

#### Keep waiting ...
watch "salt -C 'vic-* or VIC-* or v-*' test.ping --out=txt"

#### Check uptime. Look for values/seconds less than 1,000.
salt -C 'vic-* or VIC-* or v-*' status.uptime --out=txt
```

### Day 2
#### Step 1: Reboot Splunk Instances

Post to Slack [#xdr-patching Channel](https://afscyber.slack.com/archives/CJ462RRBM):
```
 Rebooting CaaSP Splunk and CaaSP Phantom now.
```

```shell
date; salt -L 'caasp-splunk-sh-dev,caasp-splunk-hf,caasp-splunk-cm,caasp-phantom' test.ping -t 5

# Check for disk usage
salt -L 'caasp-splunk-sh-dev,caasp-splunk-hf,caasp-splunk-cm,caasp-phantom' cmd.run 'df -h | egrep "[890][0-9]\%"'

# Reboot the dev search head, HF, CM, and Phantom
date; salt -L 'caasp-splunk-sh-dev,caasp-splunk-hf,caasp-splunk-cm,caasp-phantom' system.reboot --async

# Wait for them ...
watch "salt -L 'caasp-splunk-sh-dev,caasp-splunk-hf,caasp-splunk-cm,caasp-phantom' status.uptime --out=txt"

# Verify Splunk Service is Active
salt -L 'caasp-splunk-sh-dev,caasp-splunk-hf,caasp-splunk-cm' cmd.run 'systemctl status splunk | grep Active'

# Reboot the search head
salt caasp-splunk-sh test.ping --out=txt
date; salt caasp-splunk-sh system.reboot --async

# Wait for it ...
watch "salt caasp-splunk-sh status.uptime --out=txt"

# Reboot one indexer at a time (ping optional)
salt 'caasp-splunk-idx-i-<IDHERE>' test.ping --out=txt
salt 'caasp-splunk-idx-i-<IDHERE>' cmd.run 'df -h | egrep "[890][0-9]\%"'
date; 'salt caasp-splunk-idx-i-<IDHERE>' system.reboot --async

# Indexers take a while to restart
watch "salt 'caasp-splunk-idx-i-<IDHERE>' status.uptime --out=txt"
```

#### Wait for the Splunk indexing cluster to have four green checkmarks

Log in to the CaaSP Splunk platforms below and go to `Settings->Indexer clustering`:
   * [CaaSP Splunk](https://splunk.caasp.accenturefederalcyber.com) or 
   * [CaaSP Dev Search Head](https://sh-dev.caasp.accenturefederalcyber.com) 

You can also go to the Cluster Master but you must [create an SSH tunnel](https://github.xdr.accenturefederalcyber.com/content-delivery/afs_cyber_range_infrastructure/wiki/SSH-Tunneling) to do so.

Repeat the above patching steps for the additional indexers, waiting for `four` green checks in between each one.

```shell
# Do the second indexer
salt 'caasp-splunk-idx-i-<IDHERE>' test.ping --out=txt
salt 'caasp-splunk-idx-i-<IDHERE>' cmd.run 'df -h | egrep "[890][0-9]\%"'
date; salt 'caasp-splunk-idx-i-<IDHERE>' system.reboot --async

# Indexers take a while to restart
watch "salt 'caasp-splunk-idx-i-<IDHERE>' status.uptime --out=txt"
```

#### Wait for the Splunk cluster to have four green checkmarks

```shell
# Do the third indexer
salt 'caasp-splunk-idx-i-<IDHERE>' test.ping --out=txt
salt 'caasp-splunk-idx-i-<IDHERE>' cmd.run 'df -h | egrep "[890][0-9]\%"'
date; salt 'caasp-splunk-idx-i-<IDHERE>' system.reboot --async

# Indexers take a while to restart
watch "salt 'caasp-splunk-idx-i-<IDHERE>' status.uptime --out=txt"

# Verify all indexers rebooted (check for seconds less than a few thousand):
salt 'caasp-splunk-idx-i-*' status.uptime --out=txt
salt 'caasp-splunk-idx-i-*' cmd.run 'systemctl status splunk | grep Active'
```

#### Ensure all Splunk instances have been rebooted

```shell
salt 'caasp-splunk-*' status.uptime --out=txt
salt 'caasp-splunk-*' cmd.run 'systemctl status splunk | grep Active'
```

#### Step 2 (Day 2): Reboot Kali, Jenkins, the Bastion, OSCDNS, Phoenix, Cribl, and Salt Master

Post to Slack [#xdr-patching](https://afscyber.slack.com/archives/CJ462RRBM):
```
Rebooting CaaSP support infrastructure (Jenkins, Phoenix, Cribl, etc.) now.
```

```shell
salt -G 'role:caasp-cnc' test.ping --out=text
salt -G 'role:caasp-cnc' cmd.run 'df -h | egrep "[890][0-9]\%"'
date; salt -G 'role:caasp-cnc' system.reboot --async

#### Rebooting will disconnect you from the Salt Master. Once you are able to ssh back in ...
salt -G 'role:caasp-cnc' status.uptime --out=txt
```

## Patching or Upgrading the Jenkins Container


The CaaSP Jenkins container uses the `jenkins/jenkins-lts` image published on Docker Hub. When you log in to Jenkins and see that there's a new LTS release, it is recommended to upgrade the container. The way to see if a new update is available is to look at the notifications bell in the upper right hand part of the screen. The notification will state, `"New version of Jenkins (x.yyy.z) is available for download."`

If needed, post to Slack to notify of the Jenkins Upgrade in the [#xdr-patching](https://afscyber.slack.com/archives/CJ462RRBM) channel:
```
Rebooting CaaSP Jenkins for updates. 
```

SSH to `caasp-jenkins`.

The process is as follows:
* Stop the container
* Pull down the updated image
* Start the container

```shell
sudo -iu docker
cd jenkins_docker/
../bin/docker-compose down
../bin/docker-compose pull
../bin/docker-compose up -d
```

Use `docker logs -f jenkins` to watch the container's logs for `INFO    hudson.WebAppMain$3#run: Jenkins is fully up and running`.