# Phantom Upgrade Notes

[Splunk Phantom upgrade overview and prerequisites](https://docs.splunk.com/Documentation/SOARonprem/latest/Install/UpgradeOverview)

See also: the installation notes in [Phantom Notes](Phantom%20Notes.md)

# General Notes

Use the Splunk provided `Splunk Phantom` repo, NOT the XDR managed `msoc` repo.
BE SURE TO HAVE AT MOST 55% FREE space  ( 45% used space)

Backup documentation [Restore Splunk Phantom from a backup](https://docs.splunk.com/Documentation/Phantom/4.10.2/Admin/Restorefromabackup)

TODO: Switch to a non-root installation! Future Upgrade may force us to switch. 

# Upgrade Steps
See Splunk docs!

## Prep 

Calendar Invite for PROD Phantom Upgrade. Coordinate with James Kerr and Greg Rivas for a time that works with the SOC.

Required:
```
Rivas, Gregory A. <gregory.a.rivas@accenturefederal.com>; Ou, Xiaofeng <xiaofeng.ou@accenturefederal.com>; 
```

Optional:
```
Accenture Federal Cyber Center <afcc@accenturefederal.com>; XDR-Engineering <xdr.eng@accenturefederal.com>; Plas, Ryan <ryan.m.plas@accenturefederal.com>
```

Subject: PROD Splunk Soar Upgrade

```
The production Splunk Soar is going to be upgraded during this time. Please plan accordingly. 

Current version:
New version:

Reason for upgrading: 
<PUT REASONS HERE>
```

Post to xdr-soc
```
@here Phantom / Splunk Soar is shutting down for an update in 5 minutes!
```

## 1 Take a backup

> :warning: Silence Phantom Sensu checks

Stop Phantom 
`/opt/phantom/bin/stop_phantom.sh`

Take an AWS snapshot OF ALL DRIVES in addition to the automatic snapshots! Phantom uses the /tmp directory in addition to the /opt directory. Be sure to include the EBS volume that is storing the /opt data. It is 1000 GB volume ( prod ) or a 60 GB volume ( TEST ).

Update the profile, InstanceId, and tag and run this command to create snapshots of all volumes.
```
aws --profile mdr-test-c2-gov ec2 create-snapshots --instance-specification 'InstanceId=i-02a546c0de3d20030,ExcludeBootVolume=false' --tag-specifications 'ResourceType=snapshot,Tags=[{Key=Name,Value=phantom-pre-upgrade-backup-5.1.0}]'
```

```
Naming Scheme: phantom-pre-upgrade-backup-<current-version>
phantom-pre-upgrade-backup-4.10.7
```
NOTE: CAASP snapshot takes a long time. 

Take a full phantom backup while phantom is running. NOTE: to restore a phantom backup you must restore it to the same version of Phantom on a DIFFERENT server! You CAN skip the ibackup if you have a good snapshot!  
`/opt/phantom/bin/start_phantom.sh`  
`/opt/phantom/bin/phenv ibackup --setup`  
`/opt/phantom/bin/phenv ibackup --backup`  

## 2 Prerequisites
Be sure you have enough space!
`df -h | grep opt`
`df -h | grep tmp`  # must have 5 GB free in /tmp


- Stop Phantom  
`/opt/phantom/bin/stop_phantom.sh`

- disable backups 
```
sed -i -e 's/archive_mode = on/archive_mode = off/i' /opt/phantom/data/db/postgresql.phantom.conf
grep archive_mode /opt/phantom/data/db/postgresql.phantom.conf
```

- Clean yum  
`yum clean all`

- install updates excluding nginx.

> :warning: Watch out for the phantom_repo package being updated! Do not update phantom_repo, yet. If phantom is not running i don't think the package upgrade succeeds. Reboot if kernal is updated or just reboot for funzies.
`yum update --exclude=nginx --disablerepo phantom-base`
`shutdown -r now`

`ping phantom-0.pvt.xdrtest.accenturefederalcyber.com` 

- Start Phantom ( should be already started due to reboot )
`/opt/phantom/bin/start_phantom.sh`

- Install phantom repo and signing keys ( don't skip versions! )
use the rpm command to upgrade the repo package. ( RPM preferred )
```
rpm -Uvh https://repo.phantom.us/phantom/<major version.minor version>/base/7Server/x86_64/phantom_repo-<major version.minor version.release.build number>-1.x86_64.rpm

REDHAT ONLY
rpm -Uvh https://repo.phantom.us/phantom/5.2/base/7Server/x86_64/phantom_repo-5.2.1.78411-1.x86_64.rpm

CENTOS ONLY (CAASP)
rpm -Uvh https://repo.phantom.us/phantom/5.2/base/7/x86_64/phantom_repo-5.2.1.78411-1.x86_64.rpm
```

- Upgrade Splunk Soar Version 5.3.0+ by downloading the installer from the webpage (yuck)
https://my.phantom.us/login/?next=/downloads/ 

- Copy the URL and use wget to download the file. 
wget -O splunk_soar-priv-5.3.1.84890-368eab78-el7-x86_64.tgz "https://s3.amazonaws.com/phantom-downloads/5.3.1.84890/splunk_soar-priv-5.3.1.84890-368eab78-el7-x86_64.tgz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=1800&X-Amz-Credential=AKIAJQB2QCTG3EQYKMQQ%2F20220419%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Date=20220419T202336Z&X-Amz-Signature=5243ac0c1af5189b5ca65d5af19fc17cdf45eab9a3575495c1ab82c677293d0a"

- Check the sha256 with `sha256sum splunk_soar-priv-5.3.1.84890-368eab78-el7-x86_64.tgz` and verify with the download webpage. 
- Extract the installer `tar -xf <installer>.tgz`
- WARNING: Extracting in the /root folder may fill up the drive! Move the .tgz to /opt then extract it to root's home folder. 

```
mv splunk_soar-priv-5.3.1.84890-368eab78-el7-x86_64.tgz /opt/
tar -xf /opt/splunk_soar-priv-5.3.1.84890-368eab78-el7-x86_64.tgz
```

## 3 Upgrade
This takes a LONG time! Use TMUX to keep session alive!
```
tmux 
/opt/phantom/bin/phantom_setup.sh upgrade --without-apps --no-space-check # legacy
./soar-install --upgrade --with-apps
```

SUGGESTED: Open one vertical split window and one horizontal split window in xterm/tmux to watch the upgrade, watch the size of /tmp and watch the /var/log/phantom/phantom_install_log.  
`tail -f /var/log/phantom/phantom_install_log`
`watch 'df -h /tmp'`

NOTE: You should ignore the "Complete!" messages. They are not indicating that the whole upgrade is complete. They are indicating that one RPM package has been upgraded. 

## 4 Repair indicator hashes ( possibly 5.3.1 only )
`/opt/phantom/bin/repair_520_indicators.sh`
- Upgrade apps after a successful upgrade. 
- Unsilence Sensu Phantom

## Verify that Phantom is working properly
- create new playbook
- run playbook
- run search?
- verify connectivity to splunk
- verify connectivity to github
- Ensure you can edit an Event
- ?


# 5.3.1
4/2022

The upgrade replaced root's cron!!!!! A quick state.highstate resolved the issue. 
Be sure to run the Repair indicator hashes script
The upgrade might break maxmind. Fix it using the cron job on the server. 
InstallCustomerPipPackages
ERROR: Could not install packages due to an OSError: Proxy URL had no scheme, should start with http:// or https://\n\n
Solution:
/opt/phantom/bin/phenv python3 -m pip install -r /opt/phantom/usr/local/customer_requirements.txt
THIS WAS A FALSE ERROR:
Phantom startup failed: /opt/phantom/usr/python39/bin/supervisord
ERROR: CRIT Server 'unix_http_server' running without any HTTP authentication checking

/var/log/phantom/wsgi.log
ERROR: PermissionError: [Errno 13] Permission denied: '/opt/phantom/usr/python39/lib/python3.9/site-packages/Markdown-3.3.4.dist-info'
SOLUTION: change all the directories to 0755, and all the files to 0644.
```
sudo find /opt/phantom/usr/python39 -type d -exec chmod 755 {} \;
sudo find /opt/phantom/usr/python39 -type f -exec chmod 644 {} \;

# BETTER:
sudo find /opt/phantom/usr/python39 -type d -exec chmod g+rx,o+rx {} \;
sudo find /opt/phantom/usr/python39 -type f -exec chmod g+r,o+r {} \;
```

# 5.2.1
4/2022
Must follow the upgrade path. 

# 5.1.0
1/2022
To allow Phantom to run on a system without IPv6 enabled, the /etc/nginx/nginx.conf file needs to be edited and line 40 (listen [::]:80; ) needs to be commented out. This allows nginx to start and Phantom to work again.
Splunk case number: 2847652

# 4.10.6
08/2021
minor upgrade to upgrade Nginx due to Vuln scanner. Also removes use of TLSv1.1

# 4.10.4
05/2021
minor upgrade due to known issue with pgbouncer and okta auth. 

Troubleshooting
ISSUE: Phantom webpage does not load and shows "internal server error" ( See Splunk Support ticket)
RESOLUTION: check permissions on /tmp/uwsgi_invalidate_ss_cache_trigger and ensure they are 666. Then restart uwsgi with `/opt/phantom/bin/phsvc restart uwsgi`


( if needed try this )
In: "/opt/phantom/usr/python36/lib/python3.6/site-packages/django/apps/ registry.py (https://registry.py) "
The line: 'raise RuntimeError("populate() isn't reentrant")'
Should be changed to: 'self.app_configs = {}'


# 4.10.3
05/2021

Follow Splunk Docs!  
Switched XDR from offline RPM install to Phantom repo install  
I had to upgrade to latest version in 4.9 before upgrading to 4.10  
Use tmux to avoid SSH timeout during upgrade?  

# 4.9
08/2020

## Prep Work
See Splunk docs!  

Silence Phantom Sensu checks  

Stop Phantom  
`/opt/phantom/bin/stop_phantom.sh`

Clean yum   
`yum clean all`

Take an AWS snapshot in addition to the automatic snapshots! should be for a 500 GB volume  
Naming Scheme: phantom-pre-upgrade-backup-<current-version>

Run a backup!  
`sudo phenv python ibackup.pyc --backup`

Update OS & reboot (only if kernel updated)  
`yum update --exclude=nginx`

Start Phantom  
`/opt/phantom/bin/start_phantom.sh`

Disable WAL  
`sed -i -e 's/archive_mode = on/archive_mode = off/i' /opt/phantom/data/db/postgresql.phantom.conf`

restart postgres 
```
# 2021-04-12: While troubleshooting a problem, noticed we're on postgres11 now.
/opt/phantom/bin/phsvc restart postgresql-11
```

Install new repo and keys  
`rpm -Uvh https://repo.phantom.us/phantom/4.9/base/7Server/x86_64/phantom_repo-4.9.35731-1.x86_64.rpm`

Centos7 (Caasp)  
`rpm -Uvh https://repo.phantom.us/phantom/4.10/base/7/x86_64/phantom_repo-4.10.3.51237-1.x86_64.rpm`

Troubleshooting  
Error: Error - Phantom requires that the user 'phantom' has access to cron.  
Solution: `vim /etc/cron.allow` and add phantom

Error!  It looks like you don't have enough space in your `/tmp directory`  
Your `/tmp directory` must have a capacity of at least `5GB`  
If you would like to ignore this check, please re-run with the option `--no-space-check`  
## Upgrade

Upgrade script  
`/opt/phantom/bin/phantom_setup.sh upgrade`

Post Upgrade (Run IF the upgrade script produces the message!)  
`su - postgres -c '/usr/pgsql-11/bin/vacuumdb -h /tmp --all --analyze-in-stages'`

Run this to re-setup or backups  
`phenv python3 /opt/phantom/bin/ibackup.pyc --setup`

Verify postgres version  
`su - postgres -c '/usr/pgsql-11/bin/postgres --version'`

Login into web to accept EULA

Administration > Product Settings > Telemetry > OFF


Post Upgrade Steps
1. Review System Health
    1. Administration -> System Health -> System Health

Have Phantom Administrator verify that email is working properly. 

Clear Silence
Done!

# 4.8
## Vagrant VM Upgrade
vagrant phantom creds
admin/password
Password1
ssh use the brad user and ssh key

## Test Upgrade
TEST

1. Make snapshot


## Prod Upgrade
PROD

stop phantom
take snapshot of drive
clean yum cache
install RPM for repo
upgrade phantom

Phantom Upgrade Steps Do not skip versions. Upgrade incrementally.
1. Take a snapshot of the server
2. Stop all services
2.1 /opt/phantom/bin/stop_phantom.sh
3. Clear yum caches
3.1 yum clean all
4. update the OS
4.1 yum update --exclude=nginx
5. reboot if kernel was upgraded
5.1 reboot
6. after reboot login and installed the phantom repo for the correct version of the software.  
6.1 [Splunk Phantom repositories and signing keys packages](https://docs.splunk.com/Documentation/Phantom/4.8/Install/PhantomReposAndSigningKeys)  
6.2 `rpm -Uvh https://repo.phantom.us/phantom/4.6/base/7Server/x86_64/phantom_repo-4.6.19142-1.x86_64.rpm`  
6.3 `/opt/phantom/bin/phantom_setup.sh upgrade`  

Post Upgrade Steps
1. Review System Health
    1. Administration -> System Health -> System Health


