# Phantom Upgrade Notes [Splunk Phantom upgrade overview and prerequisites](https://docs.splunk.com/Documentation/Phantom/latest/Install/UpgradeOverview) See also: the installation notes in [Phantom Notes](Phantom%20Notes.md) # General Notes Use the Splunk provided `Splunk Phantom` repo, NOT the XDR managed `msoc` repo. BE SURE TO HAVE AT MOST 55% FREE space ( 45% used space) Backup documentation [Restore Splunk Phantom from a backup](https://docs.splunk.com/Documentation/Phantom/4.10.2/Admin/Restorefromabackup) TODO: Switch to a non-root installation! Future Upgrade may force us to switch. # Upgrade Steps See Splunk docs! ## Prep Calendar Invite for PROD Phantom Upgrade. Coordinate with James Kerr and Greg Rivas for a time that works with the SOC. Required: ``` Rivas, Gregory A. ; Ou, Xiaofeng ; ``` Optional: ``` Accenture Federal Cyber Center ; XDR-Engineering ; Plas, Ryan ``` Subject: PROD Phantom Upgrade ``` The production Phantom is going to be upgraded during this time. Please plan accordingly. Current version: New version: Reason for upgrading: ``` Post to xdr-soc ``` Phantom is shutting down for an update in 5 minutes! ``` ## Take a backup > :warning: Silence Phantom Sensu checks Stop Phantom `/opt/phantom/bin/stop_phantom.sh` Take an AWS snapshot OF ALL DRIVES in addition to the automatic snapshots! Phantom uses the /tmp directory in addition to the /opt directory. Be sure to include the EBS volume that is storing the /opt data. It is 500 GB volume ( prod ) or a 60 GB volume ( TEST ). ``` Naming Scheme: phantom-pre-upgrade-backup- phantom-pre-upgrade-backup-4.10.4-2 ``` Take a full phantom backup while phantom is running. NOTE: to restore a phantom backup you must restore it to the same version of Phantom on a different server! You CAN skip the ibackup if you have a good snapshot! `/opt/phantom/bin/start_phantom.sh` `/opt/phantom/bin/phenv ibackup --setup` `/opt/phantom/bin/phenv ibackup --backup` ## Prerequisites Be sure you have enough space! `df -h | grep opt` 1. Stop Phantom `/opt/phantom/bin/stop_phantom.sh` 2. disable backups ``` sed -i -e 's/archive_mode = on/archive_mode = off/i' /opt/phantom/data/db/postgresql.phantom.conf grep archive_mode /opt/phantom/data/db/postgresql.phantom.conf ``` 3. Clean yum `yum clean all` 4. install updates excluding nginx. > :warning: Watch out for the phantom_repo package being updated! Do not update phantom_repo, yet. If phantom is not running i don't think the package upgrade succeeds. Reboot if kernal is updated or just reboot for funzies. `yum update --exclude=nginx --disablerepo phantom-base` `shutdown -r now` `ping phantom-0` 5. Start Phantom ( should be already started due to reboot ) `/opt/phantom/bin/start_phantom.sh` 6. Install phantom repo and signing keys use the rpm command to upgrade the repo package. ( RPM preferred ) ``` rpm -Uvh https://repo.phantom.us/phantom//base/7Server/x86_64/phantom_repo--1.x86_64.rpm rpm -Uvh https://repo.phantom.us/phantom/4.10/base/7Server/x86_64/phantom_repo-4.10.7.63984-1.x86_64.rpm ``` ## Upgrade This takes a LONG time! Use nohup to background the process to avoid SSH timeout issue. Ask Splunk Support for nohup command. ALTERNATE: Use TMUX to keep session alive. ``` tmux /opt/phantom/bin/phantom_setup.sh upgrade --without-apps --no-space-check ``` SUGGESTED: Open one vertical split window and one horizontal split window in xterm/tmux to watch the upgrade, watch the size of /tmp and watch the /var/log/phantom/phantom_install_log. `tail -f /var/log/phantom/phantom_install_log` NOTE: You should ignore the "Complete!" messages. They are not indicating that the whole upgrade is complete. They are indicating that one RPM package has been upgraded. Upgrade apps after a successful upgrade. ## Verify that Phantom is working properly - create new playbook - run playbook - run search? - verify connectivity to splunk - verify connectivity to github - Ensure you can edit an Event - ? # 4.10.6 08/2021 minor upgrade to upgrade Nginx due to Vuln scanner. Also removes use of TLSv1.1 # 4.10.4 05/2021 minor upgrade due to known issue with pgbouncer and okta auth. Troubleshooting ISSUE: Phantom webpage does not load and shows "internal server error" ( See Splunk Support ticket) RESOLUTION: check permissions on /tmp/uwsgi_invalidate_ss_cache_trigger and ensure they are 666. Then restart uwsgi with `/opt/phantom/bin/phsvc restart uwsgi` ( if needed try this ) In: "/opt/phantom/usr/python36/lib/python3.6/site-packages/django/apps/ registry.py (https://registry.py) " The line: 'raise RuntimeError("populate() isn't reentrant")' Should be changed to: 'self.app_configs = {}' # 4.10.3 05/2021 Follow Splunk Docs! Switched XDR from offline RPM install to Phantom repo install I had to upgrade to latest version in 4.9 before upgrading to 4.10 Use tmux to avoid SSH timeout during upgrade? # 4.9 08/2020 ## Prep Work See Splunk docs! Silence Phantom Sensu checks Stop Phantom `/opt/phantom/bin/stop_phantom.sh` Clean yum `yum clean all` Take an AWS snapshot in addition to the automatic snapshots! should be for a 500 GB volume Naming Scheme: phantom-pre-upgrade-backup- Run a backup! `sudo phenv python ibackup.pyc --backup` Update OS & reboot (only if kernel updated) `yum update --exclude=nginx` Start Phantom `/opt/phantom/bin/start_phantom.sh` Disable WAL `sed -i -e 's/archive_mode = on/archive_mode = off/i' /opt/phantom/data/db/postgresql.phantom.conf` restart postgres ``` # 2021-04-12: While troubleshooting a problem, noticed we're on postgres11 now. /opt/phantom/bin/phsvc restart postgresql-11 ``` Install new repo and keys `rpm -Uvh https://repo.phantom.us/phantom/4.9/base/7Server/x86_64/phantom_repo-4.9.35731-1.x86_64.rpm` Centos7 (Caasp) `rpm -Uvh https://repo.phantom.us/phantom/4.10/base/7/x86_64/phantom_repo-4.10.3.51237-1.x86_64.rpm` Troubleshooting Error: Error - Phantom requires that the user 'phantom' has access to cron. Solution: `vim /etc/cron.allow` and add phantom Error! It looks like you don't have enough space in your `/tmp directory` Your `/tmp directory` must have a capacity of at least `5GB` If you would like to ignore this check, please re-run with the option `--no-space-check` ## Upgrade Upgrade script `/opt/phantom/bin/phantom_setup.sh upgrade` Post Upgrade (Run IF the upgrade script produces the message!) `su - postgres -c '/usr/pgsql-11/bin/vacuumdb -h /tmp --all --analyze-in-stages'` Run this to re-setup or backups `phenv python3 /opt/phantom/bin/ibackup.pyc --setup` Verify postgres version `su - postgres -c '/usr/pgsql-11/bin/postgres --version'` Login into web to accept EULA Administration > Product Settings > Telemetry > OFF Post Upgrade Steps 1. Review System Health 1. Administration -> System Health -> System Health Have Phantom Administrator verify that email is working properly. Clear Silence Done! # 4.8 ## Vagrant VM Upgrade vagrant phantom creds admin/password Password1 ssh use the brad user and ssh key ## Test Upgrade TEST 1. Make snapshot ## Prod Upgrade PROD stop phantom take snapshot of drive clean yum cache install RPM for repo upgrade phantom Phantom Upgrade Steps Do not skip versions. Upgrade incrementally. 1. Take a snapshot of the server 2. Stop all services 2.1 /opt/phantom/bin/stop_phantom.sh 3. Clear yum caches 3.1 yum clean all 4. update the OS 4.1 yum update --exclude=nginx 5. reboot if kernel was upgraded 5.1 reboot 6. after reboot login and installed the phantom repo for the correct version of the software. 6.1 [Splunk Phantom repositories and signing keys packages](https://docs.splunk.com/Documentation/Phantom/4.8/Install/PhantomReposAndSigningKeys) 6.2 `rpm -Uvh https://repo.phantom.us/phantom/4.6/base/7Server/x86_64/phantom_repo-4.6.19142-1.x86_64.rpm` 6.3 `/opt/phantom/bin/phantom_setup.sh upgrade` Post Upgrade Steps 1. Review System Health 1. Administration -> System Health -> System Health