# Phantom Upgrade Notes https://docs.splunk.com/Documentation/Phantom/latest/Install/UpgradeOverview See also: the installation notes in `Phantom Notes.md` # General Notes Use the `Splunk Phantom` repo, not the `msoc` repo. BE SURE TO HAVE AT MOST 55% FREE space ( 45% used space) Backup documentation [Restore Splunk Phantom from a backup](https://docs.splunk.com/Documentation/Phantom/4.10.2/Admin/Restorefromabackup) TODO: Switch to a non-root installation! Future Upgrade may force us to switch. # Upgrade Steps See Splunk docs! ## Take a backup Silence Phantom Sensu checks Stop Phantom `/opt/phantom/bin/stop_phantom.sh` Take an AWS snapshot OF ALL DRIVES in addition to the automatic snapshots! Phantom uses the /tmp directory in addition to the /opt directory. Be sure to include the EBS volume that is storing the /opt data. It is 500 GB volume ( prod ) or a 60 GB volume ( TEST ). Naming Scheme: phantom-pre-upgrade-backup- Take a full phantom backup while phantom is running. NOTE: to restore a phantom backup you must restore it to the same version of Phantom on a different server! You CAN skip the ibackup if you have a good snapshot! `/opt/phantom/bin/start_phantom.sh` `/opt/phantom/bin/phenv ibackup --setup` `/opt/phantom/bin/phenv ibackup --backup` ## Prerequisites Be sure you have enough space! `df -h | grep opt` ## Prep Calendar Invite for PROD Phantom Upgrade To: ``` Rivas, Gregory A. ; Accenture Federal Cyber Center ; XDR-Engineering ; Ou, Xiaofeng ``` Subject: PROD Phantom Upgrade ``` The production Phantom is going to be upgraded during this time. Please plan accordingly. Current version: New version: Reason for upgrading: ``` Stop Phantom `/opt/phantom/bin/stop_phantom.sh` disable backups `sed -i -e 's/archive_mode = on/archive_mode = off/i' /opt/phantom/data/db/postgresql.phantom.conf` `grep archive_mode /opt/phantom/data/db/postgresql.phantom.conf` Clean yum `yum clean all` install updates excluding nginx. Watch out for the phantom_repo package being updated! Do not update phantom_repo, yet. If phantom is not running i don't think the package upgrade succeeds. Reboot if kernal is updated. `yum update --exclude=nginx` `shutdown -r now` Start Phantom `/opt/phantom/bin/start_phantom.sh` Install phantom repo and signing keys use either the yum upgrade or the rpm command to upgrade the repo package. ( RPM preferred ) `rpm -Uvh https://repo.phantom.us/phantom//base/7Server/x86_64/phantom_repo--1.x86_64.rpm` `rpm -Uvh https://repo.phantom.us/phantom/4.10/base/7Server/x86_64/phantom_repo-4.10.4.56260-1.x86_64.rpm` ## Upgrade This takes a LONG time! Use nohup to background the process to avoid SSH timeout issue. Ask Splunk Support for nohup command. `/opt/phantom/bin/phantom_setup.sh upgrade --without-apps --no-space-check` SUGGESTED: Open one virtical split window and one horizontal split window in xterm/tmux to watch the upgrade, watch the size of /tmp and watch the /var/log/phantom/phantom_install_log. Upgrade apps after a successful upgrade. ## Verify that phantom is working properly - create new playbook - run search ... - verify connectivity to splunk - verify connectivity to github - # 4.10.4 05/2021 minor upgrade due to known issue with pgbouncer and okta auth. Troubleshooting ISSUE: Phantom webpage does not load and shows "internal server error" ( See Splunk Support ticket) RESOLUTION: check permissions on /tmp/uwsgi_invalidate_ss_cache_trigger and ensure they are 666. Then restart uwsgi with `/opt/phantom/bin/phsvc restart uwsgi` ( if needed try this ) In: "/opt/phantom/usr/python36/lib/python3.6/site-packages/django/apps/ registry.py (https://registry.py) " The line: 'raise RuntimeError("populate() isn't reentrant")' Should be changed to: 'self.app_configs = {}' # 4.10.3 05/2021 Follow Splunk Docs! Switched XDR from offline RPM install to Phantom repo install I had to upgrade to latest version in 4.9 before upgrading to 4.10 Use tmux to avoid SSH timeout during upgrade? # 4.9 08/2020 ## Prep Work See Splunk docs! Silence Phantom Sensu checks Stop Phantom `/opt/phantom/bin/stop_phantom.sh` Clean yum `yum clean all` Take an AWS snapshot in addition to the automatic snapshots! should be for a 500 GB volume Naming Scheme: phantom-pre-upgrade-backup- Run a backup! `sudo phenv python ibackup.pyc --backup` Update OS & reboot (only if kernel updated) `yum update --exclude=nginx` Start Phantom `/opt/phantom/bin/start_phantom.sh` Disable WAL `sed -i -e 's/archive_mode = on/archive_mode = off/i' /opt/phantom/data/db/postgresql.phantom.conf` restart postgres ``` # 2021-04-12: While troubleshooting a problem, noticed we're on postgres11 now. /opt/phantom/bin/phsvc restart postgresql-11 ``` Install new repo and keys `rpm -Uvh https://repo.phantom.us/phantom/4.9/base/7Server/x86_64/phantom_repo-4.9.35731-1.x86_64.rpm` Centos7 (Caasp) `rpm -Uvh https://repo.phantom.us/phantom/4.10/base/7/x86_64/phantom_repo-4.10.3.51237-1.x86_64.rpm` Troubleshooting Error: Error - Phantom requires that the user 'phantom' has access to cron. Solution: `vim /etc/cron.allow` and add phantom Error! It looks like you don't have enough space in your `/tmp directory` Your `/tmp directory` must have a capacity of at least `5GB` If you would like to ignore this check, please re-run with the option `--no-space-check` ## Upgrade Upgrade script `/opt/phantom/bin/phantom_setup.sh upgrade` Post Upgrade (Run IF the upgrade script produces the message!) `su - postgres -c '/usr/pgsql-11/bin/vacuumdb -h /tmp --all --analyze-in-stages'` Run this to re-setup or backups `phenv python3 /opt/phantom/bin/ibackup.pyc --setup` Verify postgres version `su - postgres -c '/usr/pgsql-11/bin/postgres --version'` Login into web to accept EULA Administration > Product Settings > Telemetry > OFF Post Upgrade Steps 1. Review System Health 1. Administration -> System Health -> System Health Have Phantom Administrator verify that email is working properly. Clear Silence Done! # 4.8 ## Vagrant VM Upgrade vagrant phantom creds admin/password Password1 ssh use the brad user and ssh key ## Test Upgrade TEST 1. Make snapshot ## Prod Upgrade PROD stop phantom take snapshot of drive clean yum cache install RPM for repo upgrade phantom Phantom Upgrade Steps Do not skip versions. Upgrade incrementally. 1. Take a snapshot of the server 2. Stop all services 2.1 /opt/phantom/bin/stop_phantom.sh 3. Clear yum caches 3.1 yum clean all 4. update the OS 4.1 yum update --exclude=nginx 5. reboot if kernel was upgraded 5.1 reboot 6. after reboot login and installed the phantom repo for the correct version of the software. 6.1 https://docs.splunk.com/Documentation/Phantom/4.8/Install/PhantomReposAndSigningKeys 6.2 rpm -Uvh https://repo.phantom.us/phantom/4.6/base/7Server/x86_64/phantom_repo-4.6.19142-1.x86_64.rpm 6.3 /opt/phantom/bin/phantom_setup.sh upgrade Post Upgrade Steps 1. Review System Health 1. Administration -> System Health -> System Health