infrastructure-notes/Phantom Upgrade Notes.md

Phantom Upgrade Notes

https://docs.splunk.com/Documentation/Phantom/latest/Install/UpgradeOverview

See also: the installation notes in Phantom Notes.md

General Notes

Use the Splunk Phantom repo, not the msoc repo. BE SURE TO HAVE AT MOST 55% FREE space ( 45% used space)

Backup documentation Restore Splunk Phantom from a backup

TODO: Switch to a non-root installation! Future Upgrade may force us to switch.

Upgrade Steps

See Splunk docs!

Take a backup

Silence Phantom Sensu checks

Stop Phantom /opt/phantom/bin/stop_phantom.sh

Take an AWS snapshot OF ALL DRIVES in addition to the automatic snapshots! Phantom uses the /tmp directory in addition to the /opt directory. Be sure to include the EBS volume that is storing the /opt data. It is 500 GB volume ( prod ) or a 60 GB volume ( TEST ). Naming Scheme: phantom-pre-upgrade-backup-

Take a full phantom backup while phantom is running. NOTE: to restore a phantom backup you must restore it to the same version of Phantom on a different server! You CAN skip the ibackup if you have a good snapshot! /opt/phantom/bin/start_phantom.sh /opt/phantom/bin/phenv ibackup --setup /opt/phantom/bin/phenv ibackup --backup

Prerequisites

Be sure you have enough space! df -h | grep opt

Prep

Calendar Invite for PROD Phantom Upgrade

To:

Rivas, Gregory A. <gregory.a.rivas@accenturefederal.com>; Accenture Federal Cyber Center <afcc@accenturefederal.com>; XDR-Engineering <xdr.eng@accenturefederal.com>; Ou, Xiaofeng <xiaofeng.ou@accenturefederal.com>

Subject: PROD Phantom Upgrade

The production Phantom is going to be upgraded during this time. Please plan accordingly. 

Current version:
New version:

Reason for upgrading: 

Stop Phantom /opt/phantom/bin/stop_phantom.sh

disable backups sed -i -e 's/archive_mode = on/archive_mode = off/i' /opt/phantom/data/db/postgresql.phantom.conf grep archive_mode /opt/phantom/data/db/postgresql.phantom.conf

Clean yum yum clean all

install updates excluding nginx. Watch out for the phantom_repo package being updated! Do not update phantom_repo, yet. If phantom is not running i don't think the package upgrade succeeds. Reboot if kernal is updated. yum update --exclude=nginx shutdown -r now

Start Phantom /opt/phantom/bin/start_phantom.sh

Install phantom repo and signing keys use either the yum upgrade or the rpm command to upgrade the repo package. ( RPM preferred )

rpm -Uvh https://repo.phantom.us/phantom/<major version.minor version>/base/7Server/x86_64/phantom_repo-<major version.minor version.release.build number>-1.x86_64.rpm

rpm -Uvh https://repo.phantom.us/phantom/4.10/base/7Server/x86_64/phantom_repo-4.10.4.56260-1.x86_64.rpm

Upgrade

This takes a LONG time! Use nohup to background the process to avoid SSH timeout issue. Ask Splunk Support for nohup command. /opt/phantom/bin/phantom_setup.sh upgrade --without-apps --no-space-check

SUGGESTED: Open one virtical split window and one horizontal split window in xterm/tmux to watch the upgrade, watch the size of /tmp and watch the /var/log/phantom/phantom_install_log.

Upgrade apps after a successful upgrade.

Verify that phantom is working properly

• create new playbook
• run search ...
• verify connectivity to splunk
• verify connectivity to github

4.10.4

05/2021 minor upgrade due to known issue with pgbouncer and okta auth.

Troubleshooting ISSUE: Phantom webpage does not load and shows "internal server error" ( See Splunk Support ticket) RESOLUTION: check permissions on /tmp/uwsgi_invalidate_ss_cache_trigger and ensure they are 666. Then restart uwsgi with /opt/phantom/bin/phsvc restart uwsgi

( if needed try this ) In: "/opt/phantom/usr/python36/lib/python3.6/site-packages/django/apps/ registry.py (https://registry.py) " The line: 'raise RuntimeError("populate() isn't reentrant")' Should be changed to: 'self.app_configs = {}'

4.10.3

05/2021

Follow Splunk Docs! Switched XDR from offline RPM install to Phantom repo install I had to upgrade to latest version in 4.9 before upgrading to 4.10 Use tmux to avoid SSH timeout during upgrade?

4.9

08/2020

Prep Work

See Splunk docs!

Silence Phantom Sensu checks

Stop Phantom /opt/phantom/bin/stop_phantom.sh

Clean yum yum clean all

Take an AWS snapshot in addition to the automatic snapshots! should be for a 500 GB volume Naming Scheme: phantom-pre-upgrade-backup-

Run a backup! sudo phenv python ibackup.pyc --backup

Update OS & reboot (only if kernel updated) yum update --exclude=nginx

Start Phantom /opt/phantom/bin/start_phantom.sh

Disable WAL sed -i -e 's/archive_mode = on/archive_mode = off/i' /opt/phantom/data/db/postgresql.phantom.conf

restart postgres

# 2021-04-12: While troubleshooting a problem, noticed we're on postgres11 now.
/opt/phantom/bin/phsvc restart postgresql-11

Install new repo and keys rpm -Uvh https://repo.phantom.us/phantom/4.9/base/7Server/x86_64/phantom_repo-4.9.35731-1.x86_64.rpm

Centos7 (Caasp) rpm -Uvh https://repo.phantom.us/phantom/4.10/base/7/x86_64/phantom_repo-4.10.3.51237-1.x86_64.rpm

Troubleshooting Error: Error - Phantom requires that the user 'phantom' has access to cron. Solution: vim /etc/cron.allow and add phantom

Error! It looks like you don't have enough space in your /tmp directory Your /tmp directory must have a capacity of at least 5GB If you would like to ignore this check, please re-run with the option --no-space-check

Upgrade

Upgrade script /opt/phantom/bin/phantom_setup.sh upgrade

Post Upgrade (Run IF the upgrade script produces the message!) su - postgres -c '/usr/pgsql-11/bin/vacuumdb -h /tmp --all --analyze-in-stages'

Run this to re-setup or backups phenv python3 /opt/phantom/bin/ibackup.pyc --setup

Verify postgres version su - postgres -c '/usr/pgsql-11/bin/postgres --version'

Login into web to accept EULA

Administration > Product Settings > Telemetry > OFF

Post Upgrade Steps

1. Review System Health
   1. Administration -> System Health -> System Health

Have Phantom Administrator verify that email is working properly.

Clear Silence Done!

4.8

Vagrant VM Upgrade

vagrant phantom creds admin/password Password1 ssh use the brad user and ssh key

Test Upgrade

TEST

1. Make snapshot

Prod Upgrade

PROD

stop phantom take snapshot of drive clean yum cache install RPM for repo upgrade phantom

Phantom Upgrade Steps Do not skip versions. Upgrade incrementally.

1. Take a snapshot of the server
2. Stop all services 2.1 /opt/phantom/bin/stop_phantom.sh
3. Clear yum caches 3.1 yum clean all
4. update the OS 4.1 yum update --exclude=nginx
5. reboot if kernel was upgraded 5.1 reboot
6. after reboot login and installed the phantom repo for the correct version of the software. 6.1 https://docs.splunk.com/Documentation/Phantom/4.8/Install/PhantomReposAndSigningKeys 6.2 rpm -Uvh https://repo.phantom.us/phantom/4.6/base/7Server/x86_64/phantom_repo-4.6.19142-1.x86_64.rpm 6.3 /opt/phantom/bin/phantom_setup.sh upgrade

Post Upgrade Steps

1. Review System Health
   1. Administration -> System Health -> System Health