Home Explore Help
Register Sign In
AFS
/
infrastructure-notes
2
0
Fork 0
Files Issues 0 Pull Requests 0 Wiki
Tree: 983b586a02
Branches Tags
infrastructure-...
/
Phantom Upgrade Notes.md

Phantom Upgrade Notes.md 11 KB
History Raw

Phantom Upgrade Notes

Splunk Phantom upgrade overview and prerequisites

See also: the installation notes in Phantom Notes

General Notes

Use the Splunk provided Splunk Phantom repo, NOT the XDR managed msoc repo. BE SURE TO HAVE AT MOST 55% FREE space ( 45% used space)

Backup documentation Restore Splunk Phantom from a backup

TODO: Switch to a non-root installation! Future Upgrade may force us to switch.

Upgrade Steps

See Splunk docs!

Prep

Calendar Invite for PROD Phantom Upgrade. Coordinate with James Kerr and Greg Rivas for a time that works with the SOC.

Required:

Rivas, Gregory A. <gregory.a.rivas@accenturefederal.com>; Ou, Xiaofeng <xiaofeng.ou@accenturefederal.com>;

Optional:

Accenture Federal Cyber Center <afcc@accenturefederal.com>; XDR-Engineering <xdr.eng@accenturefederal.com>; Plas, Ryan <ryan.m.plas@accenturefederal.com>

Subject: PROD Splunk Soar Upgrade

The production Splunk Soar is going to be upgraded during this time. Please plan accordingly. 

Current version:
New version:

Reason for upgrading: 
<PUT REASONS HERE>

Post to xdr-soc

@here Phantom / Splunk Soar is shutting down for an update in 5 minutes!

1 Take a backup

:warning: Silence Phantom Sensu checks

Stop Phantom /opt/phantom/bin/stop_phantom.sh

Take an AWS snapshot OF ALL DRIVES in addition to the automatic snapshots! Phantom uses the /tmp directory in addition to the /opt directory. Be sure to include the EBS volume that is storing the /opt data. It is 1000 GB volume ( prod ) or a 60 GB volume ( TEST ).

Update the profile, InstanceId, and tag and run this command to create snapshots of all volumes.

aws --profile mdr-test-c2-gov ec2 create-snapshots --instance-specification 'InstanceId=i-02a546c0de3d20030,ExcludeBootVolume=false' --tag-specifications 'ResourceType=snapshot,Tags=[{Key=Name,Value=phantom-pre-upgrade-backup-5.1.0}]'

Naming Scheme: phantom-pre-upgrade-backup-<current-version>
phantom-pre-upgrade-backup-4.10.7

NOTE: CAASP snapshot takes a long time.

Take a full phantom backup while phantom is running. NOTE: to restore a phantom backup you must restore it to the same version of Phantom on a DIFFERENT server! You CAN skip the ibackup if you have a good snapshot!
/opt/phantom/bin/start_phantom.sh
/opt/phantom/bin/phenv ibackup --setup
/opt/phantom/bin/phenv ibackup --backup

2 Prerequisites

Be sure you have enough space! df -h | grep opt df -h | grep tmp # must have 5 GB free in /tmp

  • Stop Phantom
    /opt/phantom/bin/stop_phantom.sh

  • disable backups

    sed -i -e 's/archive_mode = on/archive_mode = off/i' /opt/phantom/data/db/postgresql.phantom.conf
grep archive_mode /opt/phantom/data/db/postgresql.phantom.conf

  • Clean yum
    yum clean all

  • install updates excluding nginx.

:warning: Watch out for the phantom_repo package being updated! Do not update phantom_repo, yet. If phantom is not running i don't think the package upgrade succeeds. Reboot if kernal is updated or just reboot for funzies. yum update --exclude=nginx --disablerepo phantom-base shutdown -r now

ping phantom-0.pvt.xdrtest.accenturefederalcyber.com

3 Upgrade

This takes a LONG time! Use TMUX to keep session alive!

tmux 
/opt/phantom/bin/phantom_setup.sh upgrade --without-apps --no-space-check # legacy
./soar-install --upgrade --with-apps

SUGGESTED: Open one vertical split window and one horizontal split window in xterm/tmux to watch the upgrade, watch the size of /tmp and watch the /var/log/phantom/phantom_install_log.
tail -f /var/log/phantom/phantom_install_log watch 'df -h /tmp'

NOTE: You should ignore the "Complete!" messages. They are not indicating that the whole upgrade is complete. They are indicating that one RPM package has been upgraded.

4 Repair indicator hashes ( possibly 5.3.1 only )

/opt/phantom/bin/repair_520_indicators.sh

  • Upgrade apps after a successful upgrade.
  • Unsilence Sensu Phantom

Verify that Phantom is working properly

  • create new playbook
  • run playbook
  • run search?
  • verify connectivity to splunk
  • verify connectivity to github
  • Ensure you can edit an Event
  • ?

5.3.1

4/2022

The upgrade replaced root's cron!!!!! A quick state.highstate resolved the issue. Be sure to run the Repair indicator hashes script The upgrade might break maxmind. Fix it using the cron job on the server. InstallCustomerPipPackages ERROR: Could not install packages due to an OSError: Proxy URL had no scheme, should start with http:// or https://\n\n Solution: /opt/phantom/bin/phenv python3 -m pip install -r /opt/phantom/usr/local/customer_requirements.txt THIS WAS A FALSE ERROR: Phantom startup failed: /opt/phantom/usr/python39/bin/supervisord ERROR: CRIT Server 'unix_http_server' running without any HTTP authentication checking

/var/log/phantom/wsgi.log ERROR: PermissionError: [Errno 13] Permission denied: '/opt/phantom/usr/python39/lib/python3.9/site-packages/Markdown-3.3.4.dist-info' SOLUTION: change all the directories to 0755, and all the files to 0644.

sudo find /opt/phantom/usr/python39 -type d -exec chmod 755 {} \;
sudo find /opt/phantom/usr/python39 -type f -exec chmod 644 {} \;

# BETTER:
sudo find /opt/phantom/usr/python39 -type d -exec chmod g+rx,o+rx {} \;
sudo find /opt/phantom/usr/python39 -type f -exec chmod g+r,o+r {} \;

5.2.1

4/2022 Must follow the upgrade path.

5.1.0

1/2022 To allow Phantom to run on a system without IPv6 enabled, the /etc/nginx/nginx.conf file needs to be edited and line 40 (listen [::]:80; ) needs to be commented out. This allows nginx to start and Phantom to work again. Splunk case number: 2847652

4.10.6

08/2021 minor upgrade to upgrade Nginx due to Vuln scanner. Also removes use of TLSv1.1

4.10.4

05/2021 minor upgrade due to known issue with pgbouncer and okta auth.

Troubleshooting ISSUE: Phantom webpage does not load and shows "internal server error" ( See Splunk Support ticket) RESOLUTION: check permissions on /tmp/uwsgi_invalidate_ss_cache_trigger and ensure they are 666. Then restart uwsgi with /opt/phantom/bin/phsvc restart uwsgi

( if needed try this ) In: "/opt/phantom/usr/python36/lib/python3.6/site-packages/django/apps/ registry.py (https://registry.py) " The line: 'raise RuntimeError("populate() isn't reentrant")' Should be changed to: 'self.app_configs = {}'

4.10.3

05/2021

Follow Splunk Docs!
Switched XDR from offline RPM install to Phantom repo install
I had to upgrade to latest version in 4.9 before upgrading to 4.10
Use tmux to avoid SSH timeout during upgrade?

4.9

08/2020

Prep Work

See Splunk docs!

Silence Phantom Sensu checks

Stop Phantom
/opt/phantom/bin/stop_phantom.sh

Clean yum
yum clean all

Take an AWS snapshot in addition to the automatic snapshots! should be for a 500 GB volume
Naming Scheme: phantom-pre-upgrade-backup-

Run a backup!
sudo phenv python ibackup.pyc --backup

Update OS & reboot (only if kernel updated)
yum update --exclude=nginx

Start Phantom
/opt/phantom/bin/start_phantom.sh

Disable WAL
sed -i -e 's/archive_mode = on/archive_mode = off/i' /opt/phantom/data/db/postgresql.phantom.conf

restart postgres

# 2021-04-12: While troubleshooting a problem, noticed we're on postgres11 now.
/opt/phantom/bin/phsvc restart postgresql-11

Install new repo and keys
rpm -Uvh https://repo.phantom.us/phantom/4.9/base/7Server/x86_64/phantom_repo-4.9.35731-1.x86_64.rpm

Centos7 (Caasp)
rpm -Uvh https://repo.phantom.us/phantom/4.10/base/7/x86_64/phantom_repo-4.10.3.51237-1.x86_64.rpm

Troubleshooting
Error: Error - Phantom requires that the user 'phantom' has access to cron.
Solution: vim /etc/cron.allow and add phantom

Error! It looks like you don't have enough space in your /tmp directory
Your /tmp directory must have a capacity of at least 5GB
If you would like to ignore this check, please re-run with the option --no-space-check

Upgrade

Upgrade script
/opt/phantom/bin/phantom_setup.sh upgrade

Post Upgrade (Run IF the upgrade script produces the message!)
su - postgres -c '/usr/pgsql-11/bin/vacuumdb -h /tmp --all --analyze-in-stages'

Run this to re-setup or backups
phenv python3 /opt/phantom/bin/ibackup.pyc --setup

Verify postgres version
su - postgres -c '/usr/pgsql-11/bin/postgres --version'

Login into web to accept EULA

Administration > Product Settings > Telemetry > OFF

Post Upgrade Steps

  1. Review System Health
    1. Administration -> System Health -> System Health

Have Phantom Administrator verify that email is working properly.

Clear Silence Done!

4.8

Vagrant VM Upgrade

vagrant phantom creds admin/password Password1 ssh use the brad user and ssh key

Test Upgrade

TEST

  1. Make snapshot

Prod Upgrade

PROD

stop phantom take snapshot of drive clean yum cache install RPM for repo upgrade phantom

Phantom Upgrade Steps Do not skip versions. Upgrade incrementally.

  1. Take a snapshot of the server
  2. Stop all services 2.1 /opt/phantom/bin/stop_phantom.sh
  3. Clear yum caches 3.1 yum clean all
  4. update the OS 4.1 yum update --exclude=nginx
  5. reboot if kernel was upgraded 5.1 reboot
  6. after reboot login and installed the phantom repo for the correct version of the software.
    6.1 Splunk Phantom repositories and signing keys packages
    6.2 rpm -Uvh https://repo.phantom.us/phantom/4.6/base/7Server/x86_64/phantom_repo-4.6.19142-1.x86_64.rpm
    6.3 /opt/phantom/bin/phantom_setup.sh upgrade

Post Upgrade Steps

  1. Review System Health
    1. Administration -> System Health -> System Health