Home Esplora Aiuto
Registrati Accedi
AFS
/
infrastructure-notes
2
0
Forka 0
File Problemi 0 Pull Requests 0 Wiki
Albero (Tree): c5eff34045
Rami (Branch) Tag
infrastructure-...
/
Phantom Upgrade Notes.md

Phantom Upgrade Notes.md 8.1 KB
Cronologia Originale

Phantom Upgrade Notes

Splunk Phantom upgrade overview and prerequisites

See also: the installation notes in Phantom Notes

General Notes

Use the Splunk provided Splunk Phantom repo, NOT the XDR managed msoc repo. BE SURE TO HAVE AT MOST 55% FREE space ( 45% used space)

Backup documentation Restore Splunk Phantom from a backup

TODO: Switch to a non-root installation! Future Upgrade may force us to switch.

Upgrade Steps

See Splunk docs!

Take a backup

:warning: Silence Phantom Sensu checks

Stop Phantom /opt/phantom/bin/stop_phantom.sh

Take an AWS snapshot OF ALL DRIVES in addition to the automatic snapshots! Phantom uses the /tmp directory in addition to the /opt directory. Be sure to include the EBS volume that is storing the /opt data. It is 500 GB volume ( prod ) or a 60 GB volume ( TEST ).

Naming Scheme: phantom-pre-upgrade-backup-<current-version>
phantom-pre-upgrade-backup-4.10.6

Take a full phantom backup while phantom is running. NOTE: to restore a phantom backup you must restore it to the same version of Phantom on a different server! You CAN skip the ibackup if you have a good snapshot!
/opt/phantom/bin/start_phantom.sh
/opt/phantom/bin/phenv ibackup --setup
/opt/phantom/bin/phenv ibackup --backup

Prerequisites

Be sure you have enough space! df -h | grep opt

Prep

Calendar Invite for PROD Phantom Upgrade. Coordinate with James Kerr and Greg Rivas for a time that works with the SOC.

Required:

Rivas, Gregory A. <gregory.a.rivas@accenturefederal.com>; Ou, Xiaofeng <xiaofeng.ou@accenturefederal.com>;

Optional:

Accenture Federal Cyber Center <afcc@accenturefederal.com>; XDR-Engineering <xdr.eng@accenturefederal.com>; Plas, Ryan <ryan.m.plas@accenturefederal.com>

Subject: PROD Phantom Upgrade

The production Phantom is going to be upgraded during this time. Please plan accordingly. 

Current version:
New version:

Reason for upgrading: 
<PUT REASONS HERE>

Post to xdr-soc

Phantom is shutting down for an update in 5 minutes!

  1. Stop Phantom
    /opt/phantom/bin/stop_phantom.sh

  2. disable backups

    sed -i -e 's/archive_mode = on/archive_mode = off/i' /opt/phantom/data/db/postgresql.phantom.conf
grep archive_mode /opt/phantom/data/db/postgresql.phantom.conf

  3. Clean yum
    yum clean all

  4. install updates excluding nginx.

:warning: Watch out for the phantom_repo package being updated! Do not update phantom_repo, yet. If phantom is not running i don't think the package upgrade succeeds. Reboot if kernal is updated. yum update --exclude=nginx --disablerepo phantom-base shutdown -r now

  1. Start Phantom ( should be already started due to reboot ) /opt/phantom/bin/start_phantom.sh

  2. Install phantom repo and signing keys use the rpm command to upgrade the repo package. ( RPM preferred )

    rpm -Uvh https://repo.phantom.us/phantom/<major version.minor version>/base/7Server/x86_64/phantom_repo-<major version.minor version.release.build number>-1.x86_64.rpm

rpm -Uvh https://repo.phantom.us/phantom/4.10/base/7Server/x86_64/phantom_repo-4.10.7.63984-1.x86_64.rpm

Upgrade

This takes a LONG time! Use nohup to background the process to avoid SSH timeout issue. Ask Splunk Support for nohup command. ALTERNATE: Use TMUX to keep session alive.

tmux 
/opt/phantom/bin/phantom_setup.sh upgrade --without-apps --no-space-check

SUGGESTED: Open one vertical split window and one horizontal split window in xterm/tmux to watch the upgrade, watch the size of /tmp and watch the /var/log/phantom/phantom_install_log.
tail -f /var/log/phantom/phantom_install_log

NOTE: You should ignore the "Complete!" messages. They are not indicating that the whole upgrade is complete. They are indicating that one RPM package has been upgraded.

Upgrade apps after a successful upgrade.

Verify that Phantom is working properly

  • create new playbook
  • run playbook
  • run search?
  • verify connectivity to splunk
  • verify connectivity to github
  • Ensure you can edit an Event
  • ?

4.10.6

08/2021 minor upgrade to upgrade Nginx due to Vuln scanner. Also removes use of TLSv1.1

4.10.4

05/2021 minor upgrade due to known issue with pgbouncer and okta auth.

Troubleshooting ISSUE: Phantom webpage does not load and shows "internal server error" ( See Splunk Support ticket) RESOLUTION: check permissions on /tmp/uwsgi_invalidate_ss_cache_trigger and ensure they are 666. Then restart uwsgi with /opt/phantom/bin/phsvc restart uwsgi

( if needed try this ) In: "/opt/phantom/usr/python36/lib/python3.6/site-packages/django/apps/ registry.py (https://registry.py) " The line: 'raise RuntimeError("populate() isn't reentrant")' Should be changed to: 'self.app_configs = {}'

4.10.3

05/2021

Follow Splunk Docs!
Switched XDR from offline RPM install to Phantom repo install
I had to upgrade to latest version in 4.9 before upgrading to 4.10
Use tmux to avoid SSH timeout during upgrade?

4.9

08/2020

Prep Work

See Splunk docs!

Silence Phantom Sensu checks

Stop Phantom
/opt/phantom/bin/stop_phantom.sh

Clean yum
yum clean all

Take an AWS snapshot in addition to the automatic snapshots! should be for a 500 GB volume
Naming Scheme: phantom-pre-upgrade-backup-

Run a backup!
sudo phenv python ibackup.pyc --backup

Update OS & reboot (only if kernel updated)
yum update --exclude=nginx

Start Phantom
/opt/phantom/bin/start_phantom.sh

Disable WAL
sed -i -e 's/archive_mode = on/archive_mode = off/i' /opt/phantom/data/db/postgresql.phantom.conf

restart postgres

# 2021-04-12: While troubleshooting a problem, noticed we're on postgres11 now.
/opt/phantom/bin/phsvc restart postgresql-11

Install new repo and keys
rpm -Uvh https://repo.phantom.us/phantom/4.9/base/7Server/x86_64/phantom_repo-4.9.35731-1.x86_64.rpm

Centos7 (Caasp)
rpm -Uvh https://repo.phantom.us/phantom/4.10/base/7/x86_64/phantom_repo-4.10.3.51237-1.x86_64.rpm

Troubleshooting
Error: Error - Phantom requires that the user 'phantom' has access to cron.
Solution: vim /etc/cron.allow and add phantom

Error! It looks like you don't have enough space in your /tmp directory
Your /tmp directory must have a capacity of at least 5GB
If you would like to ignore this check, please re-run with the option --no-space-check

Upgrade

Upgrade script
/opt/phantom/bin/phantom_setup.sh upgrade

Post Upgrade (Run IF the upgrade script produces the message!)
su - postgres -c '/usr/pgsql-11/bin/vacuumdb -h /tmp --all --analyze-in-stages'

Run this to re-setup or backups
phenv python3 /opt/phantom/bin/ibackup.pyc --setup

Verify postgres version
su - postgres -c '/usr/pgsql-11/bin/postgres --version'

Login into web to accept EULA

Administration > Product Settings > Telemetry > OFF

Post Upgrade Steps

  1. Review System Health
    1. Administration -> System Health -> System Health

Have Phantom Administrator verify that email is working properly.

Clear Silence Done!

4.8

Vagrant VM Upgrade

vagrant phantom creds admin/password Password1 ssh use the brad user and ssh key

Test Upgrade

TEST

  1. Make snapshot

Prod Upgrade

PROD

stop phantom take snapshot of drive clean yum cache install RPM for repo upgrade phantom

Phantom Upgrade Steps Do not skip versions. Upgrade incrementally.

  1. Take a snapshot of the server
  2. Stop all services 2.1 /opt/phantom/bin/stop_phantom.sh
  3. Clear yum caches 3.1 yum clean all
  4. update the OS 4.1 yum update --exclude=nginx
  5. reboot if kernel was upgraded 5.1 reboot
  6. after reboot login and installed the phantom repo for the correct version of the software.
    6.1 Splunk Phantom repositories and signing keys packages
    6.2 rpm -Uvh https://repo.phantom.us/phantom/4.6/base/7Server/x86_64/phantom_repo-4.6.19142-1.x86_64.rpm
    6.3 /opt/phantom/bin/phantom_setup.sh upgrade

Post Upgrade Steps

  1. Review System Health
    1. Administration -> System Health -> System Health