Trang chủ Khám phá Trợ giúp
Đăng ký Đăng nhập
AFS
/
infrastructure-notes
2
0
Fork 0
Các tập tin Các vấn đề 0 Yêu cầu kéo về 0 Wiki
Branch: master
Branches Tags
infrastructure-...
/
Phantom Upgrade Notes.md

Phantom Upgrade Notes.md 12 KB
Permalink Lịch sử Raw

Phantom Upgrade Notes

Splunk Phantom upgrade overview and prerequisites

See also: the installation notes in Phantom Notes

General Notes

Use the Splunk provided Splunk Phantom repo, NOT the XDR managed msoc repo. BE SURE TO HAVE AT MOST 55% FREE space ( 45% used space)

Backup documentation Restore Splunk Phantom from a backup

TODO: Switch to a non-root installation! Future Upgrade may force us to switch.

Upgrade Steps

See Splunk docs!

Prep

Calendar Invite for PROD Phantom Upgrade. Coordinate with James Kerr and Greg Rivas for a time that works with the SOC.

Required:

Rivas, Gregory A. <gregory.a.rivas@accenturefederal.com>; Ou, Xiaofeng <xiaofeng.ou@accenturefederal.com>;

Optional:

Accenture Federal Cyber Center <afcc@accenturefederal.com>; XDR-Engineering <xdr.eng@accenturefederal.com>; Plas, Ryan <ryan.m.plas@accenturefederal.com>

Subject: PROD Splunk Soar Upgrade

The production Splunk Soar is going to be upgraded during this time. Please plan accordingly. 

Current version:
New version:

Reason for upgrading: 
<PUT REASONS HERE>

Post to xdr-soc

@here Phantom / Splunk Soar is shutting down for an update in 5 minutes!

1 Take a backup

:warning: Silence Phantom Sensu checks

Stop Phantom /opt/phantom/bin/stop_phantom.sh

Take an AWS snapshot OF ALL DRIVES in addition to the automatic snapshots! Phantom uses the /tmp directory in addition to the /opt directory. Be sure to include the EBS volume that is storing the /opt data. It is 1000 GB volume ( prod ) or a 60 GB volume ( TEST ).

Update the profile, InstanceId, and tag and run this command to create snapshots of all volumes.

aws --profile mdr-test-c2-gov ec2 create-snapshots --instance-specification 'InstanceId=i-02a546c0de3d20030,ExcludeBootVolume=false' --tag-specifications 'ResourceType=snapshot,Tags=[{Key=Name,Value=phantom-pre-upgrade-backup-5.3.1}]'

Naming Scheme: phantom-pre-upgrade-backup-<current-version>
phantom-pre-upgrade-backup-4.10.7

NOTE: AWS Snapshots occur asynchronously. While a snapshot is completing, an in-progress snapshot is not affected by ongoing reads and writes to the volume. You can use your volume while the snapshot status is pending. Pending state means the blocks are being copied to S3.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-creating-snapshot.html

Take a full phantom backup while phantom is running. NOTE: to restore a phantom backup you must restore it to the same version of Phantom on a DIFFERENT server! You CAN skip the ibackup if you have a good snapshot!
/opt/phantom/bin/start_phantom.sh
/opt/phantom/bin/phenv ibackup --setup
/opt/phantom/bin/phenv ibackup --backup

2 Prerequisites

Be sure you have enough space! df -h | grep opt df -h | grep tmp # must have 5 GB free in /tmp ( we only have 4 GB in /tmp for phantom!! )

  • Stop Phantom
    /opt/phantom/bin/stop_phantom.sh

  • disable backups

    sed -i -e 's/archive_mode = on/archive_mode = off/i' /opt/phantom/data/db/postgresql.phantom.conf
grep archive_mode /opt/phantom/data/db/postgresql.phantom.conf

  • Clean yum
    yum clean all

  • install updates excluding nginx.

:warning: Watch out for the phantom_repo package being updated! Do not update phantom_repo, yet. If phantom is not running i don't think the package upgrade succeeds. Reboot if kernel is updated or just reboot for funzies. yum update --exclude=nginx --disablerepo phantom-base shutdown -r now

ping phantom-0.pvt.xdrtest.accenturefederalcyber.com

  • Start Phantom ( should be already started due to reboot ) /opt/phantom/bin/start_phantom.sh

  • Upgrade Splunk Soar by downloading the installer from the webpage https://my.phantom.us/login/?next=/downloads/

  • Copy the URL and use wget to download the file.

    cd /opt/phantom
wget -O splunk_soar-priv-5.3.3.92213-ebef80f6-el7-x86_64.tgz "https://s3.amazonaws.com/phantom-downloads/5.3.3.92213/splunk_soar-priv-5.3.3.92213-ebef80f6-el7-x86_64.tgz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=1800&X-Amz-Credential=AKIAJQB2QCTG3EQYKMQQ%2F20220908%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Date=20220908T192344Z&X-Amz-Signature=005f79ac9aab5166496797bd1ae06ef715a52359c8e2330d2edb614e875fac01"

wget -O splunk_soar-unpriv-5.3.3.92213-ebef80f6-el7-x86_64.tgz "https://s3.amazonaws.com/phantom-downloads/5.3.3.92213/splunk_soar-unpriv-5.3.3.92213-ebef80f6-el7-x86_64.tgz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=1800&X-Amz-Credential=AKIAJQB2QCTG3EQYKMQQ%2F20220907%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Date=20220907T230817Z&X-Amz-Signature=5a9ad90d41f4f1195b0d05c77ac1e7509d5543207d600aa4abe82e3e92adeafa"

  • Check the sha256 with sha256sum <installer>.tgz and verify with the download webpage.

  • Extract the installer tar -xf <installer>.tgz

  • WARNING: Do not extract in the /root folder. This may fill up the drive! Move the .tgz to /opt/phantom then extract it.

  • EXTRACT it as the phantom user

    mv splunk_soar-priv-5.3.3.92213-ebef80f6-el7-x86_64.tgz /opt/phantom
cd /opt/phantom
tar -xf /opt/phantom/splunk_soar-priv-5.3.3.92213-ebef80f6-el7-x86_64.tgz

3 Upgrade

This takes a LONG time! Use TMUX to keep session alive! No need to upgrade apps at the same time as upgrading soar! Apps can be upgraded after soar is upgraded! For unprivleged installation, run as the phantom user.

su phantom
tmux 
cd splunk-soar
./soar-install --upgrade

If needed, use --ignore-warnings to ignore the warning about connecting to grpc.prod1-cloudgateway.spl.mobi.

SUGGESTED: Open one vertical split window and one horizontal split window in xterm/tmux to watch the upgrade, watch the size of /tmp and watch the /var/log/phantom/phantom_install_log.
tail -f /var/log/phantom/phantom_install_log watch 'df -h /tmp'

NOTE: You should ignore the "Complete!" messages. They are not indicating that the whole upgrade is complete. They are indicating that one RPM package has been upgraded.

4 Repair indicator hashes ( non-FIPS only )

/opt/phantom/bin/repair_520_indicators.sh

  • Upgrade apps after a successful upgrade.
  • Unsilence Sensu Phantom

Verify that Phantom is working properly

  • create new playbook
  • run playbook
  • run search?
  • verify connectivity to splunk
  • verify connectivity to github
  • Ensure you can edit an Event
  • ?

5.3.3

08/2022

ERROR: Warning: Failed to connect to grpc.prod1-cloudgateway.spl.mobi. If you use the mobile app integration, Splunk SOAR must be able to reach this domain. This connectivity is only necessary if using the mobile app integration. pre-deploy checks failed. Warnings can be ignored with --ignore-warnings SOLUTION: ./soar-install --upgrade --with-apps --ignore-warnings

ERROR: install.install_common.InstallError: Failed to trust git directory /opt/phantom/scm/git/Phantom Playbooks. This is casued by the space character in the folder. SOLUTION: PSAAS-9531 Edit the file per https://docs.splunk.com/Documentation/SOARonprem/5.3.3/ReleaseNotes/KnownIssues.

5.3.1

4/2022

The upgrade replaced root's cron!!!!! A quick state.highstate resolved the issue. Be sure to run the Repair indicator hashes script The upgrade might break maxmind. Fix it using the cron job on the server. InstallCustomerPipPackages ERROR: Could not install packages due to an OSError: Proxy URL had no scheme, should start with http:// or https://\n\n Solution: /opt/phantom/bin/phenv python3 -m pip install -r /opt/phantom/usr/local/customer_requirements.txt THIS WAS A FALSE ERROR: Phantom startup failed: /opt/phantom/usr/python39/bin/supervisord ERROR: CRIT Server 'unix_http_server' running without any HTTP authentication checking

/var/log/phantom/wsgi.log ERROR: PermissionError: [Errno 13] Permission denied: '/opt/phantom/usr/python39/lib/python3.9/site-packages/Markdown-3.3.4.dist-info' SOLUTION: change all the directories to 0755, and all the files to 0644.

sudo find /opt/phantom/usr/python39 -type d -exec chmod 755 {} \;
sudo find /opt/phantom/usr/python39 -type f -exec chmod 644 {} \;

# BETTER:
sudo find /opt/phantom/usr/python39 -type d -exec chmod g+rx,o+rx {} \;
sudo find /opt/phantom/usr/python39 -type f -exec chmod g+r,o+r {} \;

5.2.1

4/2022 Must follow the upgrade path.

5.1.0

1/2022 To allow Phantom to run on a system without IPv6 enabled, the /etc/nginx/nginx.conf file needs to be edited and line 40 (listen [::]:80; ) needs to be commented out. This allows nginx to start and Phantom to work again. Splunk case number: 2847652

4.10.6

08/2021 minor upgrade to upgrade Nginx due to Vuln scanner. Also removes use of TLSv1.1

4.10.4

05/2021 minor upgrade due to known issue with pgbouncer and okta auth.

Troubleshooting ISSUE: Phantom webpage does not load and shows "internal server error" ( See Splunk Support ticket) RESOLUTION: check permissions on /tmp/uwsgi_invalidate_ss_cache_trigger and ensure they are 666. Then restart uwsgi with /opt/phantom/bin/phsvc restart uwsgi

( if needed try this ) In: "/opt/phantom/usr/python36/lib/python3.6/site-packages/django/apps/ registry.py (https://registry.py) " The line: 'raise RuntimeError("populate() isn't reentrant")' Should be changed to: 'self.app_configs = {}'

4.10.3

05/2021

Follow Splunk Docs!
Switched XDR from offline RPM install to Phantom repo install
I had to upgrade to latest version in 4.9 before upgrading to 4.10
Use tmux to avoid SSH timeout during upgrade?

4.9

08/2020

Prep Work

See Splunk docs!

Silence Phantom Sensu checks

Stop Phantom
/opt/phantom/bin/stop_phantom.sh

Clean yum
yum clean all

Take an AWS snapshot in addition to the automatic snapshots! should be for a 500 GB volume
Naming Scheme: phantom-pre-upgrade-backup-

Run a backup!
sudo phenv python ibackup.pyc --backup

Update OS & reboot (only if kernel updated)
yum update --exclude=nginx

Start Phantom
/opt/phantom/bin/start_phantom.sh

Disable WAL
sed -i -e 's/archive_mode = on/archive_mode = off/i' /opt/phantom/data/db/postgresql.phantom.conf

restart postgres

# 2021-04-12: While troubleshooting a problem, noticed we're on postgres11 now.
/opt/phantom/bin/phsvc restart postgresql-11

Install new repo and keys
rpm -Uvh https://repo.phantom.us/phantom/4.9/base/7Server/x86_64/phantom_repo-4.9.35731-1.x86_64.rpm

Centos7 (Caasp)
rpm -Uvh https://repo.phantom.us/phantom/4.10/base/7/x86_64/phantom_repo-4.10.3.51237-1.x86_64.rpm

Troubleshooting
Error: Error - Phantom requires that the user 'phantom' has access to cron.
Solution: vim /etc/cron.allow and add phantom

Error! It looks like you don't have enough space in your /tmp directory
Your /tmp directory must have a capacity of at least 5GB
If you would like to ignore this check, please re-run with the option --no-space-check

Upgrade

Upgrade script
/opt/phantom/bin/phantom_setup.sh upgrade

Post Upgrade (Run IF the upgrade script produces the message!)
su - postgres -c '/usr/pgsql-11/bin/vacuumdb -h /tmp --all --analyze-in-stages'

Run this to re-setup or backups
phenv python3 /opt/phantom/bin/ibackup.pyc --setup

Verify postgres version
su - postgres -c '/usr/pgsql-11/bin/postgres --version'

Login into web to accept EULA

Administration > Product Settings > Telemetry > OFF

Post Upgrade Steps

  1. Review System Health
    1. Administration -> System Health -> System Health

Have Phantom Administrator verify that email is working properly.

Clear Silence Done!

4.8

Vagrant VM Upgrade

vagrant phantom creds admin/password Password1 ssh use the brad user and ssh key

Test Upgrade

TEST

  1. Make snapshot

Prod Upgrade

PROD

stop phantom take snapshot of drive clean yum cache install RPM for repo upgrade phantom

Phantom Upgrade Steps Do not skip versions. Upgrade incrementally.

  1. Take a snapshot of the server
  2. Stop all services 2.1 /opt/phantom/bin/stop_phantom.sh
  3. Clear yum caches 3.1 yum clean all
  4. update the OS 4.1 yum update --exclude=nginx
  5. reboot if kernel was upgraded 5.1 reboot
  6. after reboot login and installed the phantom repo for the correct version of the software.
    6.1 Splunk Phantom repositories and signing keys packages
    6.2 rpm -Uvh https://repo.phantom.us/phantom/4.6/base/7Server/x86_64/phantom_repo-4.6.19142-1.x86_64.rpm
    6.3 /opt/phantom/bin/phantom_setup.sh upgrade

Post Upgrade Steps

  1. Review System Health
    1. Administration -> System Health -> System Health