# Phantom Upgrade Notes [Splunk Phantom upgrade overview and prerequisites](https://docs.splunk.com/Documentation/SOARonprem/latest/Install/UpgradeOverview) See also: the installation notes in [Phantom Notes](Phantom%20Notes.md) # General Notes Use the Splunk provided `Splunk Phantom` repo, NOT the XDR managed `msoc` repo. BE SURE TO HAVE AT MOST 55% FREE space ( 45% used space) Backup documentation [Restore Splunk Phantom from a backup](https://docs.splunk.com/Documentation/Phantom/4.10.2/Admin/Restorefromabackup) TODO: Switch to a non-root installation! Future Upgrade may force us to switch. # Upgrade Steps See Splunk docs! ## Prep Calendar Invite for PROD Phantom Upgrade. Coordinate with James Kerr and Greg Rivas for a time that works with the SOC. Required: ``` Rivas, Gregory A. ; Ou, Xiaofeng ; ``` Optional: ``` Accenture Federal Cyber Center ; XDR-Engineering ; Plas, Ryan ``` Subject: PROD Splunk Soar Upgrade ``` The production Splunk Soar is going to be upgraded during this time. Please plan accordingly. Current version: New version: Reason for upgrading: ``` Post to xdr-soc ``` @here Phantom / Splunk Soar is shutting down for an update in 5 minutes! ``` ## 1 Take a backup > :warning: Silence Phantom Sensu checks Stop Phantom `/opt/phantom/bin/stop_phantom.sh` Take an AWS snapshot OF ALL DRIVES in addition to the automatic snapshots! Phantom uses the /tmp directory in addition to the /opt directory. Be sure to include the EBS volume that is storing the /opt data. It is 1000 GB volume ( prod ) or a 60 GB volume ( TEST ). Update the profile, InstanceId, and tag and run this command to create snapshots of all volumes. ``` aws --profile mdr-test-c2-gov ec2 create-snapshots --instance-specification 'InstanceId=i-02a546c0de3d20030,ExcludeBootVolume=false' --tag-specifications 'ResourceType=snapshot,Tags=[{Key=Name,Value=phantom-pre-upgrade-backup-5.1.0}]' ``` ``` Naming Scheme: phantom-pre-upgrade-backup- phantom-pre-upgrade-backup-4.10.7 ``` NOTE: CAASP snapshot takes a long time. Take a full phantom backup while phantom is running. NOTE: to restore a phantom backup you must restore it to the same version of Phantom on a DIFFERENT server! You CAN skip the ibackup if you have a good snapshot! `/opt/phantom/bin/start_phantom.sh` `/opt/phantom/bin/phenv ibackup --setup` `/opt/phantom/bin/phenv ibackup --backup` ## 2 Prerequisites Be sure you have enough space! `df -h | grep opt` `df -h | grep tmp` # must have 5 GB free in /tmp - Stop Phantom `/opt/phantom/bin/stop_phantom.sh` - disable backups ``` sed -i -e 's/archive_mode = on/archive_mode = off/i' /opt/phantom/data/db/postgresql.phantom.conf grep archive_mode /opt/phantom/data/db/postgresql.phantom.conf ``` - Clean yum `yum clean all` - install updates excluding nginx. > :warning: Watch out for the phantom_repo package being updated! Do not update phantom_repo, yet. If phantom is not running i don't think the package upgrade succeeds. Reboot if kernal is updated or just reboot for funzies. `yum update --exclude=nginx --disablerepo phantom-base` `shutdown -r now` `ping phantom-0.pvt.xdrtest.accenturefederalcyber.com` - Start Phantom ( should be already started due to reboot ) `/opt/phantom/bin/start_phantom.sh` - Install phantom repo and signing keys ( don't skip versions! ) use the rpm command to upgrade the repo package. ( RPM preferred ) ``` rpm -Uvh https://repo.phantom.us/phantom//base/7Server/x86_64/phantom_repo--1.x86_64.rpm REDHAT ONLY rpm -Uvh https://repo.phantom.us/phantom/5.2/base/7Server/x86_64/phantom_repo-5.2.1.78411-1.x86_64.rpm CENTOS ONLY (CAASP) rpm -Uvh https://repo.phantom.us/phantom/5.2/base/7/x86_64/phantom_repo-5.2.1.78411-1.x86_64.rpm ``` - Upgrade Splunk Soar Version 5.3.0+ by downloading the installer from the webpage (yuck) https://my.phantom.us/login/?next=/downloads/ - Copy the URL and use wget to download the file. wget -O splunk_soar-priv-5.3.1.84890-368eab78-el7-x86_64.tgz "https://s3.amazonaws.com/phantom-downloads/5.3.1.84890/splunk_soar-priv-5.3.1.84890-368eab78-el7-x86_64.tgz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=1800&X-Amz-Credential=AKIAJQB2QCTG3EQYKMQQ%2F20220419%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Date=20220419T202336Z&X-Amz-Signature=5243ac0c1af5189b5ca65d5af19fc17cdf45eab9a3575495c1ab82c677293d0a" - Check the sha256 with `sha256sum splunk_soar-priv-5.3.1.84890-368eab78-el7-x86_64.tgz` and verify with the download webpage. - Extract the installer `tar -xf .tgz` - WARNING: Extracting in the /root folder may fill up the drive! Move the .tgz to /opt then extract it to root's home folder. ``` mv splunk_soar-priv-5.3.1.84890-368eab78-el7-x86_64.tgz /opt/ tar -xf /opt/splunk_soar-priv-5.3.1.84890-368eab78-el7-x86_64.tgz ``` ## 3 Upgrade This takes a LONG time! Use TMUX to keep session alive! ``` tmux /opt/phantom/bin/phantom_setup.sh upgrade --without-apps --no-space-check # legacy ./soar-install --upgrade --with-apps ``` SUGGESTED: Open one vertical split window and one horizontal split window in xterm/tmux to watch the upgrade, watch the size of /tmp and watch the /var/log/phantom/phantom_install_log. `tail -f /var/log/phantom/phantom_install_log` `watch 'df -h /tmp'` NOTE: You should ignore the "Complete!" messages. They are not indicating that the whole upgrade is complete. They are indicating that one RPM package has been upgraded. ## 4 Repair indicator hashes ( possibly 5.3.1 only ) `/opt/phantom/bin/repair_520_indicators.sh` - Upgrade apps after a successful upgrade. - Unsilence Sensu Phantom ## Verify that Phantom is working properly - create new playbook - run playbook - run search? - verify connectivity to splunk - verify connectivity to github - Ensure you can edit an Event - ? # 5.3.1 4/2022 The upgrade replaced root's cron!!!!! A quick state.highstate resolved the issue. Be sure to run the Repair indicator hashes script The upgrade might break maxmind. Fix it using the cron job on the server. InstallCustomerPipPackages ERROR: Could not install packages due to an OSError: Proxy URL had no scheme, should start with http:// or https://\n\n Solution: /opt/phantom/bin/phenv python3 -m pip install -r /opt/phantom/usr/local/customer_requirements.txt THIS WAS A FALSE ERROR: Phantom startup failed: /opt/phantom/usr/python39/bin/supervisord ERROR: CRIT Server 'unix_http_server' running without any HTTP authentication checking /var/log/phantom/wsgi.log ERROR: PermissionError: [Errno 13] Permission denied: '/opt/phantom/usr/python39/lib/python3.9/site-packages/Markdown-3.3.4.dist-info' SOLUTION: change all the directories to 0755, and all the files to 0644. ``` sudo find /opt/phantom/usr/python39 -type d -exec chmod 755 {} \; sudo find /opt/phantom/usr/python39 -type f -exec chmod 644 {} \; # BETTER: sudo find /opt/phantom/usr/python39 -type d -exec chmod g+rx,o+rx {} \; sudo find /opt/phantom/usr/python39 -type f -exec chmod g+r,o+r {} \; ``` # 5.2.1 4/2022 Must follow the upgrade path. # 5.1.0 1/2022 To allow Phantom to run on a system without IPv6 enabled, the /etc/nginx/nginx.conf file needs to be edited and line 40 (listen [::]:80; ) needs to be commented out. This allows nginx to start and Phantom to work again. Splunk case number: 2847652 # 4.10.6 08/2021 minor upgrade to upgrade Nginx due to Vuln scanner. Also removes use of TLSv1.1 # 4.10.4 05/2021 minor upgrade due to known issue with pgbouncer and okta auth. Troubleshooting ISSUE: Phantom webpage does not load and shows "internal server error" ( See Splunk Support ticket) RESOLUTION: check permissions on /tmp/uwsgi_invalidate_ss_cache_trigger and ensure they are 666. Then restart uwsgi with `/opt/phantom/bin/phsvc restart uwsgi` ( if needed try this ) In: "/opt/phantom/usr/python36/lib/python3.6/site-packages/django/apps/ registry.py (https://registry.py) " The line: 'raise RuntimeError("populate() isn't reentrant")' Should be changed to: 'self.app_configs = {}' # 4.10.3 05/2021 Follow Splunk Docs! Switched XDR from offline RPM install to Phantom repo install I had to upgrade to latest version in 4.9 before upgrading to 4.10 Use tmux to avoid SSH timeout during upgrade? # 4.9 08/2020 ## Prep Work See Splunk docs! Silence Phantom Sensu checks Stop Phantom `/opt/phantom/bin/stop_phantom.sh` Clean yum `yum clean all` Take an AWS snapshot in addition to the automatic snapshots! should be for a 500 GB volume Naming Scheme: phantom-pre-upgrade-backup- Run a backup! `sudo phenv python ibackup.pyc --backup` Update OS & reboot (only if kernel updated) `yum update --exclude=nginx` Start Phantom `/opt/phantom/bin/start_phantom.sh` Disable WAL `sed -i -e 's/archive_mode = on/archive_mode = off/i' /opt/phantom/data/db/postgresql.phantom.conf` restart postgres ``` # 2021-04-12: While troubleshooting a problem, noticed we're on postgres11 now. /opt/phantom/bin/phsvc restart postgresql-11 ``` Install new repo and keys `rpm -Uvh https://repo.phantom.us/phantom/4.9/base/7Server/x86_64/phantom_repo-4.9.35731-1.x86_64.rpm` Centos7 (Caasp) `rpm -Uvh https://repo.phantom.us/phantom/4.10/base/7/x86_64/phantom_repo-4.10.3.51237-1.x86_64.rpm` Troubleshooting Error: Error - Phantom requires that the user 'phantom' has access to cron. Solution: `vim /etc/cron.allow` and add phantom Error! It looks like you don't have enough space in your `/tmp directory` Your `/tmp directory` must have a capacity of at least `5GB` If you would like to ignore this check, please re-run with the option `--no-space-check` ## Upgrade Upgrade script `/opt/phantom/bin/phantom_setup.sh upgrade` Post Upgrade (Run IF the upgrade script produces the message!) `su - postgres -c '/usr/pgsql-11/bin/vacuumdb -h /tmp --all --analyze-in-stages'` Run this to re-setup or backups `phenv python3 /opt/phantom/bin/ibackup.pyc --setup` Verify postgres version `su - postgres -c '/usr/pgsql-11/bin/postgres --version'` Login into web to accept EULA Administration > Product Settings > Telemetry > OFF Post Upgrade Steps 1. Review System Health 1. Administration -> System Health -> System Health Have Phantom Administrator verify that email is working properly. Clear Silence Done! # 4.8 ## Vagrant VM Upgrade vagrant phantom creds admin/password Password1 ssh use the brad user and ssh key ## Test Upgrade TEST 1. Make snapshot ## Prod Upgrade PROD stop phantom take snapshot of drive clean yum cache install RPM for repo upgrade phantom Phantom Upgrade Steps Do not skip versions. Upgrade incrementally. 1. Take a snapshot of the server 2. Stop all services 2.1 /opt/phantom/bin/stop_phantom.sh 3. Clear yum caches 3.1 yum clean all 4. update the OS 4.1 yum update --exclude=nginx 5. reboot if kernel was upgraded 5.1 reboot 6. after reboot login and installed the phantom repo for the correct version of the software. 6.1 [Splunk Phantom repositories and signing keys packages](https://docs.splunk.com/Documentation/Phantom/4.8/Install/PhantomReposAndSigningKeys) 6.2 `rpm -Uvh https://repo.phantom.us/phantom/4.6/base/7Server/x86_64/phantom_repo-4.6.19142-1.x86_64.rpm` 6.3 `/opt/phantom/bin/phantom_setup.sh upgrade` Post Upgrade Steps 1. Review System Health 1. Administration -> System Health -> System Health