infrastructure-notes/Phantom Notes.md

Phantom Notes

Stop and Start the services

/opt/phantom/bin/stop_phantom.sh
/opt/phantom/bin/start_phantom.sh

Postgres log location /opt/phantom/data/db/pg_log Phantom nginx log location /var/log/nginx Phantom log location /var/log/phantom

Restart just pgbouncer systemctl restart pgbouncer

How Do I View Moose Events in Phantom?

Drop down > Cases > Filter on TENANT > MOOSE

How Do I promote an Analyst to Shift Lead?

When the user is already in Splunk SOAR, making the change in OKTA should be sufficent. If Splunk SOAR hates you it might not be enough.

psql -d phantom -h /tmp (as the postgres user)

phantom=# select c.is_active, c.username, a.name from role a join role_users b on a.id=b.role_id join ph_user c on b.phuser_id=c.id where c.username='duane.waddle' and c.is_active='t';
is_active | username | name
-----------+--------------+-----------------
t | duane.waddle | Administrator
t | duane.waddle | MDR Shift Leads
(2 rows)

Installation in unprivledged + FIPS mode

08/2022

FOLLOW the steps in the DOCS!

Ensure FIPS is enable

cat /proc/sys/crypto/fips_enabled

Why use /opt/phantom? So i don't have to change the notes and i am afraid of hidden errors where splunk team forgot to change the code and it still has a static folder reference to /opt/phantom.

cd /opt Download the .tar.gz file to /opt/ (it has enough room for the large size) Extract .tar

as your XDR user run sudo ./splunk-soar/soar-prepare-system --no-spinners --splunk-soar-home /opt/phantom --https-port 8443

Yes download packages in the the default repos No GlusterFS we are not using external file share No ntpd ( we are using chronyd ) Yes to basic firewall No to https redirect ( let the LB do that) Yes to create non-priviledged user Yes to create password for phantom user Yes to adjust file descriptor limits (/etc/security/limits.d/25-phantom-limits.conf)

copy the .tar, change it's owner to phantom and extract it again in /opt/phantom as hte phantom user.

Give phantom cron permissions

chmod +w /etc/cron.allow
vim /etc/cron.allow
chmod -w /etc/cron.allow

tmux
splunk-soar/soar-install --splunk-soar-home /opt/phantom --https-port 8443 --ignore-warnings

ignore warnings about space issues if in TEST.

External DB setup Prep

Follow docs associated with AWS RDS. These commands can be run as root.

/opt/phantom/usr/postgresql/bin/psql --host splunk-soar.c5cmrfedysrr.us-gov-east-1.rds.amazonaws.com --port 5432 --username postgres --echo-all --dbname phantom --command "CREATE ROLE pgbouncer WITH PASSWORD '[input-password-here]' login;"

/opt/phantom/usr/postgresql/bin/psql --host splunk-soar.c5cmrfedysrr.us-gov-east-1.rds.amazonaws.com --port 5432 --username postgres --echo-all --dbname phantom --command "GRANT rds_superuser TO pgbouncer;"

If the postgres user has issues, reset the master password to match the password generated by TF and put it in Vault.

Backup and restore database to AWS RDS

Continue following the Docs!

tmux # just in case
su phantom
cd /opt/phantom
/opt/phantom/bin/phenv ibackup --setup # not sure this is needed.....

cd /opt/phantom/bin/
./phenv python backup.pyc --all

• Look for the .tgz file.
• Edit the pbbouncer.database.ini file per the docs
• phenv prepare_db --db-exist-ok # add the missing flag for existing DB
• ./phenv python restore.pyc --file /opt/phantom/data/phantom_backups/phantom_backup_2022-09-09-23-20-26.tgz

Splunk Soar Application Configuration

Fill out the company info Administration > Company Settings

Fill out/verify the proxy information. Administration settings > Environment Settings

TEST no_proxy 127.0.0.1,localhost HTTP_PROXY http://proxy.pvt.xdrtest.accenturefederalcyber.com:80 http_proxy http://proxy.pvt.xdrtest.accenturefederalcyber.com:80 HTTPS_PROXY http://proxy.pvt.xdrtest.accenturefederalcyber.com:80 https_proxy http://proxy.pvt.xdrtest.accenturefederalcyber.com:80

PROD -TODO-

Configure SAML Auth. Start In Okta first. Single Sign On URL: https://splunk-soar.pvt.xdrtest.accenturefederalcyber.com/saml2/callback/login Recipient URL and Dest URL are the same Audience Restriction: https://phantom.us/saml2/service_provider ... See Okta for other configs. Save the config and view the SAML setup instructions by viewing Sign On > SAML setup instructions.

In Splunk Soar: Administration > User management > Authentication > SAML2 Provider Name: Okta Single-sign-on URL: Obtain from Saml Setup Instructions in the Okta Splunk SOAR app Issuer ID: Obtain from Saml Setup Instructions in the Okta Splunk SOAR app Splunk SOAR Base URL: https://splunk-soar.pvt.xdrtest.accenturefederalcyber.com Metadata XML: Obtain from Saml Setup Instructions in the Okta Splunk SOAR app Advanced: Group Mapping phantom-role-administrator -> Administrator ... add others

External Attribute username -> username email -> email firstName -> first_name lastName -> last_name

Troubleshooting ERROR: SAML2 Authentication Error User is not authorized to use this application. SOLUTION: You did not setup Group mapping in Advanced

Phantom pgbouncer Issue (legacy)

[gc-prod]root@phantom-0:~:# ps -ef | grep pgbouncer | wc -l
96

/var/log/pgbouncer/pgbouncer.log

2021-06-03 02:18:20.981 UTC [3034] WARNING C-0x7f66adca0ae8: (nodb)/(nouser)@unix(11235):6432 pooler error: no more connections allowed (max_client_conn)

/var/log/phantom/wsgi.log

/var/log/phantom/wsgi.log.4:psycopg2.OperationalError: ERROR: no more connections allowed (max_client_conn)

there's a config file, /etc/pgbouncer/pgbouncer.ini. I bumped some limits in there last night from max connections = 750 to max connections = 2000

[gc-prod]root@phantom-0:~:# egrep "750|2000" /etc/pgbouncer/pgbouncer.ini
;max_client_conn = 750
max_client_conn = 2000
;default_pool_size = 750
default_pool_size = 2000
;max_db_connections = 750
;max_user_connections = 750
max_db_connections = 2000
max_user_connections = 2000

Salesforce app needs outbound 443

When setting up a new "asset" (salesforce instance), Greg has to go through a "Connectivity Test" that uses oAuth. This doesn't very well support our outbound proxy.

When he's doing this (and it's only needed during the setup / test) go into AWS console in legacy-mdr-prod and update sg-04de5c2a4a.

Add an outbound rule to 0.0.0.0/0 port 443.

Remove it when he's done.

TLS version 1.1 Vuln

Phantom (v4.9) is allowing TLS version 1.1. This is a Qualys finding.

openssl s_client -connect 10.80.101.221:443 -tls1_1

grep ssl_protocols /etc/nginx/conf.d/default.conf

2021-04-21 Backup Issue - FTD

While trying to migrate to govcloud, backups were unable to be taken.

$ sudo phenv python3 /opt/phantom/bin/ibackup.pyc --setup
[pid: 26829] [12/Apr/2021 16:30:09]   ibackup.py:293  INFO: Running ibackup.pyc - details will be logged to /var/log/phantom/backup/ibackup_2021-04-12T16:30:09.231947Z.log
Setup will temporarily stop phantom
If you wish to continue, enter yes to proceed: yes
[pid: 26829] [12/Apr/2021 16:30:12]    phproc.py:146  WARNING: unable to open log file '/var/log/phantom/backup/phantom-stanza-create.log': Permission denied
      NOTE: process will continue without log file.

[pid: 26829] [12/Apr/2021 16:31:14]    phproc.py:146  WARNING: ERROR [082]: : could not find WAL segment 00000001000000EC00000054 after 60 second(s)
HINT: is archive_command configured correctly?
HINT: use the check command to verify that PostgreSQL is archiving.
ERROR [082]: : could not find WAL segment 00000001000000EC00000054 after 60 second(s)
HINT: is archive_command configured correctly?
HINT: use the check command to verify that PostgreSQL is archiving.
WAL segment 00000001000000EC00000054 did not reach the archive:11-1
HINT: Check the archive_command to ensure that all options are correct (especially --stanza).
HINT: Check the PostgreSQL server log for errors.

Traceback (most recent call last):
  File "../setup/ibackup.py", line 377, in <module>
  File "../setup/ibackup.py", line 319, in main
  File "../pycommon/phantom_common/backup/backup_manager.py", line 1204, in setup
  File "../pycommon/phantom_common/backup/pgbackrest.py", line 576, in setup
  File "../pycommon/phantom_common/backup/pgbackrest.py", line 607, in create
  File "../pycommon/phantom_common/backup/pgbackrest.py", line 706, in _run_pgbackrest_cmd
  File "../pycommon/phantom_common/phproc.py", line 249, in run_command
  File "../pycommon/phantom_common/phproc.py", line 157, in communicate
phantom_common.phproc.PhCalledProcessError: Command 'pgbackrest --stanza=phantom --config=/opt/phantom/etc/pgbackrest.conf --log-level-console=info --log-level-file=info check' returned non-zero exit status 82.
Output: 2021-04-12 16:30:12.084 P00   INFO: check command begin 2.15: --config=/opt/phantom/etc/pgbackrest.conf --log-level-console=info --log-level-file=info --log-path=/var/log/phantom/backup --pg1-path=/opt/phantom/data/db --pg1-socket-path=/tmp --repo1-path=/opt/phantom/data/ibackup/repo/pg --stanza=phantom
2021-04-12 16:31:14.140 P00   INFO: check command end: aborted with exception [082]

Error output: ERROR [082]: : could not find WAL segment 00000001000000EC00000054 after 60 second(s)
HINT: is archive_command configured correctly?
HINT: use the check command to verify that PostgreSQL is archiving.
ERROR [082]: : could not find WAL segment 00000001000000EC00000054 after 60 second(s)
HINT: is archive_command configured correctly?
HINT: use the check command to verify that PostgreSQL is archiving.
WARN: WAL segment 00000001000000EC00000054 did not reach the archive:11-1
HINT: Check the archive_command to ensure that all options are correct (especially --stanza).
HINT: Check the PostgreSQL server log for errors.

logfile /var/log/phantom/backup/ibackup_2021-04-12T16:03:51.468153Z.log:

$ sudo cat /var/log/phantom/backup/ibackup_2021-04-12T16:03:51.468153Z.log
[pid:  8104] [12/Apr/2021 16:03:51]   ibackup.py:288  DEBUG: Command: /opt/phantom/bin/ibackup.pyc --setup
[pid:  8104] [12/Apr/2021 16:03:51]   ibackup.py:289  DEBUG: Initializing BackupManager
[pid:  8104] [12/Apr/2021 16:03:51]   ibackup.py:293  INFO: Running ibackup.pyc - details will be logged to /var/log/phantom/backup/ibackup_2021-04-12T16:03:51.468153Z.log
[pid:  8104] [12/Apr/2021 16:03:54] backup_manager.py:1177 INFO: Exiting setup

FIX:

chown -R postgres: /opt/phantom/data/ibackup

Migration to GovCloud

Prep / Installation Notes

1. Stand it up

   cd ~/xdr-terraform-live/test/aws-us-gov/mdr-test-c2/250-phantom
   terragrunt apply
   

2. Highstate it

   ssh gc-dev-salt-master
   salt 'phantom-0.pvt.xdr.accenturefederalcyber.com' state.highstate --output-diff; salt 'phantom-0.pvt.xdr.accenturefederalcyber.com' state.highstate --output-diff;
   salt 'phantom-0.pvt.xdr.accenturefederalcyber.com' pkg.upgrade
   exit
   

3. Disable FIPS

   ssh gc-dev-phantom-0
   sudo yum remove dracut-fips*
   sudo cp -p /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).backup.beforeremovingfips
   sudo dracut -f
   sudo vim /etc/default/grub
   # Change "fips=1" to "fips=0"
   sudo grub2-mkconfig -o /boot/grub2/grub.cfg
   sudo shutdown -r now
   cat /proc/sys/crypto/fips_enabled
   

4. Enable the optionals repo:

   sudo vim /etc/yum.repos.d/redhat-rhui.repo
   # Find rhel-7-server-rhui-optional-rpms and change 'enabled' to 1
   sudo yum update
   

   Add phantom user to cron allow ( see salt/cis/rhel7-3_1_1/parameters/id/phantom-0.pvt.xdr.accenturefederalcyber.com.yaml in xdr-cis-benchmarks git repo. ) vim /etc/cron.allow # and add phantom

5. Install the installer

NOTE: To install a particular version, you have to use the offline installer steps, available here: https://docs.splunk.com/Documentation/Phantom/4.10.2/Install/InstallOffline NOTE: See BRAD's WAY below.

ssh dev-phantom
# Find the current version by sudo yum list installed | grep phantom.x86
sudo yum list installed | grep phantom.x86

# dev
VERSION=4.9.37880
# prod
VERSION=4.9.35731
wget https://download.splunk.com/products/phantom/release/linux/${VERSION}/phantom_offline_setup_rhel7-${VERSION}.tgz
sudo mkdir -p /usr/local/src/upgrade-${VERSION}
sudo chmod 755 /usr/local/src/upgrade-${VERSION}
cd /usr/local/src/upgrade-${VERSION}
sudo tar xvzf ~/phantom_offline_setup_rhel7-${VERSION}.tgz
cd phantom_offline_setup_rhel7-${VERSION}

sudo ./phantom_offline_setup_rhel.sh install
# answer 'y'

BRAD's WAY: Don't use the offline installer, just use the specific version RPM and install with a specific version. /opt/phantom/bin/phantom_setup.sh install --version=4.10.3.51237-1 --no-space-check

If you're installing, you're good. If you're migrating, continue:

1. Enable cross-system ssh

   ssh gc-dev-phantom-0
   ssh-keygen
   cat ~/.ssh/id_rsa.pub
   exit
   ssh dev-phantom
   mkdir .ssh
   cat > .ssh/authorized_keys
   # paste from above, then ctrl-d
   exit
   ssh gc-dev-phantom-0
   ssh phantom.msoc.defpoint.local
   # validate that you can log in
   

2. Run Initial Backup

   ssh dev-phantom
   time sudo phenv python3 /opt/phantom/bin/ibackup.pyc --backup
   sudo ls -l /opt/phantom/data/backup/
   

3. Copy to new system

   ssh gc-dev-phantom-0
   sudo mkdir -p /opt/phantom/data/restore
   # copy only changed files
   time sudo rsync -r --progress \
   -e "ssh -i /home/frederick_t_damstra/.ssh/id_rsa" \
   --rsync-path="sudo rsync" \
   frederick_t_damstra@phantom.msoc.defpoint.local:/opt/phantom/data/backup/ \
   /opt/phantom/data/restore/
   sudo chown -R postgres:postgres /opt/phantom/data/backup /opt/phantom/data/restore
   sudo ls -l /opt/phantom/data/restore
   

4. Prep new system for restore

   # setup backups (required for restore)
   # This will fail the first time, but it has to be done
   sudo phenv python3 /opt/phantom/bin/ibackup.pyc --setup
   
   # fix errors
   sudo chown -R postgres: /opt/phantom/data/ibackup # Fixes WAL error
   sudo chmod 644 /opt/phantom/etc/pgbackrest.conf # secodn fix for WAL error
   sudo find /opt/phantom/data/ -type d -exec chmod o+rx {} \;
   sudo find /opt/phantom/data/db -type d -exec chmod o-rx {} \;
   
   # Disable WAL
   sudo vim /opt/phantom/data/db/postgresql.phantom.conf
   # change 'archive_mode' to 'off'
   
   # restart postgres
   sudo /opt/phantom/bin/phsvc restart postgresql-11
   
   # setup backups (required for restore) - should work this time
   sudo phenv python3 /opt/phantom/bin/ibackup.pyc --setup
   

Final cutover

1. Stop phantom and create the last backup

   ssh dev-phantom
   time sudo phenv python3 /opt/phantom/bin/ibackup.pyc --backup
   sudo ls -l /opt/phantom/data/backup/
   sudo /opt/phantom/bin/stop_phantom.sh
   sudo systemctl disable phantom_watchdogd
   exit
   

2. Copy the backup across

   ssh gc-dev-phantom-0
   time sudo rsync -r --progress \
   -e "ssh -i /home/frederick_t_damstra/.ssh/id_rsa" \
   --rsync-path="sudo rsync" \
   frederick_t_damstra@phantom.msoc.defpoint.local:/opt/phantom/data/backup/ \
   /opt/phantom/data/restore/
   sudo chown -R postgres:postgres /opt/phantom/data/backup /opt/phantom/data/restore
   sudo ls -l /opt/phantom/data/restore
   

3. Restore the backup

   cd /opt/phantom/bin/
   sudo ls -l /opt/phantom/data/restore/
   # Specify the latest backup file:
   time sudo phenv python3 /opt/phantom/bin/ibackup.pyc --restore /opt/phantom/data/restore/TODO
   

4. Reset Root PW

   # Update the admin pw:
   sudo bash
   cd /opt/phantom/www
   phenv python3 manage.py changepassword admin
   # set password
   

5. Restart phantom

   sudo /opt/phantom/bin/stop_phantom.sh
   sudo /opt/phantom/bin/start_phantom.sh
   

6. Fix settings:

login to the website, go to administration->app settings, update the proxy to http://proxy.pvt.xdrtest.accenturefederalcyber.com:80 and click save changes administration->user management->authentication->saml2 Record original values:

    SSO Url: https://mdr-multipass.okta.com/app/mdrmultipass_mdrphantom_1/exk1m6x7ri1WgvXCB297/sso/saml
    New URL: 
    Issuer ID: http://www.okta.com/exk1m6x7ri1WgvXCB297
    New ID   : 
    Base URL: https://phantom.msoc.defpoint.local
    New URL:  
    Metadata:
<?xml version="1.0" encoding="UTF-8"?><md:EntityDescriptor xmlns:md="urn:oasis:names:tc:SAML:2.0:metadata" entityID="http://www.okta.com/exk1m6x7ri1WgvXCB297"><md:IDPSSODescriptor WantAuthnRequestsSigned="false" protocolSupportEnumeration="urn:oasis:names:tc:SAML:2.0:protocol"><md:KeyDescriptor use="signing"><ds:KeyInfo xmlns:ds="http://www.w3.org/2000/09/xmldsig#"><ds:X509Data><ds:X509Certificate>MIIDqjCCApKgAwIBAgIGAWrbB00GMA0GCSqGSIb3DQEBCwUAMIGVMQswCQYDVQQGEwJVUzETMBEG
A1UECAwKQ2FsaWZvcm5pYTEWMBQGA1UEBwwNU2FuIEZyYW5jaXNjbzENMAsGA1UECgwET2t0YTEU
MBIGA1UECwwLU1NPUHJvdmlkZXIxFjAUBgNVBAMMDW1kci1tdWx0aXBhc3MxHDAaBgkqhkiG9w0B
CQEWDWluZm9Ab2t0YS5jb20wHhcNMTkwNTIxMTUzMzA5WhcNMjkwNTIxMTUzNDA5WjCBlTELMAkG
A1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExFjAUBgNVBAcMDVNhbiBGcmFuY2lzY28xDTAL
BgNVBAoMBE9rdGExFDASBgNVBAsMC1NTT1Byb3ZpZGVyMRYwFAYDVQQDDA1tZHItbXVsdGlwYXNz
MRwwGgYJKoZIhvcNAQkBFg1pbmZvQG9rdGEuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
CgKCAQEAjVWbGnlG3G858/K0b8jVw5OFAef+eFWNmjD6eAfGMgOzQ3ZhJmZ5TAFxaUaH15Q7Vi10
p/zKHo8rZAurh31r35ED9JT+45J/IsDtOUK55quSEeh4d0Ih7NTBXgP5yEsSa7YVqBL4mI450JRr
8BTTfatUP0/TRxSx92QxlNhLi0jYmGtgzQ/3TeTEWIzZKntTkX7Arn42Dt7JkCdI+ElEfcNQYV3l
//Olv0TEVFasbmIb8iNgVOi+ssq5UyqAjoWYJOc2VvkerUE9FDs7DkC3S1/sXR72vpTfXpz1fW+x
/aHJjgwXgB2SW9fZk8CQjqEI5s6QCMBsHSOhU+xDkbzAnwIDAQABMA0GCSqGSIb3DQEBCwUAA4IB
AQCKqio8wrvhbkGRptCD6sEnRmC7/NBE133tIv7Z3R/Cve8DgO3GcKKrCUh+gZJLFV3eWw95FTWW
MY7KrYEd353mKP8hL7mEc+qSmWuwfFw+6JePHsNDiFKCY2PfzbWgsG9nX7T6H7n8cn2hzVn4gBmb
8TAXei+x0id9h24oSvtISZhMg+ED72c0BbO4wPZOQeisXPO4vugdRdbyB5wvIU2ILHb7WJnDNSai
XSHqKUBigvQua2KSjh+GW7fMlvRbDkYxq3okj6sZlyCLN79IM4NZgKfCC4t8FoUA9ofIDUV9u70G
+Utb6eeVogPzFlv4LuMRAEKbnV9G3yyDbxYsEcpY</ds:X509Certificate></ds:X509Data></ds:KeyInfo></md:KeyDescriptor><md:NameIDFormat>urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress</md:NameIDFormat><md:NameIDFormat>urn:oasis:names:tc:SAML:2.0:nameid-format:persistent</md:NameIDFormat><md:SingleSignOnService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST" Location="https://mdr-multipass.okta.com/app/mdrmultipass_mdrphantom_1/exk1m6x7ri1WgvXCB297/sso/saml"/><md:SingleSignOnService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect" Location="https://mdr-multipass.okta.com/app/mdrmultipass_mdrphantom_1/exk1m6x7ri1WgvXCB297/sso/saml"/></md:IDPSSODescriptor></md:EntityDescriptor>

Update saml with settings from the saml provider metadata (available from okta, application, login settings)

Log out and log back in via okta.

Run the backup prep.

---

I got 500: Server Error.

Things I did: tried accepting eula at https://phantom.pvt.xdrtest.accenturefederalcyber.com/eula/ Double checked saml config Set hostname and fqdn in administration->company settings

1. Start phantom:

   sudo /opt/phantom/bin/stop_phantom.sh
   sudo /opt/phantom/bin/start_phantom.sh