首頁 探索 說明
註冊 登入
AFS
/
infrastructure-notes
2
0
複刻 0
檔案 問題管理 0 合併請求 0 Wiki
目錄樹: c66d2e04ba
分支列表 標籤列表
infrastructure-...
/
Phantom Notes.md

Phantom Notes.md 14 KB
文件歷史 原始文件

Phantom Notes

Stop and Start the services /opt/phantom/bin/stop_phantom.sh /opt/phantom/bin/start_phantom.sh

postgres log location /opt/phantom/data/db/pg_log

Restart just pgbouncer systemctl restart pgbouncer

How Do I View Moose Events in Phantom?

Drop down > Cases > Filter on TENANT > MOOSE

Phantom pgbouncer Issue (legacy)

[gc-prod]root@phantom-0:~:# ps -ef | grep pgbouncer | wc -l
96

/var/log/pgbouncer/pgbouncer.log

2021-06-03 02:18:20.981 UTC [3034] WARNING C-0x7f66adca0ae8: (nodb)/(nouser)@unix(11235):6432 pooler error: no more connections allowed (max_client_conn)

/var/log/phantom/wsgi.log

/var/log/phantom/wsgi.log.4:psycopg2.OperationalError: ERROR: no more connections allowed (max_client_conn)

there's a config file, /etc/pgbouncer/pgbouncer.ini. I bumped some limits in there last night from max connections = 750 to max connections = 2000

[gc-prod]root@phantom-0:~:# egrep "750|2000" /etc/pgbouncer/pgbouncer.ini
;max_client_conn = 750
max_client_conn = 2000
;default_pool_size = 750
default_pool_size = 2000
;max_db_connections = 750
;max_user_connections = 750
max_db_connections = 2000
max_user_connections = 2000

Salesforce app needs outbound 443

When setting up a new "asset" (salesforce instance), Greg has to go through a "Connectivity Test" that uses oAuth. This doesn't very well support our outbound proxy.

When he's doing this (and it's only needed during the setup / test) go into AWS console in legacy-mdr-prod and update sg-04de5c2a4a.

Add an outbound rule to 0.0.0.0/0 port 443.

Remove it when he's done.

TLS version 1.1 Vuln

Phantom (v4.9) is allowing TLS version 1.1. This is a Qualys finding.

openssl s_client -connect 10.80.101.221:443 -tls1_1

grep ssl_protocols /etc/nginx/conf.d/default.conf

2021-04-21 Backup Issue - FTD

While trying to migrate to govcloud, backups were unable to be taken.

$ sudo phenv python3 /opt/phantom/bin/ibackup.pyc --setup
[pid: 26829] [12/Apr/2021 16:30:09]   ibackup.py:293  INFO: Running ibackup.pyc - details will be logged to /var/log/phantom/backup/ibackup_2021-04-12T16:30:09.231947Z.log
Setup will temporarily stop phantom
If you wish to continue, enter yes to proceed: yes
[pid: 26829] [12/Apr/2021 16:30:12]    phproc.py:146  WARNING: unable to open log file '/var/log/phantom/backup/phantom-stanza-create.log': Permission denied
      NOTE: process will continue without log file.

[pid: 26829] [12/Apr/2021 16:31:14]    phproc.py:146  WARNING: ERROR [082]: : could not find WAL segment 00000001000000EC00000054 after 60 second(s)
HINT: is archive_command configured correctly?
HINT: use the check command to verify that PostgreSQL is archiving.
ERROR [082]: : could not find WAL segment 00000001000000EC00000054 after 60 second(s)
HINT: is archive_command configured correctly?
HINT: use the check command to verify that PostgreSQL is archiving.
WAL segment 00000001000000EC00000054 did not reach the archive:11-1
HINT: Check the archive_command to ensure that all options are correct (especially --stanza).
HINT: Check the PostgreSQL server log for errors.

Traceback (most recent call last):
  File "../setup/ibackup.py", line 377, in <module>
  File "../setup/ibackup.py", line 319, in main
  File "../pycommon/phantom_common/backup/backup_manager.py", line 1204, in setup
  File "../pycommon/phantom_common/backup/pgbackrest.py", line 576, in setup
  File "../pycommon/phantom_common/backup/pgbackrest.py", line 607, in create
  File "../pycommon/phantom_common/backup/pgbackrest.py", line 706, in _run_pgbackrest_cmd
  File "../pycommon/phantom_common/phproc.py", line 249, in run_command
  File "../pycommon/phantom_common/phproc.py", line 157, in communicate
phantom_common.phproc.PhCalledProcessError: Command 'pgbackrest --stanza=phantom --config=/opt/phantom/etc/pgbackrest.conf --log-level-console=info --log-level-file=info check' returned non-zero exit status 82.
Output: 2021-04-12 16:30:12.084 P00   INFO: check command begin 2.15: --config=/opt/phantom/etc/pgbackrest.conf --log-level-console=info --log-level-file=info --log-path=/var/log/phantom/backup --pg1-path=/opt/phantom/data/db --pg1-socket-path=/tmp --repo1-path=/opt/phantom/data/ibackup/repo/pg --stanza=phantom
2021-04-12 16:31:14.140 P00   INFO: check command end: aborted with exception [082]

Error output: ERROR [082]: : could not find WAL segment 00000001000000EC00000054 after 60 second(s)
HINT: is archive_command configured correctly?
HINT: use the check command to verify that PostgreSQL is archiving.
ERROR [082]: : could not find WAL segment 00000001000000EC00000054 after 60 second(s)
HINT: is archive_command configured correctly?
HINT: use the check command to verify that PostgreSQL is archiving.
WARN: WAL segment 00000001000000EC00000054 did not reach the archive:11-1
HINT: Check the archive_command to ensure that all options are correct (especially --stanza).
HINT: Check the PostgreSQL server log for errors.

logfile /var/log/phantom/backup/ibackup_2021-04-12T16:03:51.468153Z.log:

$ sudo cat /var/log/phantom/backup/ibackup_2021-04-12T16:03:51.468153Z.log
[pid:  8104] [12/Apr/2021 16:03:51]   ibackup.py:288  DEBUG: Command: /opt/phantom/bin/ibackup.pyc --setup
[pid:  8104] [12/Apr/2021 16:03:51]   ibackup.py:289  DEBUG: Initializing BackupManager
[pid:  8104] [12/Apr/2021 16:03:51]   ibackup.py:293  INFO: Running ibackup.pyc - details will be logged to /var/log/phantom/backup/ibackup_2021-04-12T16:03:51.468153Z.log
[pid:  8104] [12/Apr/2021 16:03:54] backup_manager.py:1177 INFO: Exiting setup

FIX:

chown -R postgres: /opt/phantom/data/ibackup

Migration to GovCloud

Prep / Installation Notes

  1. Stand it up

    cd ~/xdr-terraform-live/test/aws-us-gov/mdr-test-c2/250-phantom
terragrunt apply

  2. Highstate it

    ssh gc-dev-salt-master
salt 'phantom-0.pvt.xdr.accenturefederalcyber.com' state.highstate --output-diff; salt 'phantom-0.pvt.xdr.accenturefederalcyber.com' state.highstate --output-diff;
salt 'phantom-0.pvt.xdr.accenturefederalcyber.com' pkg.upgrade
exit

  3. Disable FIPS

    ssh gc-dev-phantom-0
sudo yum remove dracut-fips*
sudo cp -p /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).backup.beforeremovingfips
sudo dracut -f
sudo vim /etc/default/grub
# Change "fips=1" to "fips=0"
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
sudo shutdown -r now
cat /proc/sys/crypto/fips_enabled

  4. Enable the optionals repo:

    sudo vim /etc/yum.repos.d/redhat-rhui.repo
# Find rhel-7-server-rhui-optional-rpms and change 'enabled' to 1
sudo yum update

    Add phantom user to cron allow vim /etc/cron.allow # and add phantom

  5. Install the installer

NOTE: To install a particular version, you have to use the offline installer steps, available here: https://docs.splunk.com/Documentation/Phantom/4.10.2/Install/InstallOffline NOTE: See BRAD's WAY below.

ssh dev-phantom
# Find the current version by sudo yum list installed | grep phantom.x86
sudo yum list installed | grep phantom.x86

# dev
VERSION=4.9.37880
# prod
VERSION=4.9.35731
wget https://download.splunk.com/products/phantom/release/linux/${VERSION}/phantom_offline_setup_rhel7-${VERSION}.tgz
sudo mkdir -p /usr/local/src/upgrade-${VERSION}
sudo chmod 755 /usr/local/src/upgrade-${VERSION}
cd /usr/local/src/upgrade-${VERSION}
sudo tar xvzf ~/phantom_offline_setup_rhel7-${VERSION}.tgz
cd phantom_offline_setup_rhel7-${VERSION}

sudo ./phantom_offline_setup_rhel.sh install
# answer 'y'

BRAD's WAY: Don't use the offline installer, just use the specific version RPM and install with a specific version. /opt/phantom/bin/phantom_setup.sh install --version=4.10.3.51237-1 --no-space-check

If you're installing, you're good. If you're migrating, continue:

  1. Enable cross-system ssh

    ssh gc-dev-phantom-0
ssh-keygen
cat ~/.ssh/id_rsa.pub
exit
ssh dev-phantom
mkdir .ssh
cat > .ssh/authorized_keys
# paste from above, then ctrl-d
exit
ssh gc-dev-phantom-0
ssh phantom.msoc.defpoint.local
# validate that you can log in

  2. Run Initial Backup

    ssh dev-phantom
time sudo phenv python3 /opt/phantom/bin/ibackup.pyc --backup
sudo ls -l /opt/phantom/data/backup/

  3. Copy to new system

    ssh gc-dev-phantom-0
sudo mkdir -p /opt/phantom/data/restore
# copy only changed files
time sudo rsync -r --progress \
-e "ssh -i /home/frederick_t_damstra/.ssh/id_rsa" \
--rsync-path="sudo rsync" \
frederick_t_damstra@phantom.msoc.defpoint.local:/opt/phantom/data/backup/ \
/opt/phantom/data/restore/
sudo chown -R postgres:postgres /opt/phantom/data/backup /opt/phantom/data/restore
sudo ls -l /opt/phantom/data/restore

  4. Prep new system for restore

    # setup backups (required for restore)
# This will fail the first time, but it has to be done
sudo phenv python3 /opt/phantom/bin/ibackup.pyc --setup

# fix errors
sudo chown -R postgres: /opt/phantom/data/ibackup # Fixes WAL error
sudo chmod 644 /opt/phantom/etc/pgbackrest.conf # secodn fix for WAL error
sudo find /opt/phantom/data/ -type d -exec chmod o+rx {} \;
sudo find /opt/phantom/data/db -type d -exec chmod o-rx {} \;

# Disable WAL
sudo vim /opt/phantom/data/db/postgresql.phantom.conf
# change 'archive_mode' to 'off'

# restart postgres
sudo /opt/phantom/bin/phsvc restart postgresql-11

# setup backups (required for restore) - should work this time
sudo phenv python3 /opt/phantom/bin/ibackup.pyc --setup

Final cutover

  1. Stop phantom and create the last backup

    ssh dev-phantom
time sudo phenv python3 /opt/phantom/bin/ibackup.pyc --backup
sudo ls -l /opt/phantom/data/backup/
sudo /opt/phantom/bin/stop_phantom.sh
sudo systemctl disable phantom_watchdogd
exit

  2. Copy the backup across

    ssh gc-dev-phantom-0
time sudo rsync -r --progress \
-e "ssh -i /home/frederick_t_damstra/.ssh/id_rsa" \
--rsync-path="sudo rsync" \
frederick_t_damstra@phantom.msoc.defpoint.local:/opt/phantom/data/backup/ \
/opt/phantom/data/restore/
sudo chown -R postgres:postgres /opt/phantom/data/backup /opt/phantom/data/restore
sudo ls -l /opt/phantom/data/restore

  3. Restore the backup

    cd /opt/phantom/bin/
sudo ls -l /opt/phantom/data/restore/
# Specify the latest backup file:
time sudo phenv python3 /opt/phantom/bin/ibackup.pyc --restore /opt/phantom/data/restore/TODO

  4. Reset Root PW

    # Update the admin pw:
sudo bash
cd /opt/phantom/www
phenv python3 manage.py changepassword admin
# set password

  5. Restart phantom

    sudo /opt/phantom/bin/stop_phantom.sh
sudo /opt/phantom/bin/start_phantom.sh

  6. Fix settings:

login to the website, go to administration->app settings, update the proxy to http://proxy.pvt.xdrtest.accenturefederalcyber.com:80 and click save changes administration->user management->authentication->saml2 Record original values:

    SSO Url: https://mdr-multipass.okta.com/app/mdrmultipass_mdrphantom_1/exk1m6x7ri1WgvXCB297/sso/saml
    New URL: 
    Issuer ID: http://www.okta.com/exk1m6x7ri1WgvXCB297
    New ID   : 
    Base URL: https://phantom.msoc.defpoint.local
    New URL:  
    Metadata:
<?xml version="1.0" encoding="UTF-8"?><md:EntityDescriptor xmlns:md="urn:oasis:names:tc:SAML:2.0:metadata" entityID="http://www.okta.com/exk1m6x7ri1WgvXCB297"><md:IDPSSODescriptor WantAuthnRequestsSigned="false" protocolSupportEnumeration="urn:oasis:names:tc:SAML:2.0:protocol"><md:KeyDescriptor use="signing"><ds:KeyInfo xmlns:ds="http://www.w3.org/2000/09/xmldsig#"><ds:X509Data><ds:X509Certificate>MIIDqjCCApKgAwIBAgIGAWrbB00GMA0GCSqGSIb3DQEBCwUAMIGVMQswCQYDVQQGEwJVUzETMBEG
A1UECAwKQ2FsaWZvcm5pYTEWMBQGA1UEBwwNU2FuIEZyYW5jaXNjbzENMAsGA1UECgwET2t0YTEU
MBIGA1UECwwLU1NPUHJvdmlkZXIxFjAUBgNVBAMMDW1kci1tdWx0aXBhc3MxHDAaBgkqhkiG9w0B
CQEWDWluZm9Ab2t0YS5jb20wHhcNMTkwNTIxMTUzMzA5WhcNMjkwNTIxMTUzNDA5WjCBlTELMAkG
A1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExFjAUBgNVBAcMDVNhbiBGcmFuY2lzY28xDTAL
BgNVBAoMBE9rdGExFDASBgNVBAsMC1NTT1Byb3ZpZGVyMRYwFAYDVQQDDA1tZHItbXVsdGlwYXNz
MRwwGgYJKoZIhvcNAQkBFg1pbmZvQG9rdGEuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
CgKCAQEAjVWbGnlG3G858/K0b8jVw5OFAef+eFWNmjD6eAfGMgOzQ3ZhJmZ5TAFxaUaH15Q7Vi10
p/zKHo8rZAurh31r35ED9JT+45J/IsDtOUK55quSEeh4d0Ih7NTBXgP5yEsSa7YVqBL4mI450JRr
8BTTfatUP0/TRxSx92QxlNhLi0jYmGtgzQ/3TeTEWIzZKntTkX7Arn42Dt7JkCdI+ElEfcNQYV3l
//Olv0TEVFasbmIb8iNgVOi+ssq5UyqAjoWYJOc2VvkerUE9FDs7DkC3S1/sXR72vpTfXpz1fW+x
/aHJjgwXgB2SW9fZk8CQjqEI5s6QCMBsHSOhU+xDkbzAnwIDAQABMA0GCSqGSIb3DQEBCwUAA4IB
AQCKqio8wrvhbkGRptCD6sEnRmC7/NBE133tIv7Z3R/Cve8DgO3GcKKrCUh+gZJLFV3eWw95FTWW
MY7KrYEd353mKP8hL7mEc+qSmWuwfFw+6JePHsNDiFKCY2PfzbWgsG9nX7T6H7n8cn2hzVn4gBmb
8TAXei+x0id9h24oSvtISZhMg+ED72c0BbO4wPZOQeisXPO4vugdRdbyB5wvIU2ILHb7WJnDNSai
XSHqKUBigvQua2KSjh+GW7fMlvRbDkYxq3okj6sZlyCLN79IM4NZgKfCC4t8FoUA9ofIDUV9u70G
+Utb6eeVogPzFlv4LuMRAEKbnV9G3yyDbxYsEcpY</ds:X509Certificate></ds:X509Data></ds:KeyInfo></md:KeyDescriptor><md:NameIDFormat>urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress</md:NameIDFormat><md:NameIDFormat>urn:oasis:names:tc:SAML:2.0:nameid-format:persistent</md:NameIDFormat><md:SingleSignOnService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST" Location="https://mdr-multipass.okta.com/app/mdrmultipass_mdrphantom_1/exk1m6x7ri1WgvXCB297/sso/saml"/><md:SingleSignOnService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect" Location="https://mdr-multipass.okta.com/app/mdrmultipass_mdrphantom_1/exk1m6x7ri1WgvXCB297/sso/saml"/></md:IDPSSODescriptor></md:EntityDescriptor>

Update saml with settings from the saml provider metadata (available from okta, application, login settings)

Log out and log back in via okta.

Run the backup prep.

I got 500: Server Error.

Things I did: tried accepting eula at https://phantom.pvt.xdrtest.accenturefederalcyber.com/eula/ Double checked saml config Set hostname and fqdn in administration->company settings

  1. Start phantom:

    sudo /opt/phantom/bin/stop_phantom.sh
sudo /opt/phantom/bin/start_phantom.sh