infrastructure-notes/Splunk Migration from Commercial to GovCloud - 2. Search Head.md

Splunk Migration from Commercial to GovCloud

Migrate the SH (This needs to be fleshed out to deal with the name change)

Terraform and highstate a new search head

cd ~/xdr-terraform-live/test/aws-us-gov/mdr-test-c2/170-splunk-searchhead
tfswitch
terragrunt apply
sleep 300
ssh gc-dev-salt-master
salt 'moose-splunk-sh.pvt.xdrtest.accenturefederalcyber.com' state.highstate --output-diff

*** FYI: It became kind of a mess during the sh migration. Expect to have to fix some things from these docs ***

pre-rsync legacy to local:

cd
mkdir tmp/moose-sh
# this can be run multiple times without issue. You may wish to do 
# it first before you've stopped splunk to minimize the interruption.
time rsync --rsync-path="sudo rsync" -avz --delete --progress prod-moose-splunk-sh:/opt/splunk tmp/moose-sh  \
  --exclude '*.log' \
  --exclude '*.log.*' \
  --exclude '*.bundle'

Excluding directories seems to be a recipe for trouble. But if you really want to try, you can add the following:

  --exclude 'splunk/etc/system/default/' \
  --exclude 'splunk/etc/system/bin/' \
  --exclude 'splunk/etc/system/static/' \
  --exclude 'splunk/etc/system/README' \
  --exclude 'splunk/var/log/' \
  --exclude 'splunk/var/run/*.bundle' \
  --exclude 'splunk/var/run/dispatch/' \
  --exclude 'splunk/bin/'

Stop new SH and copy files across:

ssh gc-prod-moose-splunk-sh
sudo systemctl stop splunk
sudo systemctl disable splunk # To prevent accidents, but dont forget to turn it back on!
exit
time rsync --rsync-path="sudo rsync" -avz --delete --progress tmp/moose-sh/splunk/ gc-prod-moose-splunk-sh:/opt/splunk/ \
  --exclude '*.log' \
  --exclude '*.log.*' \
  --exclude '*.bundle'

Post to slack:

The commercial MOOSE Search Head is going down for the transition to GovCloud. I will notify again when the new server is operational.

Final sync from legacy SH:

ssh prod-moose-splunk-sh
sudo systemctl stop splunk
sudo systemctl disable splunk
exit
# Run both rsyncs above

Fix permissions on the new sh

ssh gc-prod-moose-splunk-sh
sudo chown -R splunk:splunk /opt/splunk
# Fix hostnames
sudo sed -i 's/moose-splunk-sh.msoc.defpoint.local/moose-splunk-sh.pvt.xdr.accenturefederalcyber.com/' /opt/splunk/etc/system/local/*.conf
sudo sed -i 's/moose-splunk-sh.msoc.defpoint.local/moose-splunk-sh.pvt.xdr.accenturefederalcyber.com/' /opt/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/inputs.conf
sudo sed -i 's/moose-splunk-sh.msoc.defpoint.local/moose-splunk-sh.pvt.xdr.accenturefederalcyber.com/' /opt/splunk/etc/apps/SA-AFS-ES-KO/local/inputs.conf

Run highstate to fix saml:

ssh gc-prod-salt-master
salt 'moose-splunk-sh.pvt*' state.highstate test=true --output-diff
salt 'moose-splunk-sh.pvt*' state.highstate test=false --output-diff

ssh gc-prod-moose-splunk-sh
sudo systemctl start splunk
sudo systemctl enable splunk

Validate that you can log into https://moose-splunk.pvt.xdr.accenturefederalcyber.com

Post to slack:

The govcloud MOOSE Search Head in govcloud is up. We are commencing testing of functionality and resolving any issues we find. Please let us know if you find anything here and we will resolve them as we are able. Note: The URL has changes. The new url is `https://moose-splunk.pvt.xdr.accenturefederalcyber.com`. In the Okta launch page, it is listed as `Moose Splunk SH [Prod] [GC]`.

Disable okta legacy entries

Update phantom to point to new SH?

???

Post to slack:

We believe all issues related to the migration of the moose search head have been resolved. If you find further issues, please @mention me here, send me an email, or call me at 616-634-4933 if it's critical. Please remember to include as much detail as possible, including steps to reproduce the issue, expected behavior, and actual behavior. Thanks!