infrastructure-notes/Splunk Migration from Commercial to GovCloud - 2. Search Head.md

Splunk Migration from Commercial to GovCloud

Migrate the SH (This needs to be fleshed out to deal with the name change)

Terraform and highstate a new search head

cd ~/xdr-terraform-live/ENV/aws-us-gov/CUST/
cd 170-splunk-searchhead
tfswitch
terragrunt apply
sleep 300
tshp salt-master
salt ${NEWFQDN} state.highstate --output-diff
salt ${NEWFQDN} state.highstate --output-diff
salt ${NEWFQDN} cmd.run 'systemctl stop splunk'
salt ${NEWFQDN} cmd.run 'systemctl disable splunk'
salt ${NEWFQDN} pkg.upgrade
salt ${NEWFQDN} system.reboot

Prep for scp:

# generate key on new
tshp dc-c19-splunk-sh
systemctl stop splunk
systemctl disable splunk
sudo su - splunk
ssh-keygen
# enter x3
cat ~/.ssh/id_rsa.pub
exit

# authorize key on old
tshp dc-c19-splunk-sh.msoc.defpoint.local
mkdir .ssh
cat >> .ssh/authorized_keys
# paste from above
exit

# Validate that it's working
tshp dc-c19-splunk-sh
sudo su - splunk
ssh frederick.t.damstra@dc-c19-splunk-sh.msoc.defpoint.local

pre-rsync legacy to new:

tshp dc-c19-splunk-sh
sudo su - splunk
# may be errors
time rsync --rsync-path="sudo rsync" -avz --delete --progress \
  frederick.t.damstra@dc-c19-splunk-sh.msoc.defpoint.local:/opt/splunk/ /opt/splunk/ \
  --exclude="*.log"   --exclude '*.log.*'   --exclude '*.bundle' --exclude ".ssh"

Excluding directories seems to be a recipe for trouble. But if you really want to try, you can add the following:

  --exclude 'splunk/etc/system/default/' \
  --exclude 'splunk/etc/system/bin/' \
  --exclude 'splunk/etc/system/static/' \
  --exclude 'splunk/etc/system/README' \
  --exclude 'splunk/var/log/' \
  --exclude 'splunk/var/run/*.bundle' \
  --exclude 'splunk/var/run/dispatch/' \
  --exclude 'splunk/bin/'

Post to slack:

The Search Head for CUST is going down for the transition to GovCloud. I will notify again when the new server is operational.

Stop the old SH:

tshp dc-c19-splunk-sh.msoc.defpoint.local
sudo systemctl stop splunk
sudo systemctl disable splunk
exit

Stop new SH and copy files across:

tshp dc-c19-splunk-sh
sudo systemctl stop splunk
sudo su - splunk
time rsync --rsync-path="sudo rsync" -avz --delete --progress \
  frederick.t.damstra@dc-c19-splunk-sh.msoc.defpoint.local:/opt/splunk/ /opt/splunk/ \
  --exclude="*.log"   --exclude '*.log.*'   --exclude '*.bundle' --exclude ".ssh"

# Fix permissions on the new sh
exit
sudo chown -R splunk:splunk /opt/splunk
# Fix hostnames
sudo sed -i 's/msoc.defpoint.local/pvt.xdr.accenturefederalcyber.com/' /opt/splunk/etc/system/local/*.conf
#sudo sed -i 's/splunk-mc-0.msoc.defpoint.local/splunk-mc-0.pvt.xdr.accenturefederalcyber.com/' /opt/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/inputs.conf
#sudo sed -i 's/splunk-mc-0.msoc.defpoint.local/splunk-mc-0.pvt.xdr.accenturefederalcyber.com/' /opt/splunk/etc/apps/SA-AFS-ES-KO/local/inputs.conf

Run highstate to fix saml:

tshp salt-master
salt 'dc-c19-splunk-sh.pvt*' state.highstate test=true --output-diff
salt 'dc-c19-splunk-sh.pvt*' state.highstate test=true --output-diff

tshp dc-c19-splunk-sh
sudo systemctl start splunk
sudo systemctl enable splunk

Validate that you can log into https://dc-19-splunk.pvt.xdr.accenturefederalcyber.com

Post to slack:

The CUST Search Head is up. We are commencing testing of functionality and resolving any issues we find. Please let us know if you find anything here and we will resolve them as we are able. Note: The URL has changes. The new url is `https://<CUST>-splunk.pvt.xdr.accenturefederalcyber.com`. In the Okta launch page, it is listed as `<CUST> Splunk SH [Prod] [GC]`.

Disable okta legacy entries

Update phantom to point to new SH?

???

Post to slack:

We believe all issues related to the migration of the moose search head have been resolved. If you find further issues, please @mention me here, send me an email, or call me at 616-634-4933 if it's critical. Please remember to include as much detail as possible, including steps to reproduce the issue, expected behavior, and actual behavior. Thanks!