infrastructure-notes/Splunk Migration from Commercial to GovCloud - 3. Remaining Servers.md

Splunk Migration from Commercial to GovCloud

Migrate the HF (this will a no brainer in test, but in prod has implications)

Terraform the HF:

cd ~/xdr-terraform-live/test/aws-us-gov/mdr-test-c2/180-splunk-heavy-forwarder
terragrunt apply

Highstate it:

tshp gc-dev-salt-master
salt 'moose-splunk-hf.pvt.xdrtest.accenturefederalcyber.com' state.highstate --output-diff
# Highstate a second time
salt 'moose-splunk-hf.pvt.xdrtest.accenturefederalcyber.com' state.highstate --output-diff

Prep the keys:

tshp CUST-splunk-hf
sudo systemctl stop splunk
sudo systemctl disable splunk
sudo su - splunk
ssh-keygen
# enter x3
cat ~/.ssh/id_rsa.pub
exit
exit

tshp CUST-splunk-hf.msoc.defpoint.local
mkdir .ssh
cat >> .ssh/authorized_keys
# paste from above
exit

Initial rsyncs:

tshp CUST-splunk-hf
sudo systemctl stop splunk
sudo su - splunk
time rsync --rsync-path="sudo rsync" -avz --delete --progress \
  frederick.t.damstra@CUST-splunk-hf.msoc.defpoint.local:/opt/splunk/ /opt/splunk/ \
  --exclude="*.log"   --exclude '*.log.*'   --exclude '*.bundle' --exclude ".ssh"

Log into new HF and stop splunkd

Final cutover:

Stop splunk on the old HF

tshp CUST-splunk-hf.msoc.defpoint.local
sudo systemctl stop splunk
sudo systemctl disable splunk
exit

tshp CUST-splunk-hf
sudo systemctl stop splunk
sudo su - splunk
time rsync --rsync-path="sudo rsync" -avz --delete --progress \
  frederick.t.damstra@CUST-splunk-hf.msoc.defpoint.local:/opt/splunk/ /opt/splunk/ \
  --exclude="*.log"   --exclude '*.log.*'   --exclude '*.bundle' --exclude ".ssh"
exit
sudo chown -R splunk:splunk /opt/splunk
cd /opt/splunk
sudo grep 'msoc.defpoint.local' `sudo find . -name "*.conf"`
# replace anything found with a commadn like the following:
sudo sed -i 's/msoc.defpoint.local/pvt.xdr.accenturefederalcyber.com/' /opt/splunk/etc/system/local/*.conf

ssh gc-dev-salt-master
salt 'moose-splunk-hf.pvt*' state.highstate --output-diff
exit

ssh gc-prod-moose-splunk-hf
sudo systemctl start splunk

Clean up indexers

Indexers will unnecessarily use up disk space for old searchheads. Clean up the bundles:

ssh gc-dev-salt-master
salt 'moose-splunk-idx-63f*' cmd.run 'systemctl stop splunk; rm -rf /opt/splunk/var/run/searchpeers/*defpoint.local*; systemctl start splunk'
# wait for sf/rf, and repeat for other indexers