infrastructure-notes/Splunk Migration from Commercial to GovCloud - 3. Remaining Servers.md

Splunk Migration from Commercial to GovCloud

Migrate the HF (this will a no brainer in test, but in prod has implications)

Terraform the HF

cd ~/xdr-terraform-live/test/aws-us-gov/mdr-test-c2/180-splunk-heavy-forwarder
terragrunt apply

Highstate it:

ssh gc-dev-salt-master
salt 'moose-splunk-hf.pvt.xdrtest.accenturefederalcyber.com' state.highstate --output-diff
# Highstate a second time
salt 'moose-splunk-hf.pvt.xdrtest.accenturefederalcyber.com' state.highstate --output-diff

Initial rsyncs:

# Log into new HF and stop splunkd
ssh gc-dev-moose-splunk-hf
sudo systemctl stop splunk
exit
cd
# Do a 'live' sync between the two
rsync --rsync-path="sudo rsync" -avz --delete --progress prod-moose-splunk-hf:/opt/splunk tmp/moose-hf  \
  --exclude '*.log' \
  --exclude '*.log.*' \
  --exclude '*.bundle'
rsync --rsync-path="sudo rsync" -avz --delete --progress tmp/moose-hf/splunk/ gc-prod-moose-splunk-hf:/opt/splunk/ \
  --exclude '*.log' \
  --exclude '*.log.*' \
  --exclude '*.bundle'

rsync Final cutover:

# Stop splunk on the old HF
ssh dev-moose-splunk-hf
sudo systemctl stop splunk
sudo systemctl disable splunk
exit

rsync --rsync-path="sudo rsync" -avz --delete --progress prod-moose-splunk-hf:/opt/splunk tmp/moose-hf  \
  --exclude '*.log' \
  --exclude '*.log.*' \
  --exclude '*.bundle'
rsync --rsync-path="sudo rsync" -avz --delete --progress tmp/moose-hf/splunk/ gc-prod-moose-splunk-hf:/opt/splunk/ \
  --exclude '*.log' \
  --exclude '*.log.*' \
  --exclude '*.bundle'

ssh gc-dev-moose-splunk-hf
sudo chown -R splunk:splunk /opt/splunk
cd /opt/splunk
sudo grep 'msoc.defpoint.local' `sudo find . -name "*.conf"`
# replace anything found with a commadn like the following:
sudo sed -i 's/msoc.defpoint.local/pvt.xdrtest.accenturefederalcyber.com/' /opt/splunk/etc/system/local/*.conf

ssh gc-dev-salt-master
salt 'moose-splunk-hf.pvt*' state.highstate --output-diff
exit

ssh gc-prod-moose-splunk-hf
sudo systemctl start splunk

Clean up indexers

Indexers will unnecessarily use up disk space for old searchheads. Clean up the bundles:

ssh gc-dev-salt-master
salt 'moose-splunk-idx-63f*' cmd.run 'systemctl stop splunk; rm -rf /opt/splunk/var/run/searchpeers/*defpoint.local*; systemctl start splunk'
# wait for sf/rf, and repeat for other indexers