splunk_colddb_migration

Playbook to Migrate ColdDB to the SplunkCold Filesystem

Notes: The scripts now support multiple folders at once, so this readme may be slightly out of date, but multiple folders should be straightforward if you look at the playbook.

Expectations: Old cold path is /opt/splunk/var/lib/splunk//colddb New cold path is /opt/splunk/var/lib/splunkcold//colddb

Ansible Method:

Step 1:

Recommendation: Use Screen so you don't lose your session!

ansible-playbook install_rsync --extra-vars="target=TARGETS" ansible-playbook rsync_colddb --extra-vars="target=TARGETS folder=FOLDERNAME"

Watch progress in another window with: watch --interval 30 'ansible TARGETS --sudo --sudo-user=splunk -m shell -a "du -h --summarize /opt/splunk/var/lib/splunk/FOLDER/colddb /opt/splunk/var/lib/splunkcold/FOLDER/colddb"'

Step 2:

Run a search for year-to-date | tstats count where index=FOLDER by _time span=1d. Keep this window open for comparison at the end.

ansible-playbook install_rsync --extra-vars="target=Indexers" ansible-playbook rsync_colddb --extra-vars="target=Indexers folder=FOLDERNAME"

Watch progress in another window with: watch 'du -h --summarize splunk/FOLDERNAME/colddb splunkcold/FOLDERNAME/colddb/; echo ""; ps auxfw | grep rsync'

Step 2:

Run a search for year-to-date | tstats count where index=FOLDERNAME by _time span=1d. Keep this window open for comparison at the end.

On the MN:

# Enable maintenance mode: 
sudo -u splunk /opt/splunk/bin/splunk enable maintenance-mode
# Backup indexes.conf
sudo -u splunk cp /opt/splunk/etc/master-apps/_cluster/local/indexes.conf{,.20170725}
# Edit indexes.conf
sudo -u splunk vi /opt/splunk/etc/master-apps/_cluster/local/indexes.conf

If it doesn't exist, add the volume:

[volume:coldvol]
path = /opt/splunk/var/lib/splunkcold

Modify the index you are working on and add:

coldPath = volume:coldvol/<indexname>/colddb

DO NOT apply the bundle. DO NOT let anybody /else/ apply the bundle. Transfer indexes.conf to the ansible master into <ansible_home>/os_modifications/roles/splunk_colddb_migration/files/indexes.conf

On the MN, run: watch sudo -u splunk /opt/splunk/bin/splunk show cluster-status

Step 3:

For each indexer, run from ansible server:

ansible-playbook migrate_single_indexer --extra-vars="target=IP folder=FOLDERNAME"

• Check the cluster status before moving onto the next indexer! It takes a minute or two after starting before the indexer is back operational *

To verify you hit everybody, run: ansible --sudo --sudo-user=splunk Indexers -m shell -a "ls /opt/splunk/var/lib/splunk/FOLDER/colddb/". You should get error messages from every host.

Step 4: Disable maintenance mode, apply cluster bundle:

sudo -u splunk /opt/splunk/bin/splunk show maintenance-mode
sudo -u splunk /opt/splunk/bin/splunk disable maintenance-mode
sudo -u splunk /opt/splunk/bin/splunk show cluster-bundle-status
sudo -u splunk /opt/splunk/bin/splunk validate cluster-bundle
sudo -u splunk /opt/splunk/bin/splunk show cluster-bundle-status
sudo -u splunk /opt/splunk/bin/splunk apply cluster-bundle

Step 5: Clean up the /opt/splunk/var/lib/splunk/*/colddb.migrated directories

For the daring: ansible TARGETS --sudo --sudo-user=splunk -m shell -a 'rm -rfv /opt/splunk/var/lib/splunk/FOLDERNAME/colddb.migrated'

####################################################################

Manual Method (Just for reference, use the ansible method above)

1) Do a presync to minimize downtime (can be run multiple times before cutover):

    a. sudo -u splunk mkdir -p /opt/splunk/var/lib/splunkcold/FOLDER/colddb
    b. sudo -u splunk rsync -avz --delete /opt/splunk/var/lib/splunk/FOLDER/colddb /opt/splunk/var/lib/splunkcold/FOLDER/colddb

2) Update the master node:

    a. sudo -u splunk /opt/splunk/bin/splunk enable maintenance-mode 
    b. cp /opt/splunk/etc/master-apps/_cluster/local/indexes.conf{,.20170725}
    c. vi /opt/splunk/etc/master-apps/_cluster/local/indexes.conf
         i. Add:
        [volume:coldvol]
        path = /opt/splunk/var/lib/splunkcold
         ii. Then update the coldPath for FOLDER to be volume:coldvol/indexname/colddb

    d. Do NOT deploy the changes. Make sure EVERYBODY KNOWS, no touching the master node!

3) On each indexer in turn:

    a. sudo su - splunk
    b. /opt/splunk/bin/splunk stop
    c. rsync -avz --delete /opt/splunk/var/lib/splunk/FOLDER/colddb /opt/splunk/var/lib/splunkcold/FOLDER/colddb
    c. rsync -avz --delete /opt/splunk/var/lib/splunk/FOLDERNAME/colddb /opt/splunk/var/lib/splunkcold/FOLDERNAME/colddb
    d. Manually copy the indexes.conf from the master node to /opt/splunk/etc/slave-apps/_cluster/local/indexes.conf
    e. mv /opt/splunk/var/lib/splunk/FOLDERNAME/colddb{,.20170725}
    f. /opt/splunk/bin/splunk btool check
    g. /opt/splunk/bin/splunk start

4) After all indexes are completed, run a search: | tstats count where index=FOLDERNAME by _time span=1d

    a. Year to date. There should not be gaps.

5) If everything checks out, turn off maintenance mode and apply the cluster bundle (if changes were made exactly, no bundle update will go out).