First commit.

README.md

@@ -0,0 +1,94 @@
+# Playbook to Migrate ColdDB to the SplunkCold Filesystem
+Expectations: 
+Old cold path is /opt/splunk/var/lib/splunk/<foldername>/colddb
+New cold path is /opt/splunk/var/lib/splunkcold/<foldername>/colddb
+
+## Ansible Method:
+### Step 1:
+Recommendation: Use Screen so you don't lose your session!
+
+`ansible-playbook install_rsync --extra-vars="target=Indexers"`
+`ansible-playbook rsync_colddb --extra-vars="target=Indexers folder=FOLDERNAME"`
+
+Watch progress in another window with:
+`watch 'du -h --summarize splunk/FOLDERNAME/colddb splunkcold/FOLDERNAME/colddb/; echo ""; ps auxfw | grep rsync'`
+
+### Step 2:
+Run a search for year-to-date `| tstats count where index=FOLDERNAME by _time span=1d`. Keep this window open for comparison at the end.
+
+On the MN:
+```
+# Enable maintenance mode: 
+sudo -u spunk /opt/splunk/bin/splunk enable maintenance-mode
+# Backup indexes.conf
+sudo -u splunk cp /opt/splunk/etc/master-apps/_cluster/local/indexes.conf{,.20170725}
+# Edit indexes.conf
+sudo -u splunk vi /opt/splunk/etc/master-apps/_cluster/local/indexes.conf
+```
+If it doesn't exist, add the volume:
+```
+[volume:coldvol]
+path = /opt/splunk/var/lib/splunkcold
+```
+
+Modify the index you are working on and add:
+```
+coldPath = volume:coldvol/<indexname>/colddb
+```
+
+DO NOT apply the bundle. DO NOT let anybody /else/ apply the bundle.
+Transfer indexes.conf to the ansible master into 
+`<ansible_home>/os_modifications/roles/splunk_colddb_migration/files/indexes.conf`
+
+On the MN, run:
+`watch sudo -u splunk /opt/splunk/bin/splunk show cluster-status`
+
+### Step 3:
+For each indexer, run from ansible server:
+ansible-playbook migrate_single_indexer --extra-vars="target=IP folder=FOLDERNAME"
+* Check the cluster status before moving onto the next indexer! It takes a minute or two after starting before the indexer is back operational *
+
+To verify you hit everybody, run:
+`ansible --sudo --sudo-user=splunk Indexers -m shell -a "ls /opt/splunk/var/lib/splunk/FOLDER/colddb/"`. You should get error messages from every host.
+
+### Step 4: Disable maintenance mode, apply cluster bundle:
+```
+sudo -u splunk /opt/splunk/bin/splunk show maintenance-mode
+sudo -u splunk /opt/splunk/bin/splunk disable maintenance-mode
+sudo -u splunk /opt/splunk/bin/splunk show cluster-bundle-status
+sudo -u splunk /opt/splunk/bin/splunk validate cluster-bundle
+sudo -u splunk /opt/splunk/bin/splunk show cluster-bundle-status
+sudo -u splunk /opt/splunk/bin/splunk apply cluster-bundle
+```
+
+### Step 5: Clean up the `/opt/splunk/var/lib/splunk/*/colddb.migrated` directories
+For the daring:
+`ansible Indexers --sudo --sudo-user=splunk -m shell -a 'rm -rfv /opt/splunk/var/lib/splunk/FOLDERNAME/colddb.migrated'`
+
+####################################################################
+## Manual Method (Just for reference, use the ansible method above)
+1) Do a presync to minimize downtime (can be run multiple times before cutover):
+		a. sudo -u splunk mkdir -p /opt/splunk/var/lib/splunkcold/FOLDERNAME/colddb
+		b. sudo -u splunk rsync -avz --delete /opt/splunk/var/lib/splunk/FOLDERNAME/colddb /opt/splunk/var/lib/splunkcold/FOLDERNAME/colddb
+2) Update the master node:
+		a. sudo -u splunk /opt/splunk/bin/splunk enable maintenance-mode 
+		b. cp /opt/splunk/etc/master-apps/_cluster/local/indexes.conf{,.20170725}
+		c. vi /opt/splunk/etc/master-apps/_cluster/local/indexes.conf
+			 i. Add:
+			[volume:coldvol]
+			path = /opt/splunk/var/lib/splunkcold
+			 ii. Then update the coldPath for FOLDERNAME to be volume:coldvol/indexname/colddb
+		d. Do NOT deploy the changes. Make sure EVERYBODY KNOWS, no touching the master node!
+3) On each indexer in turn:
+		a. sudo su - splunk
+		b. /opt/splunk/bin/splunk stop
+		c. rsync -avz --delete /opt/splunk/var/lib/splunk/FOLDERNAME/colddb /opt/splunk/var/lib/splunkcold/FOLDERNAME/colddb
+		d. Manually copy the indexes.conf from the master node to /opt/splunk/etc/slave-apps/_cluster/local/indexes.conf
+		e. mv /opt/splunk/var/lib/splunk/FOLDERNAME/colddb{,.20170725}
+		f. /opt/splunk/bin/splunk btool check
+		g. /opt/splunk/bin/splunk start
+4) After all indexes are completed, run a search:
+| tstats count where index=FOLDERNAME by _time span=1d
+		a. Year to date. There should not be gaps.
+5) If everything checks out, turn off maintenance mode and apply the cluster bundle (if changes were made exactly, no bundle update will go out).
+

files/.gitignore

@@ -0,0 +1 @@
+indexes.conf

files/README.md

@@ -0,0 +1,4 @@
+# Instructions
+You will need to manually copy an indexes.conf into this folder
+for distribution to all peers.
+

tasks/install_rsync.yml

@@ -0,0 +1,11 @@
+---
+- hosts: "{{ target }}"
+  become: true
+
+  tasks:
+  # Verify rsync is installed
+  - name: Ensure rsync is installed
+    package:
+      name: rsync
+      state: latest
+

tasks/migrate_single_indexer.yml

@@ -0,0 +1,80 @@
+---
+# Perform actual migration of a server.
+# PREREQUITES:
+#       1) An initial rsync should have been performed (see rsync_colddb)
+#       2) Cluster should be in maintenance mode
+#       3) This should be called on a single target, though the serial
+#          limit probably makes it safe.
+#       4) DO NOT RUN TWICE (though it should prevent that)
+#
+# Specify extra vars for both "target" and "folder"
+# 
+# e.g.:
+#   ansible-playbook migrate_single_indexer.yml --extra-vars="target=10.10.10.10 folder=defaultdb"
+- hosts: "{{ target }}"
+  become: true
+  become_user: splunk
+  serial: 1
+
+  tasks:
+  # Verify folder is defined
+  - name: Variable check
+    fail: msg="Variable 'folder' is not defined or is invalid. Please run with --extra-vars=\"target=x folder=dbfolder\""
+    when: (folder is not defined)
+
+  # Verify folder exists
+  - name: Ensure folder already exists
+    stat:
+      path: /opt/splunk/var/lib/splunkcold/{{ folder }}/colddb
+    register: colddbpath
+
+  - name: Fail if the folder doesn't exist
+    fail: msg="The colddb folder does not exist."
+    when: not(colddbpath.stat.isdir is defined and colddbpath.stat.isdir)
+
+  - debug:
+      msg: "Cold Path exists. Good."
+    when: colddbpath.stat.isdir is defined and colddbpath.stat.isdir
+   
+  # Verify migrated folder does not exist (DO NOT RUN TWICE!)
+  - name: Ensure migrated folder does not exist
+    stat:
+      path: /opt/splunk/var/lib/splunk/{{ folder }}/colddb.migrated
+    register: colddbmigratedpath
+
+  - name: Fail if the migrated folder exist
+    fail: msg="The migrated folder already exists. (Already run on this index?)"
+    when: colddbmigratedpath.stat.isdir is defined and colddbmigratedpath.stat.isdir
+
+  - debug:
+      msg: "Migrated Cold Path does not exist. Good."
+    when: not(colddbmigratedpath.stat.isdir is defined and coldmigratedpath.stat.isdir)
+   
+  # Stop Splunk
+  - name: Stop Splunk
+    command: /opt/splunk/bin/splunk stop
+ 
+  - name: rsync cold data
+    command: rsync -avz --delete /opt/splunk/var/lib/splunk/{{ folder }}/colddb/ /opt/splunk/var/lib/splunkcold/{{ folder }}/colddb/
+    # Run this asynchyronously for one hour, polling every 30.
+#    async: 604800
+#    poll: 60
+    register: rsync_result
+
+  - name: overwrite indexes.conf
+    copy:
+      src: ../files/indexes.conf
+      dest: /opt/splunk/etc/slave-apps/_cluster/local/indexes.conf
+      owner: splunk
+      group: splunk
+      mode: 0600
+
+  - name: Rename Colddb path
+    command: mv /opt/splunk/var/lib/splunk/{{ folder }}/colddb /opt/splunk/var/lib/splunk/{{ folder }}/colddb.migrated
+
+  - name: Btool Check for Good Measure
+    command: /opt/splunk/bin/splunk btool check
+  
+  - name: start splunk
+    command: /opt/splunk/bin/splunk start
+

tasks/rsync_colddb.yml

@@ -0,0 +1,34 @@
+---
+# Synchronize a folder to the new colddb path.
+# Specify extra vars for both "target" and "folder"
+# 
+# e.g.:
+#   ansible-playbook rsync_colddb.yml --extra-vars="target=AWS-Indexers folder=defaultdb"
+- hosts: "{{ target }}"
+  become: true
+  become_user: splunk
+
+  tasks:
+  # Verify folder is defined
+  - name: Variable check
+    fail: msg="Variable 'folder' is not defined or is invalid. Please run with --extra-vars=\"target=x folder=dbfolder\""
+    when: (folder is not defined)
+
+  # Verify folder exists
+  - name: Ensure folder exists
+    file:
+      path: /opt/splunk/var/lib/splunkcold/{{ folder }}/colddb
+      state: directory
+      mode: 0750
+    
+  - name: rsync cold data
+    command: rsync -avz --delete /opt/splunk/var/lib/splunk/{{ folder }}/colddb/ /opt/splunk/var/lib/splunkcold/{{ folder }}/colddb/
+    # Run this asynchyronously for one hour, polling every 30.
+#    async: 604800
+#    poll: 60
+    register: rsync_result
+
+  - debug: msg="{{ rsync_result.stdout}}"
+  - debug: msg="{{ rsync_result.stderr}}"
+
+