|
@@ -3,6 +3,10 @@ Notes: The scripts now support multiple folders at once, so this
|
|
readme may be slightly out of date, but multiple folders should
|
|
readme may be slightly out of date, but multiple folders should
|
|
be straightforward if you look at the playbook.
|
|
be straightforward if you look at the playbook.
|
|
|
|
|
|
|
|
+Expectations:
|
|
|
|
+Old cold path is /opt/splunk/var/lib/splunk/<foldername>/colddb
|
|
|
|
+New cold path is /opt/splunk/var/lib/splunkcold/<foldername>/colddb
|
|
|
|
+
|
|
## Ansible Method:
|
|
## Ansible Method:
|
|
### Step 1:
|
|
### Step 1:
|
|
Recommendation: Use Screen so you don't lose your session!
|
|
Recommendation: Use Screen so you don't lose your session!
|
|
@@ -13,9 +17,17 @@ Recommendation: Use Screen so you don't lose your session!
|
|
Watch progress in another window with:
|
|
Watch progress in another window with:
|
|
`watch --interval 30 'ansible TARGETS --sudo --sudo-user=splunk -m shell -a "du -h --summarize /opt/splunk/var/lib/splunk/FOLDER/colddb /opt/splunk/var/lib/splunkcold/FOLDER/colddb"'`
|
|
`watch --interval 30 'ansible TARGETS --sudo --sudo-user=splunk -m shell -a "du -h --summarize /opt/splunk/var/lib/splunk/FOLDER/colddb /opt/splunk/var/lib/splunkcold/FOLDER/colddb"'`
|
|
|
|
|
|
-
|
|
|
|
### Step 2:
|
|
### Step 2:
|
|
Run a search for year-to-date `| tstats count where index=FOLDER by _time span=1d`. Keep this window open for comparison at the end.
|
|
Run a search for year-to-date `| tstats count where index=FOLDER by _time span=1d`. Keep this window open for comparison at the end.
|
|
|
|
+=======
|
|
|
|
+`ansible-playbook install_rsync --extra-vars="target=Indexers"`
|
|
|
|
+`ansible-playbook rsync_colddb --extra-vars="target=Indexers folder=FOLDERNAME"`
|
|
|
|
+
|
|
|
|
+Watch progress in another window with:
|
|
|
|
+`watch 'du -h --summarize splunk/FOLDERNAME/colddb splunkcold/FOLDERNAME/colddb/; echo ""; ps auxfw | grep rsync'`
|
|
|
|
+
|
|
|
|
+### Step 2:
|
|
|
|
+Run a search for year-to-date `| tstats count where index=FOLDERNAME by _time span=1d`. Keep this window open for comparison at the end.
|
|
|
|
|
|
On the MN:
|
|
On the MN:
|
|
```
|
|
```
|
|
@@ -46,11 +58,12 @@ On the MN, run:
|
|
|
|
|
|
### Step 3:
|
|
### Step 3:
|
|
For each indexer, run from ansible server:
|
|
For each indexer, run from ansible server:
|
|
-ansible-playbook migrate_single_indexer --extra-vars="target=IP folder=defaultdb"
|
|
|
|
|
|
+
|
|
|
|
+ansible-playbook migrate_single_indexer --extra-vars="target=IP folder=FOLDERNAME"
|
|
* Check the cluster status before moving onto the next indexer! It takes a minute or two after starting before the indexer is back operational *
|
|
* Check the cluster status before moving onto the next indexer! It takes a minute or two after starting before the indexer is back operational *
|
|
|
|
|
|
To verify you hit everybody, run:
|
|
To verify you hit everybody, run:
|
|
-`ansible --sudo --sudo-user=splunk TARGETS -m shell -a "ls /opt/splunk/var/lib/splunk/FOLDER/colddb/"`. You should get error messages from every host.
|
|
|
|
|
|
+`ansible --sudo --sudo-user=splunk Indexers -m shell -a "ls /opt/splunk/var/lib/splunk/FOLDER/colddb/"`. You should get error messages from every host.
|
|
|
|
|
|
### Step 4: Disable maintenance mode, apply cluster bundle:
|
|
### Step 4: Disable maintenance mode, apply cluster bundle:
|
|
```
|
|
```
|
|
@@ -79,17 +92,18 @@ For the daring:
|
|
[volume:coldvol]
|
|
[volume:coldvol]
|
|
path = /opt/splunk/var/lib/splunkcold
|
|
path = /opt/splunk/var/lib/splunkcold
|
|
ii. Then update the coldPath for FOLDER to be volume:coldvol/indexname/colddb
|
|
ii. Then update the coldPath for FOLDER to be volume:coldvol/indexname/colddb
|
|
|
|
+
|
|
d. Do NOT deploy the changes. Make sure EVERYBODY KNOWS, no touching the master node!
|
|
d. Do NOT deploy the changes. Make sure EVERYBODY KNOWS, no touching the master node!
|
|
3) On each indexer in turn:
|
|
3) On each indexer in turn:
|
|
a. sudo su - splunk
|
|
a. sudo su - splunk
|
|
b. /opt/splunk/bin/splunk stop
|
|
b. /opt/splunk/bin/splunk stop
|
|
c. rsync -avz --delete /opt/splunk/var/lib/splunk/FOLDER/colddb /opt/splunk/var/lib/splunkcold/FOLDER/colddb
|
|
c. rsync -avz --delete /opt/splunk/var/lib/splunk/FOLDER/colddb /opt/splunk/var/lib/splunkcold/FOLDER/colddb
|
|
|
|
+ c. rsync -avz --delete /opt/splunk/var/lib/splunk/FOLDERNAME/colddb /opt/splunk/var/lib/splunkcold/FOLDERNAME/colddb
|
|
d. Manually copy the indexes.conf from the master node to /opt/splunk/etc/slave-apps/_cluster/local/indexes.conf
|
|
d. Manually copy the indexes.conf from the master node to /opt/splunk/etc/slave-apps/_cluster/local/indexes.conf
|
|
- e. mv /opt/splunk/var/lib/splunk/FOLDER/colddb{,.20170725}
|
|
|
|
|
|
+ e. mv /opt/splunk/var/lib/splunk/FOLDERNAME/colddb{,.20170725}
|
|
f. /opt/splunk/bin/splunk btool check
|
|
f. /opt/splunk/bin/splunk btool check
|
|
g. /opt/splunk/bin/splunk start
|
|
g. /opt/splunk/bin/splunk start
|
|
4) After all indexes are completed, run a search:
|
|
4) After all indexes are completed, run a search:
|
|
-| tstats count where index=FOLDER by _time span=1d
|
|
|
|
|
|
+| tstats count where index=FOLDERNAME by _time span=1d
|
|
a. Year to date. There should not be gaps.
|
|
a. Year to date. There should not be gaps.
|
|
5) If everything checks out, turn off maintenance mode and apply the cluster bundle (if changes were made exactly, no bundle update will go out).
|
|
5) If everything checks out, turn off maintenance mode and apply the cluster bundle (if changes were made exactly, no bundle update will go out).
|
|
-
|
|
|