Update Splunk SmartStore migration document

Splunk SmartStore Migration.md

@@ -2,6 +2,12 @@
 
 Splunk documentation: https://docs.splunk.com/Documentation/Splunk/latest/Indexer/AboutSmartStore
 
+Items of note:
+* SmartStore data retention is managed cluster-wide
+* Only `maxGlobalDataSizeMB`, `maxGlobalRawDataSizeMB`, and `frozenTimePeriodInSecs` control when to freeze data.
+  * The most restrictive rule applies.
+  * When buckets freeze, they are removed from both remote and local storage.
+
 ## Prerequisites
 
 ### Create the Target S3 Bucket
@@ -90,9 +96,15 @@ splunk cmd splunkd rfs -- ls --starts-with volume:smartstore/<some-index>/
 
 ## Migrate Indices to SmartStore
 
+### Confirm That the Cluster is Healthy and in the _Complete_ State
+
+Set `constrain_singlesite_buckets = false` in the Cluster Manager's `server.conf` under `[clustering]` and restart Splunk on the Cluster Manager.
+
+Check the Bucket Status panel and resolve any pending fixup tasks.
+
 ### Test With One Index to Start
 
-Add `remotePath = volume:smartstore/$_index_name` to an index such as `_introspection` in the Cluster Manager's copy of `master-apps/all_indexes/local/indexes.conf` and apply the change via Salt.
+Add `remotePath = volume:smartstore/$_index_name` to an index such as `_introspection` in the Cluster Manager's copy of `master-apps/all_indexes/local/indexes.conf`, set `frozenTimePeriodInSecs = 0` and `maxGlobalDataSizeMB = 0`, then apply the change via Salt.
 
 ### Check Splunk's Log
 
@@ -104,6 +116,22 @@ index=_internal sourcetype=splunkd TERM(action=upload)
 
 The _introspection index should appear in the search results with values under "Attempting" and "Succeeded". If the value under "Failed" is greater than zero, check splunkd.log on one of the indexers to troubleshoot.
 
+Additional Splunk Searches:
+
+```
+| rest /services/admin/cacheman/_metrics | fields splunk_server migration.*
+```
+
+If Splunk restarts before the migration completes, the endpoint data may not be accurate. If that happens, run:
+
+```
+| rest /services/admin/cacheman splunk_server=<peer>
+| search cm:bucket.stable=0
+| stats count
+```
+
+IMPORTANT: Do not forget to reconfigure the retention settings after the migration.
+
 ### Move Remaining Indices to SmartStore
 
 Add `remotePath = volume:smartstore/$_index_name` under the `[default]` stanza as well as under all other index definitions. This is a good time to update `indexes.conf` entries where a stanza is relying on values from `[default]` rather than having them defined per index.
@@ -129,4 +157,10 @@ quarantinePastSecs = 604800
 repFactor = auto
 ```
 
-Once all the index definitions have `remotePath` defined, use Salt to apply the bundle change to the indexers. Observe the progress of the bundle application from the Cluster Manager as mentioned above and use the Splunk search to observe data moving to S3.
+Once all the index definitions have `remotePath` defined, use Salt to apply the bundle change to the indexers. Observe the progress of the bundle application from the Cluster Manager as mentioned above and use the Splunk search to observe data moving to S3.
+
+# Thawing Frozen Data
+
+DO NOT thaw an archived (frozen) bucket into a SmartStore index!
+
+Create a separate, "classic" index that does not utilize SmartStore (no `remotePath`) and thaw the buckets into the `thawedPath` of that index.