infrastructure-notes/Splunk Upgrade Notes.md

Splunk Upgrade Notes

Splunk Upgrade 2020 "The Big One"

08/11/2020

Software is located in Duane's One drive.

Overall Plan

1. Upgrade AFS/NGA 7.0.3 -> 8.0.5

   1. Why? bc of SOC blockers.
   2. Prep Work
      1. Ensure Apps are 8x compatible. Make a list of apps that will be upgraded prior upgrade and after upgrade.
      2. Check modular apps to see for python3 compatability
      3. Create one drive doc to track compatability
      4. Pull Brandon into app upgrade checks
      5. Pillar if/then for test/prod
      6. The Python Upgrade will be completed at a later date
         1. Upgrade using the Python 2 runtime and make minimal changes to Python code
   3. AFS/NGA upgrade
      1. Update salt pillar data to 8.0.5 repo to reflect new splunk repo.
         • 0.2 Dump all passwords from the password store PRIOR to upgrade.
           • 0.2.1 Run on the HF: | rest /services/storage/passwords
      2. Ensure recent backup of SH EBS
      3. upgrade indexers: stop all at the same time
         • 3.1. apply the updated pillar datasalt afs* saltutil.refresh_pillar
         • 3.2. verify the pillar is updatedsalt afs* pillar.item yumrepos:splunk
         • 3.3. verify there is enough disk space
      4. Upgrade CM
         • 0.1 Setup silence on Sensu for ALL servers
           1. Run: state.sls splunk.new_install to update repo ; yes it will restart splunk. (ROOM FOR IMPROVEMENT: Make new saltstate for splunk repo)
           2. Stop splunk cmd.run 'systemctl stop splunk'
           3. Upgrade splunk pkg.upgrade name=splunk
              • 3.1 Splunk is now waiting for accept license. Do Not Start Splunk Until after indexers are upgraded.
      5. Upgrade SH
         • 0.1 Setup silence on Sensu
           1. Run: state.sls splunk.new_install to update repo
           2. Stop splunk cmd.run 'systemctl stop splunk'
              • 2.1 Backup /opt/splunk tar -cvzf /opt/splunk/opt-splunk-backup.tar.gz /opt/splunk
           3. Upgrade splunk pkg.upgrade name=splunk
              • 3.1 Splunk is now waiting for accept license.
      6. Upgrade Indexers
         • 0.1 Setup silence on Sensu
           1. Run: state.sls splunk.new_install to update repo
           2. Stop splunk cmd.run 'systemctl stop splunk'
           3. Upgrade splunk pkg.upgrade name=splunk
           4. Start indexers and accept license cmd.run 'systemctl start splunk'
              • 4.1 cmd.run '/opt/splunk/bin/splunk version'
              • 3.2 cmd.run '/opt/splunk/bin/splunk status'
      7. Start CM and SH
         1. Start CM/SH and accept license cmd.run 'systemctl start splunk'
      8. Upgrade HF (slice only, not POPs)
         1. Run: state.sls splunk.new_install to update repo
         2. Stop splunk cmd.run 'systemctl stop splunk'
            • 2.1 Backup /opt/splunk tar -cvzf /opt/splunk/opt-splunk-backup.tar.gz /opt/splunk
         3. Upgrade splunk pkg.upgrade name=splunk
         4. Start indexers and accept license cmd.run 'systemctl start splunk'
      9. After Splunk App Upgrades
         

         1. Upgrade ES 5.0.1 -> 6.2.0

            1. The app failed to upload to the SH. ( takes a long time ). Modify the etc/system/local/web.conf to allow large uploads.

               max_upload_size = 1024
               

         2. See Matrix for other apps ( upgrade apps slowly so Brandon can troubleshoot errors!!!!)

         3. run geo ip DB update

            1. /usr/local/bin/maxmind-downloader.sh

         4. (Prevents 3 green checkmarks on CM) Update the CM bundle to include _cluster see here: Fixes for not replicating indexes? (index _metrics and _introspection not in _cluster)

         5. NGA has an additional check on the splunk HF IAM role for externalID. Besure to add the "patch" back in. See here: Jira Ticket - MSOCI-623 - Splunk AWS TA doesn't support --external-id when assuming an IAM role. This is for the splunk_TA_aws app.

      10. Delete Sensu Silences
      11. Check lastchance index for unusual data. If the upgrade of ES introducing new indexes, and the new indexes are not on the Splunk indexers, then the data will be put into the lastchance index.

2. Upgrade Moose 7.2.1 ->8.0.5 DONE!

   1. test
      1. CM, SH, indexer, HF, Forwarders
   2. prod
   3. Upgrade *.local Universal Forwarders

3. Upgrade Covids 8.0.4 -> 8.0.5

   1. test
   2. prod

4. Upgrade POP nodes