infrastructure-notes/Sensu Go Upgrade Notes.md

Sensu Upgrade Notes

Places where Official Sensu Go code and Documentation is located

---

• Official Sensu Go Repo Github
• Official Sensu Go Website
• Official Sensu Go Upgrade Documentation.
  
• Official Sensu Hosted Package Repo Service Packagecloud

:warning: We will use our XDR Internal Reposerver for all upgrade methods - See How to add a new package to the Reposerver

Sensu Go Upgrade History

---

• MSOCI-1565 ticket - Upgrade Sensu to 6.2.x
• MSOCI-1908 ticket - Upgrade Sensu to 6.4.3
• MSOCI-1969 ticket - Upgrade Sensu to 6.6.1
• MSOCI-2027 ticket - Upgrade Sensu to 6.7.0
• MSOCI-2173 ticket - Upgrade Sensu to 6.7.2
• MSOCI-2244 ticket - Upgrade Sensu to 6.7.4
• MSOCI-2288 ticket - Upgrade Sensu to 6.7.5

Sensu Go Upgrade Process

---

We want to deploy the new code in iterations so that we can quickly abort deployment if we run in to any issues. Start with GC Test XDR Infrastructure first.

Starting with Moose and Internal infra within GC TEST. After deployment is verfied and functional, let it bake for 24-48 hrs before GC Prod deployment.

1. Download latest packages for Sensu backend, Sensu agents, Sensuctl (Sensu CLI) to Repo server and run yum clean all on Sensu Backend server - See Reposerver notes.

2. If needed, update Salt states to ensure they are up-to-date - Salt Upgrade Notes

   salt sensu* state.sls salt_minion.minion_upgrade --output-diff test=true
   

:warning: Remember to silence Sensu alerts before restarting services

1. Sensu first; Login to GC TEST Salt-Master and Stop Sensu services on Sensu Backend server; do the same process for GC PROD afterwards

   date; salt sensu* cmd.run 'systemctl stop sensu-agent'
   date; salt sensu* cmd.run 'systemctl stop sensu-backend'
   

2. Update Sensu Backend server

   date; salt sensu* cmd.run 'yum clean all && yum makecache fast'
   salt sensu* cmd.run 'yum --disablerepo="*" --enablerepo="msoc" list available'
   date; salt sensu* cmd.run 'yum update -y sensu-go-backend'
   date; salt sensu* cmd.run 'yum update -y sensu-go-cli'
   date; salt sensu* cmd.run 'yum update -y sensu-go-agent'
   date; salt sensu* cmd.run 'systemctl daemon-reload'
   

3. Restart the Sensu services and check the Status

   date; salt sensu* cmd.run 'systemctl start sensu-backend'
   salt sensu* cmd.run 'systemctl start sensu-agent'
   
   date; salt sensu* cmd.run 'systemctl status sensu-backend'
   salt sensu* cmd.run 'systemctl status sensu-agent'
   

   :warning: Did you silence Sensu alerts before restarting services?

4. GC Test first; GC PROD second; From target servers; clean out the cache

   # XDR Infrastructure - be sure to note the different Salt minions to target between TEST and PROD
   date; salt -C '* not ( afs* or nga* or ma-* or dc-c19* or doed* or la-c19* or bas-* or ca-c19* or frtib* or dgi* or vmray* or sensu* )' cmd.run 'yum clean all && yum makecache fast'
   
   # From target servers; view the available packages
   salt -C '* not ( afs* or nga* or ma-* or dc-c19* or doed* or la-c19* or bas-* or ca-c19* or frtib* or dgi* or vmray* or sensu* )' cmd.run 'yum --disablerepo="*" --enablerepo="msoc" list available'
   
   # Customer Slices Search Heads Only
   date; salt -C '*-sh* and not *moose* and not fm-shared-search*' cmd.run 'yum clean all && yum makecache fast'
   
   salt -C '*-sh* and not *moose* and not fm-shared-search*' cmd.run 'yum --disablerepo="*" --enablerepo="msoc" list available'
   
   # Customer Slices Cluster masters and Heavy Forwarders 
   date; salt -C '( *splunk-cm* or *splunk-hf* ) not moose*' cmd.run 'yum clean all && yum makecache fast'
   
   salt -C '( *splunk-cm* or *splunk-hf* ) not moose*' cmd.run 'yum --disablerepo="*" --enablerepo="msoc" list available'
       
   # Customer Slices Indexers
   # us-east-1a
   salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1a or G@ec2:placement:availability_zone:us-gov-east-1a ) not moose*' test.ping --out=txt
   
   salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1a or G@ec2:placement:availability_zone:us-gov-east-1a ) not moose*' cmd.run 'sensu-agent version'
   
   date; salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1a or G@ec2:placement:availability_zone:us-gov-east-1a ) not moose*' cmd.run 'yum clean all && yum makecache fast'
   
   salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1a or G@ec2:placement:availability_zone:us-gov-east-1a ) not moose*' cmd.run 'yum --disablerepo="*" --enablerepo="msoc" list available'
   
   # us-gov-east-1b
   salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1b or G@ec2:placement:availability_zone:us-gov-east-1b ) not moose*' test.ping --out=txt
       
   salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1b or G@ec2:placement:availability_zone:us-gov-east-1b ) not moose*' cmd.run 'sensu-agent version'
   
   date; salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1b or G@ec2:placement:availability_zone:us-gov-east-1b ) not moose*' cmd.run 'yum clean all && yum makecache fast'
   
   salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1b or G@ec2:placement:availability_zone:us-gov-east-1b ) not moose*' cmd.run 'yum --disablerepo="*" --enablerepo="msoc" list available'
   
   # us-gov-east-1c
   salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1c or G@ec2:placement:availability_zone:us-gov-east-1c ) not moose*' test.ping --out=txt
   
   salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1c or G@ec2:placement:availability_zone:us-gov-east-1c ) not moose*' cmd.run 'sensu-agent version'
   
   date; salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1c or G@ec2:placement:availability_zone:us-gov-east-1c ) not moose*' cmd.run 'yum clean all && yum makecache fast'
   
   salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1c or G@ec2:placement:availability_zone:us-gov-east-1c ) not moose*' cmd.run 'yum --disablerepo="*" --enablerepo="msoc" list available'
       
   

5. Stop / Update / Reload daemon / Start agent on minions systemctl stop sensu-agent && yum update -y sensu-go-agent && systemctl daemon-reload && systemctl start sensu-agent

   # XDR Infrastructure 
   salt -C '* not ( afs* or nga* or ma-* or dc-c19* or doed* or la-c19* or bas-* or ca-c19* or frtib* or dgi* or vmray* or sensu* )' cmd.run 'sensu-agent version'
       
   date; salt -C '* not ( afs* or nga* or ma-* or dc-c19* or doed* or la-c19* or bas-* or ca-c19* or frtib* or dgi* or vmray* or sensu* )' cmd.run 'systemctl stop sensu-agent && yum update -y sensu-go-agent && systemctl daemon-reload && systemctl start sensu-agent'
   
   # LCPs
   salt -C '* not *.local not *.pvt.xdr.accenturefederalcyber.com' cmd.run 'sensu-agent version'
   
   date; salt -C '* not *.local not *.pvt.xdr.accenturefederalcyber.com' cmd.run 'systemctl stop sensu-agent && yum update -y sensu-go-agent && systemctl daemon-reload && systemctl start sensu-agent'
       
   # Customer Slices
   salt -C 'afs*local or afs*com or ma-*com or la-*com or nga*com or nga*local or dc*com or bas-*com or frtib*com or ca-c19*com or dgi*com' cmd.run 'sensu-agent version'
   
   date; salt -C 'afs*local or afs*com or ma-*com or la-*com or nga*com or nga*local or dc*com or bas-*com or frtib*com or ca-c19*com or dgi*com' cmd.run 'systemctl stop sensu-agent && yum update -y sensu-go-agent && systemctl daemon-reload && systemctl start sensu-agent'
   
   # Customer Slices Search Heads Only
   salt -C '*-sh* and not *moose* and not fm-shared-search*' cmd.run 'sensu-agent version'
       
   date; salt -C '*-sh* and not *moose* and not fm-shared-search*' cmd.run 'systemctl stop sensu-agent && yum update -y sensu-go-agent && systemctl daemon-reload && systemctl start sensu-agent'
   
   # Customer Slices Cluster masters and Heavy Forwarders 
   salt -C '( *splunk-cm* or *splunk-hf* ) not moose*' cmd.run 'sensu-agent version'
       
   date; salt -C '( *splunk-cm* or *splunk-hf* ) not moose*' cmd.run 'systemctl stop sensu-agent && yum update -y sensu-go-agent && systemctl daemon-reload && systemctl start sensu-agent'
   
   # Customer Slices Indexers
   # us-east-1a
   salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1a or G@ec2:placement:availability_zone:us-gov-east-1a ) not moose*' test.ping --out=txt
   
   date; salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1a or G@ec2:placement:availability_zone:us-gov-east-1a ) not moose*' cmd.run 'systemctl stop sensu-agent && yum update -y sensu-go-agent && systemctl daemon-reload && systemctl start sensu-agent'
   
   # us-gov-east-1b
   salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1b or G@ec2:placement:availability_zone:us-gov-east-1b ) not moose*' test.ping --out=txt
       
   date; salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1b or G@ec2:placement:availability_zone:us-gov-east-1b ) not moose*' cmd.run 'systemctl stop sensu-agent && yum update -y sensu-go-agent && systemctl daemon-reload && systemctl start sensu-agent'
   
   # us-gov-east-1c
   salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1c or G@ec2:placement:availability_zone:us-gov-east-1c ) not moose*' test.ping --out=txt
   
   date; salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1c or G@ec2:placement:availability_zone:us-gov-east-1c ) not moose*' cmd.run 'systemctl stop sensu-agent && yum update -y sensu-go-agent && systemctl daemon-reload && systemctl start sensu-agent'
   
   

6. Verify with this:

   salt '*' cmd.run 'sensu-agent version'
   salt -C '* not salt* not sensu* not jira*' cmd.run 'sensu-agent version'
   

:warning: Don't forget to un-silence Sensu.

---

Sensu Go caveats

---

In version 5.16 the default password was removed in favor of a sensu-backend init with bash variables.

Sen$uP@ssw0rd!

systemctl start sensu-backend
export SENSU_BACKEND_CLUSTER_ADMIN_USERNAME=YOUR_USERNAME
export SENSU_BACKEND_CLUSTER_ADMIN_PASSWORD=YOUR_PASSWORD
sensu-backend init
sensuctl create --file filename.json