infrastructure-notes/Sensu Go Upgrade Notes.md

Sensu Upgrade Notes

Places where Official Sensu Go code and Documentation is located

---

• Official Sensu Go Repo Github
• Official Sensu Go Website
• Official Sensu Go Upgrade Documentation.
  
• Official Sensu Hosted Package Repo Service Packagecloud

:warning: We will use our XDR Internal Reposerver for all upgrade methods - See How to add a new package to the Reposerver

Sensu Go Upgrade History

---

• MSOCI-1565 ticket - Upgrade Sensu to 6.2.x
• MSOCI-1908 ticket - Upgrade Sensu to 6.4.3
• MSOCI-1969 ticket - Upgrade Sensu to 6.6.1
• MSOCI-2027 ticket - Upgrade Sensu to 6.7.0
• MSOCI-2173 ticket - Upgrade Sensu to 6.7.2

Sensu Go Upgrade Process

---

We want to deploy the new code in iterations so that we can quickly abort deployment if we run in to any issues. Start with GC Test XDR Infrastructure first.

Starting with Moose and Internal infra within GC TEST. After deployment is verfied and functional, let it bake for 24-48 hrs before GC Prod deployment.

1. Download latest packages for Sensu backend, Sensu agents, Sensuctl (Sensu CLI) to Repo server and run yum clean all on Sensu Backend server - See Reposerver notes.

2. If needed, update Salt states to ensure they are up-to-date

:warning: Remember to silence Sensu alerts before restarting services

1. Sensu first; Login to GC TEST Salt-Master and Stop Sensu services on Sensu Backend server; do the same process for GC PROD afterwards

   salt -C sensu* cmd.run 'systemctl stop sensu-agent'
   salt -C sensu* cmd.run 'systemctl stop sensu-backend'
   

2. Update Sensu Backend server

   salt -C sensu* cmd.run 'yum update -y sensu-go-backend'
   salt -C sensu* cmd.run 'yum update -y sensu-go-cli'
   salt -C sensu* cmd.run 'yum update -y sensu-go-agent'
   salt -C sensu* cmd.run 'systemctl daemon-reload'
   

3. Restart the Sensu services and check the Status

   salt -C sensu* cmd.run 'systemctl start sensu-backend'
   salt -C sensu* cmd.run 'systemctl start sensu-agent'
   
   salt -C sensu* cmd.run 'systemctl status sensu-backend'
   salt -C sensu* cmd.run 'systemctl status sensu-agent'
   

4. GC Test first; GC PROD second; From target servers; clean out the cache

   # XDR Infrastructure - be sure to note the different Salt minions to target between TEST and PROD
   salt -C '* not ( afs* or nga* or ma-* or dc-c19* or la-c19* or bas-* or ca-c19* or frtib* or dgi* or threatq* or vmray* or sensu* or rhsso-0* or fm-shared-search-0* or modelclient-splunk-idx-326* or modelclient-splunk-idx-8b8* or moose-splunk-idx-eed* )' cmd.run 'yum clean all && yum makecache fast'
   
   # From target servers; view the available packages
   salt -C '* not ( afs* or nga* or ma-* or dc-c19* or la-c19* or bas-* or ca-c19* or frtib* or dgi* or threatq* or vmray* or sensu* or rhsso-0* or fm-shared-search-0* or modelclient-splunk-idx-326* or modelclient-splunk-idx-8b8* or moose-splunk-idx-eed* )' cmd.run 'yum --disablerepo="*" --enablerepo="msoc" list available'
   
   # Customer Slices Search Heads Only
   salt -C '*-sh* and not *moose* and not fm-shared-search*' cmd.run 'yum clean all && yum makecache fast'
   
   salt -C '*-sh* and not *moose* and not fm-shared-search*' cmd.run 'yum --disablerepo="*" --enablerepo="msoc" list available'
   
   # Customer Slices Cluster masters and Heavy Forwarders 
   salt -C '( *splunk-cm* or *splunk-hf* ) not moose*' cmd.run 'yum clean all && yum makecache fast'
   
   salt -C '( *splunk-cm* or *splunk-hf* ) not moose*' cmd.run 'yum --disablerepo="*" --enablerepo="msoc" list available'
       
   # Customer Slices Indexers
   # us-east-1a
   salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1a or G@ec2:placement:availability_zone:us-gov-east-1a ) not moose*' test.ping --out=txt
   
   salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1a or G@ec2:placement:availability_zone:us-gov-east-1a ) not moose*' cmd.run 'sensu-agent version'
   
   salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1a or G@ec2:placement:availability_zone:us-gov-east-1a ) not moose*' cmd.run 'yum clean all && yum makecache fast'
   
   salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1a or G@ec2:placement:availability_zone:us-gov-east-1a ) not moose*' cmd.run 'yum --disablerepo="*" --enablerepo="msoc" list available'
   
   # us-gov-east-1b
   salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1b or G@ec2:placement:availability_zone:us-gov-east-1b ) not moose*' test.ping --out=txt
       
   salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1b or G@ec2:placement:availability_zone:us-gov-east-1b ) not moose*' cmd.run 'sensu-agent version'
   
   salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1b or G@ec2:placement:availability_zone:us-gov-east-1b ) not moose*' cmd.run 'yum clean all && yum makecache fast'
   
   salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1b or G@ec2:placement:availability_zone:us-gov-east-1b ) not moose*' cmd.run 'yum --disablerepo="*" --enablerepo="msoc" list available'
   
   # us-gov-east-1c
   salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1c or G@ec2:placement:availability_zone:us-gov-east-1c ) not moose*' test.ping --out=txt
   
   salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1c or G@ec2:placement:availability_zone:us-gov-east-1c ) not moose*' cmd.run 'sensu-agent version'
   
   salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1c or G@ec2:placement:availability_zone:us-gov-east-1c ) not moose*' cmd.run 'yum clean all && yum makecache fast'
   
   salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1c or G@ec2:placement:availability_zone:us-gov-east-1c ) not moose*' cmd.run 'yum --disablerepo="*" --enablerepo="msoc" list available'
       
   

5. Verify and then Stop agent on minions systemctl stop sensu-agent

   # XDR Infrastructure 
   salt -C '* not ( afs* or nga* or ma-* or dc-c19* or la-c19* or bas-* or ca-c19* or frtib* or dgi* or threatq* or vmray* or sensu* or rhsso-0* or fm-shared-search-0* or modelclient-splunk-idx-326* or modelclient-splunk-idx-8b8* or moose-splunk-idx-eed* )' cmd.run 'sensu-agent version'
       
   date; salt -C '* not ( afs* or nga* or ma-* or dc-c19* or la-c19* or bas-* or ca-c19* or frtib* or dgi* or threatq* or vmray* or sensu* or rhsso-0* or fm-shared-search-0* or modelclient-splunk-idx-326* or modelclient-splunk-idx-8b8* or moose-splunk-idx-eed* )' cmd.run 'systemctl stop sensu-agent'
   
   # LCPs
   salt -C '* not *.local not *.pvt.xdr.accenturefederalcyber.com' cmd.run 'sensu-agent version'
       
   date; salt -C '* not *.local not *.pvt.xdr.accenturefederalcyber.com' cmd.run 'systemctl stop sensu-agent'
   
   # Customer Slices
   salt -C 'afs*local or afs*com or ma-*com or la-*com or nga*com or nga*local or dc*com or bas-*com or frtib*com or ca-c19*com or dgi*com' cmd.run 'sensu-agent version'
   
   date; salt -C 'afs*local or afs*com or ma-*com or la-*com or nga*com or nga*local or dc*com or bas-*com or frtib*com or ca-c19*com or dgi*com' cmd.run 'systemctl stop sensu-agent'
   
   # Customer Slices Search Heads Only
   date; salt -C '*-sh* and not *moose* and not fm-shared-search*' cmd.run 'systemctl stop sensu-agent'
   
   # Customer Slices Cluster masters and Heavy Forwarders 
   date; salt -C '( *splunk-cm* or *splunk-hf* ) not moose*' cmd.run 'systemctl stop sensu-agent'
   
   # Customer Slices Indexers
       
   # us-east-1a
   salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1a or G@ec2:placement:availability_zone:us-gov-east-1a ) not moose*' test.ping --out=txt
   
   date; salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1a or G@ec2:placement:availability_zone:us-gov-east-1a ) not moose*' cmd.run 'systemctl stop sensu-agent'
   
   # us-gov-east-1b
   salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1b or G@ec2:placement:availability_zone:us-gov-east-1b ) not moose*' test.ping --out=txt
       
   date; salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1b or G@ec2:placement:availability_zone:us-gov-east-1b ) not moose*' cmd.run 'systemctl stop sensu-agent'
   
   # us-gov-east-1c
   salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1c or G@ec2:placement:availability_zone:us-gov-east-1c ) not moose*' test.ping --out=txt
   
   date; salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1c or G@ec2:placement:availability_zone:us-gov-east-1c ) not moose*' cmd.run 'systemctl stop sensu-agent'
   
   

6. Update the agent on minion yum update -y sensu-go-agent

   # XDR Infrastructure
   date; salt -C '* not ( afs* or nga* or ma-* or dc-c19* or la-c19* or bas-* or ca-c19* or frtib* or dgi* or threatq* or vmray* or sensu* or rhsso-0* or fm-shared-search-0* or modelclient-splunk-idx-326* or modelclient-splunk-idx-8b8* or moose-splunk-idx-eed* )' cmd.run 'yum update -y sensu-go-agent'
   
   # LCPs
   date; salt -C '* not *.local not *.pvt.xdr.accenturefederalcyber.com' cmd.run 'yum update -y sensu-go-agent'
   
   # Customer Slices
   salt -C 'afs*local or afs*com or ma-*com or la-*com or nga*com or nga*local or dc*com or bas-*com or frtib*com or ca-c19*com or dgi*com' cmd.run 'sensu-agent version'
   
   date; salt -C 'afs*local or afs*com or ma-*com or la-*com or nga*com or nga*local or dc*com or bas-*com or frtib*com or ca-c19*com or dgi*com' cmd.run 'systemctl stop sensu-agent'
   
   # Customer Slices Search Heads Only
   salt -C '*-sh* and not *moose* and not fm-shared-search*' cmd.run 'yum update -y sensu-go-agent'
   
   # Customer Slices Cluster masters and Heavy Forwarders 
   salt -C '( *splunk-cm* or *splunk-hf* ) not moose*' cmd.run 'yum update -y sensu-go-agent'
   
   # Customer Slices Indexers
       
   # us-east-1a
   salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1a or G@ec2:placement:availability_zone:us-gov-east-1a ) not moose*' cmd.run 'yum update -y sensu-go-agent'
   
   # us-gov-east-1b
   salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1b or G@ec2:placement:availability_zone:us-gov-east-1b ) not moose*' cmd.run 'yum update -y sensu-go-agent'
   
   # us-gov-east-1c
   salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1c or G@ec2:placement:availability_zone:us-gov-east-1c ) not moose*' cmd.run 'yum update -y sensu-go-agent'
   
   

7. Reload the daemon systemctl daemon-reload

   # XDR Infrastructure
   date; salt -C '* not ( afs* or nga* or ma-* or dc-c19* or la-c19* or bas-* or ca-c19* or frtib* or dgi* or threatq* or vmray* or sensu* or rhsso-0* or fm-shared-search-0* or modelclient-splunk-idx-326* or modelclient-splunk-idx-8b8* or moose-splunk-idx-eed* )' cmd.run 'systemctl daemon-reload'
   
   # LCPs
   date; salt -C '* not *.local not *.pvt.xdr.accenturefederalcyber.com' cmd.run 'systemctl daemon-reload'
   
   # Customer Slices Search Heads Only
   date; salt -C '*-sh* and not *moose* and not fm-shared-search*' cmd.run 'systemctl daemon-reload'
   
   # Customer Slices Cluster masters and Heavy Forwarders 
   date; salt -C '( *splunk-cm* or *splunk-hf* ) not moose*' cmd.run 'systemctl daemon-reload'
   
   # Customer Slices Indexers
   # us-east-1a
   date; salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1a or G@ec2:placement:availability_zone:us-gov-east-1a ) not moose*' cmd.run 'systemctl daemon-reload'
   
   # us-gov-east-1b
   date; salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1b or G@ec2:placement:availability_zone:us-gov-east-1b ) not moose*' cmd.run 'systemctl daemon-reload'
   
   # us-gov-east-1c
   date; salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1c or G@ec2:placement:availability_zone:us-gov-east-1c ) not moose*' cmd.run 'systemctl daemon-reload'
   
   

8. Start agent systemctl start sensu-agent

   # XDR Infrastructure
   date; salt -C '* not ( afs* or nga* or ma-* or dc-c19* or la-c19* or bas-* or ca-c19* or frtib* or dgi* or threatq* or vmray* or sensu* or rhsso-0* or fm-shared-search-0* or modelclient-splunk-idx-326* or modelclient-splunk-idx-8b8* or moose-splunk-idx-eed* )' cmd.run 'systemctl start sensu-agent'
   
   # LCPs
   date; salt -C '* not *.local not *.pvt.xdr.accenturefederalcyber.com' cmd.run 'systemctl start sensu-agent'
   
   # Customer Slices Search Heads Only
   date; salt -C '*-sh* and not *moose* and not fm-shared-search*' cmd.run 'systemctl start sensu-agent'
   
   # Customer Slices Cluster masters and Heavy Forwarders 
   date; salt -C '( *splunk-cm* or *splunk-hf* ) not moose*' cmd.run 'systemctl start sensu-agent'
   
   # Customer Slices Indexers
   # us-east-1a
   date; salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1a or G@ec2:placement:availability_zone:us-gov-east-1a ) not moose*' cmd.run 'systemctl start sensu-agent'
   
   # us-gov-east-1b
   date; salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1b or G@ec2:placement:availability_zone:us-gov-east-1b ) not moose*' cmd.run 'systemctl start sensu-agent'
   
   # us-gov-east-1c
   date; salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1c or G@ec2:placement:availability_zone:us-gov-east-1c ) not moose*' cmd.run 'systemctl start sensu-agent'
       
   

9. Verify with this:

   salt '*' cmd.run 'sensu-agent version'
   salt -C '* not salt* not sensu* not jira*' cmd.run 'sensu-agent version'
   

:warning: Don't forget to un-silence Sensu.

---

Sensu Go caveats

---

In version 5.16 the default password was removed in favor of a sensu-backend init with bash variables.

Sen$uP@ssw0rd!

systemctl start sensu-backend
export SENSU_BACKEND_CLUSTER_ADMIN_USERNAME=YOUR_USERNAME
export SENSU_BACKEND_CLUSTER_ADMIN_PASSWORD=YOUR_PASSWORD
sensu-backend init
sensuctl create --file filename.json