infrastructure-notes/Sensu Go Upgrade Notes.md

Sensu Upgrade Notes

Places where Official Sensu Go code and Documentation is located

---

• Official Sensu Go Repo Github
• Official Sensu Go Website
• Official Sensu Go Upgrade Documentation.
  
• Official Sensu Hosted Package Repo Service Packagecloud

:warning: We will use our XDR Internal Reposerver for all upgrade methods - See How to add a new package to the Reposerver

Sensu Go Upgrade History

---

• MSOCI-1565 ticket - Upgrade Sensu to 6.2.x
• MSOCI-1908 ticket - Upgrade Sensu to 6.4.3
• MSOCI-1969 ticket - Upgrade Sensu to 6.6.1
• MSOCI-2027 ticket - Upgrade Sensu to 6.7.0

Sensu Go Upgrade Process

---

We want to deploy the new code in iterations so that we can quickly abort deployment if we run in to any issues. Start with GC Test XDR Infrastructure first.

Starting with Moose and Internal infra within GC TEST. After deployment is verfied and functional, let it bake for 24-48 hrs before GC Prod deployment.

1. Download latest packages for Sensu backend, Sensu agents, Sensuctl (Sensu CLI) to Repo server and run yum clean all on Sensu Backend server - See Reposerver notes.

2. If needed, update Salt states to ensure they are up-to-date

:warning: Remember to silence Sensu alerts before restarting services

1. Login to GC TEST Salt-Master and Stop Sensu services on Sensu Backend server; do the same process for GC PROD afterwards

   salt -C sensu* cmd.run 'systemctl stop sensu-agent'
   salt -C sensu* cmd.run 'systemctl stop sensu-backend'
   

2. Update Sensu Go software on the Sensu Backend server

   salt -C sensu* cmd.run 'yum update -y sensu-go-backend'
   salt -C sensu* cmd.run 'yum update -y sensu-go-cli'
   salt -C sensu* cmd.run 'yum update -y sensu-go-agent'
   salt -C sensu* cmd.run 'systemctl daemon-reload'
   

3. Restart the Sensu services and check the Status

   salt -C sensu* cmd.run 'systemctl start sensu-backend'
   salt -C sensu* cmd.run 'systemctl start sensu-agent'
   
   salt -C sensu* cmd.run 'systemctl status sensu-backend'
   salt -C sensu* cmd.run 'systemctl status sensu-agent'
   

4. XDR Infrastructure in GC Test first; Run yum clean all on Salt minions; then LCPs and customers

   salt -C '* not ( afs* or nga* or ma-* or dc-c19* or la-c19* or bas-* or ca-c19* or frtib* or dgi* or threatq* or qcompliance* or vmray* or resolver-vmray* or sensu* )' cmd.run 'yum clean all'
       
   salt -C '* not *.local not *.pvt.xdr.accenturefederalcyber.com' cmd.run 'sensu-agent version'
   

5. Verify and then Stop agent on minions systemctl stop sensu-agent

   # XDR Infrastructure
   salt -C '* not ( afs* or nga* or ma-* or dc-c19* or la-c19* or bas-* or ca-c19* or frtib* or dgi* or threatq* or qcompliance* or vmray* or resolver-vmray* or sensu* )' cmd.run 'sensu-agent version'
       
   date; salt -C '* not ( afs* or nga* or ma-* or dc-c19* or la-c19* or bas-* or ca-c19* or frtib* or dgi* or threatq* or qcompliance* or vmray* or resolver-vmray* or sensu* )' cmd.run 'systemctl stop sensu-agent'
   
   # LCPs
   salt -C '* not *.local not *.pvt.xdr.accenturefederalcyber.com' cmd.run 'sensu-agent version'
   date; salt -C '* not *.local not *.pvt.xdr.accenturefederalcyber.com' cmd.run 'systemctl stop sensu-agent'
   
   # Customer Slices
   salt -C 'afs*local or afs*com or ma-*com or la-*com or nga*com or nga*local or dc*com or bas-*com or frtib*com or ca-c19*com or dgi*com' cmd.run 'sensu-agent version'
   
   date; salt -C 'afs*local or afs*com or ma-*com or la-*com or nga*com or nga*local or dc*com or bas-*com or frtib*com or ca-c19*com or dgi*com' cmd.run 'systemctl stop sensu-agent'
   
   

6. Update the agent on minion yum update -y sensu-go-agent

   # XDR Infrastructure
   date; salt -C '* not ( afs* or nga* or ma-* or dc-c19* or la-c19* or bas-* or ca-c19* or frtib* or dgi* or threatq* or qcompliance* or vmray* or resolver-vmray* or sensu* )' cmd.run 'yum update -y sensu-go-agent'
   
   # LCPs
   salt -C '* not *.local not *.pvt.xdr.accenturefederalcyber.com' cmd.run 'yum update -y sensu-go-agent'
   
   # Customer Slices
   salt -C 'afs*local or afs*com or ma-*com or la-*com or nga*com or nga*local or dc*com or bas-*com or frtib*com or ca-c19*com or dgi*com' cmd.run 'yum update -y sensu-go-agent'
   

7. Reload the daemon systemctl daemon-reload

   # XDR Infrastructure
   date; salt -C '* not ( afs* or nga* or ma-* or dc-c19* or la-c19* or bas-* or ca-c19* or frtib* or dgi* or threatq* or qcompliance* or vmray* or resolver-vmray* or sensu* )' cmd.run 'systemctl daemon-reload'
   
   # LCPs
   date; salt -C '* not *.local not *.pvt.xdr.accenturefederalcyber.com' cmd.run 'systemctl daemon-reload'
   
   # Customer Slices
   date; salt -C 'afs*local or afs*com or ma-*com or la-*com or nga*com or nga*local or dc*com or bas-*com or frtib*com or ca-c19*com or dgi*com' cmd.run 'systemctl daemon-reload'
   

8. Start agent systemctl start sensu-agent

   # XDR Infrastructure
   date; salt -C '* not ( afs* or nga* or ma-* or dc-c19* or la-c19* or bas-* or ca-c19* or frtib* or dgi* or threatq* or qcompliance* or vmray* or resolver-vmray* or sensu* )' cmd.run 'systemctl start sensu-agent'
   
   # LCPs
   date; salt -C '* not *.local not *.pvt.xdr.accenturefederalcyber.com' cmd.run 'systemctl start sensu-agent'
   
   # Customer Slices
   date; salt -C 'afs*local or afs*com or ma-*com or la-*com or nga*com or nga*local or dc*com or bas-*com or frtib*com or ca-c19*com or dgi*com' cmd.run 'systemctl start sensu-agent'
   

9. Verify with this:

   salt '*' cmd.run 'sensu-agent version'
   salt -C '* not salt* not sensu* not jira*' cmd.run 'sensu-agent version'
   

:warning: Don't forget to un-silence Sensu.

Sensu Go caveats

---

In version 5.16 the default password was removed in favor of a sensu-backend init with bash variables.

Sen$uP@ssw0rd!

systemctl start sensu-backend
export SENSU_BACKEND_CLUSTER_ADMIN_USERNAME=YOUR_USERNAME
export SENSU_BACKEND_CLUSTER_ADMIN_PASSWORD=YOUR_PASSWORD
sensu-backend init
sensuctl create --file filename.json