infrastructure-notes/Sensu Upgrade Notes.md

Sensu Upgrade Notes

Places where Official Sensu Go code and Documentation is located

• Official Sensu Go Repo Github
• Official Sensu Go Website
• Official Sensu Go Upgrade Documentation.
  
• Official Sensu Hosted Package Repo Service Packagecloud

:warning: We will use our XDR Internal Reposerver for all upgrade methods - See How to add a new package to the Reposerver

Sensu Go Upgrade History

• Jira MSOCI-1565 ticket - Upgrade Sensu to 6.2.X
• Jira MSOCI-1908 ticket - Upgrade Sensu to 6.4.3
• Jira MSOCI-1969 ticket - Upgrade Sensu to 6.6.1
• Pending MSOCI-2027 ticket - Upgrade Sensu to 6.7

We want to deploy the new code in iterations so that we can quickly abort deployment if we run in to any issues. Start with GC Test XDR Infrastructure first.

Starting with Moose and Internal infra within GC TEST. After deployment is verfied and functional, let it bake for 24-48 hrs before GC Prod deployment.

1. Download latest packages for Sensu backend, Sensu agents, Sensuctl (Sensu CLI) to Repo server and run yum clean all on Sensu Backend server - See Reposerver notes.

2. If needed, update Salt states to ensure they are up-to-date

3. Stop Sensu services on Sensu Backend server

   systemctl stop sensu-agent 
   systemctl stop sensu-backend
   

4. Update Sensu Go software on the Sensu Backend server

   yum update sensu-go-backend
   yum update sensu-go-cli
   yum update sensu-go-agent
   systemctl daemon-reload
   

5. Restart the Sensu services

   systemctl start sensu-backend
   systemctl start sensu-agent
   
   systemctl status sensu-backend
   systemctl status sensu-agent
   

6. XDR Infrastructure in GC Test first; Run yum clean all on Salt minions; then LCPs and customers

   salt -C '* not ( afs* or nga* or ma-* or dc-c19* or la-c19* or bas-* or ca-c19* or frtib* or dgi* or threatq* or qcompliance* or vmray* )' cmd.run 'yum clean all'
       
   salt -C '* not *.local not *.pvt.xdr.accenturefederalcyber.com' cmd.run 'sensu-agent version'
   

7. Verify and then Stop agent on minions systemctl stop sensu-agent

   #XDR Infrastructure
   salt -C '* not ( afs* or nga* or ma-* or dc-c19* or la-c19* or bas-* or ca-c19* or frtib* or dgi* or threatq* or qcompliance* or vmray* )' cmd.run 'sensu-agent version'
   date; salt -C '* not ( afs* or nga* or ma-* or dc-c19* or la-c19* or bas-* or ca-c19* or frtib* or dgi* or threatq* )' cmd.run 'systemctl stop sensu-agent'
   
   #LCPs
   salt -C '* not *.local not *.pvt.xdr.accenturefederalcyber.com' cmd.run 'sensu-agent version'
   date; salt -C '* not *.local not *.pvt.xdr.accenturefederalcyber.com' cmd.run 'systemctl stop sensu-agent'
   
   #Customer Slices
   salt -C 'afs*local or afs*com or ma-*com or la-*com or nga*com or nga*local or dc*com or bas-*com or frtib*com or ca-c19*com or dgi*com' cmd.run 'sensu-agent version'
   
   date; salt -C 'afs*local or afs*com or ma-*com or la-*com or nga*com or nga*local or dc*com or bas-*com or frtib*com or ca-c19*com or dgi*com' cmd.run 'systemctl stop sensu-agent'
   
   

8. Update the agent on minion yum update sensu-go-agent -y

   #XDR Infrastructure
   salt -C '* not ( afs* or nga* or ma-* or dc-c19* or la-c19* or bas-* or ca-c19* or frtib* or dgi* or threatq* )' cmd.run 'yum update sensu-go-agent -y'
   
   #LCPs
   salt -C '* not *.local not *.pvt.xdr.accenturefederalcyber.com' cmd.run 'yum update sensu-go-agent -y'
   
   #Customer Slices
   salt -C 'afs*local or afs*com or ma-*com or la-*com or nga*com or nga*local or dc*com or bas-*com or frtib*com or ca-c19*com or dgi*com' cmd.run 'yum update sensu-go-agent -y'
   

9. Reload the daemon systemctl daemon-reload

   #XDR Infrastructure
   date; salt -C '* not ( afs* or nga* or ma-* or dc-c19* or la-c19* or bas-* or ca-c19* or frtib* or dgi* or threatq* )' cmd.run 'systemctl daemon-reload'
   
   #LCPs
   date; salt -C '* not *.local not *.pvt.xdr.accenturefederalcyber.com' cmd.run 'systemctl daemon-reload'
   
   #Customer Slices
   date; salt -C 'afs*local or afs*com or ma-*com or la-*com or nga*com or nga*local or dc*com or bas-*com or frtib*com or ca-c19*com or dgi*com' cmd.run 'systemctl daemon-reload'
   

10. Start agent systemctl start sensu-agent

    #XDR Infrastructure
    date; salt -C '* not ( afs* or nga* or ma-* or dc-c19* or la-c19* or bas-* or ca-c19* or frtib* or dgi* or threatq* )' cmd.run 'systemctl start sensu-agent'
    
    #LCPs
    date; salt -C '* not *.local not *.pvt.xdr.accenturefederalcyber.com' cmd.run 'systemctl start sensu-agent'
    
    #Customer Slices
    date; salt -C 'afs*local or afs*com or ma-*com or la-*com or nga*com or nga*local or dc*com or bas-*com or frtib*com or ca-c19*com or dgi*com' cmd.run 'systemctl start sensu-agent'
    

11. Verify with this:

    salt '*' cmd.run 'sensu-agent version'
    salt -C '* not salt* not sensu* not jira*' cmd.run 'sensu-agent version'
    

In version 5.16 the default password was removed in favor of a sensu-backend init with bash variables.

Sen$uP@ssw0rd!

systemctl start sensu-backend
export SENSU_BACKEND_CLUSTER_ADMIN_USERNAME=YOUR_USERNAME
export SENSU_BACKEND_CLUSTER_ADMIN_PASSWORD=YOUR_PASSWORD
sensu-backend init
sensuctl create --file filename.json