# Sensu Upgrade Notes

### Places where Official Sensu Go code and Documentation is located
---
- Official [Sensu Go Repo Github](https://github.com/sensu/sensu-go/releases)
- Official [Sensu Go Website](https://sensu.io/)
- Official [Sensu Go Upgrade Documentation](https://docs.sensu.io/sensu-go/latest/operations/maintain-sensu/upgrade/).  
- Official Sensu Hosted Package Repo Service [Packagecloud](https://packagecloud.io/sensu/stable/)

> :warning: We will use our XDR Internal `Reposerver` for all upgrade methods - See [How to add a new package to the Reposerver](Reposerver%20Notes.md)


### Sensu Go Upgrade History
---
 - [MSOCI-1565 ticket - Upgrade Sensu to 6.2.x](https://jira.xdr.accenturefederalcyber.com/browse/MSOCI-1565)
 - [MSOCI-1908 ticket - Upgrade Sensu to 6.4.3](https://jira.xdr.accenturefederalcyber.com/browse/MSOCI-1908)
 - [MSOCI-1969 ticket - Upgrade Sensu to 6.6.1](https://jira.xdr.accenturefederalcyber.com/browse/MSOCI-1969)
 - [MSOCI-2027 ticket - Upgrade Sensu to 6.7.0](https://jira.xdr.accenturefederalcyber.com/browse/MSOCI-2027)
 - [MSOCI-2173 ticket - Upgrade Sensu to 6.7.2](https://jira.xdr.accenturefederalcyber.com/browse/MSOCI-2173)
 - [MSOCI-2244 ticket - Upgrade Sensu to 6.7.4](https://jira.xdr.accenturefederalcyber.com/browse/MSOCI-2244)

### Sensu Go Upgrade Process
---
We want to deploy the new code in iterations so that we can quickly abort deployment if we run in to any issues.  Start with `GC Test` XDR Infrastructure first.

Starting with Moose and Internal infra within `GC TEST`.  After deployment is verfied and functional, let it bake for 24-48 hrs before `GC Prod` deployment.

1. Download latest packages for `Sensu backend`, `Sensu agents`, `Sensuctl` (Sensu CLI) to `Repo server` and run `yum clean all` on `Sensu Backend` server - See [Reposerver](Reposerver%20Notes.md) notes.

2. If needed, update Salt states to ensure they are up-to-date - [Salt Upgrade Notes](Salt%20Upgrade%20Notes.md)
   ```
   salt sensu* state.sls salt_minion.minion_upgrade --output-diff test=true
   ```

> :warning: Remember to silence Sensu alerts before restarting services

3. Sensu first; Login to `GC TEST` Salt-Master and Stop Sensu services on `Sensu Backend` server; do the same process for `GC PROD` afterwards
    ```
    salt sensu* cmd.run 'systemctl stop sensu-agent'
    salt sensu* cmd.run 'systemctl stop sensu-backend'
    ```

4. Update `Sensu Backend` server
    ```
    salt sensu* cmd.run 'yum clean all && yum makecache fast'
    salt sensu* cmd.run 'yum --disablerepo="*" --enablerepo="msoc" list available'
    salt sensu* cmd.run 'yum update -y sensu-go-backend'
    salt sensu* cmd.run 'yum update -y sensu-go-cli'
    salt sensu* cmd.run 'yum update -y sensu-go-agent'
    salt sensu* cmd.run 'systemctl daemon-reload'
    ```
5. Restart the Sensu services and check the Status 
    ```
    salt sensu* cmd.run 'systemctl start sensu-backend'
    salt sensu* cmd.run 'systemctl start sensu-agent'

    salt sensu* cmd.run 'systemctl status sensu-backend'
    salt sensu* cmd.run 'systemctl status sensu-agent'
    ```
6. `GC Test` first; `GC PROD` second; From target servers; clean out the cache
    ```
    # XDR Infrastructure - be sure to note the different Salt minions to target between TEST and PROD
    salt -C '* not ( afs* or nga* or ma-* or dc-c19* or la-c19* or bas-* or ca-c19* or frtib* or dgi* or vmray* or sensu* )' cmd.run 'yum clean all && yum makecache fast'

    # From target servers; view the available packages
    salt -C '* not ( afs* or nga* or ma-* or dc-c19* or la-c19* or bas-* or ca-c19* or frtib* or dgi* or vmray* or sensu* )' cmd.run 'yum --disablerepo="*" --enablerepo="msoc" list available'

    # Customer Slices Search Heads Only
    salt -C '*-sh* and not *moose* and not fm-shared-search*' cmd.run 'yum clean all && yum makecache fast'

    salt -C '*-sh* and not *moose* and not fm-shared-search*' cmd.run 'yum --disablerepo="*" --enablerepo="msoc" list available'

    # Customer Slices Cluster masters and Heavy Forwarders 
    salt -C '( *splunk-cm* or *splunk-hf* ) not moose*' cmd.run 'yum clean all && yum makecache fast'

    salt -C '( *splunk-cm* or *splunk-hf* ) not moose*' cmd.run 'yum --disablerepo="*" --enablerepo="msoc" list available'
    
    # Customer Slices Indexers
    # us-east-1a
    salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1a or G@ec2:placement:availability_zone:us-gov-east-1a ) not moose*' test.ping --out=txt

    salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1a or G@ec2:placement:availability_zone:us-gov-east-1a ) not moose*' cmd.run 'sensu-agent version'

    salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1a or G@ec2:placement:availability_zone:us-gov-east-1a ) not moose*' cmd.run 'yum clean all && yum makecache fast'

    salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1a or G@ec2:placement:availability_zone:us-gov-east-1a ) not moose*' cmd.run 'yum --disablerepo="*" --enablerepo="msoc" list available'

    # us-gov-east-1b
    salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1b or G@ec2:placement:availability_zone:us-gov-east-1b ) not moose*' test.ping --out=txt
    
    salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1b or G@ec2:placement:availability_zone:us-gov-east-1b ) not moose*' cmd.run 'sensu-agent version'

    salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1b or G@ec2:placement:availability_zone:us-gov-east-1b ) not moose*' cmd.run 'yum clean all && yum makecache fast'

    salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1b or G@ec2:placement:availability_zone:us-gov-east-1b ) not moose*' cmd.run 'yum --disablerepo="*" --enablerepo="msoc" list available'

    # us-gov-east-1c
    salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1c or G@ec2:placement:availability_zone:us-gov-east-1c ) not moose*' test.ping --out=txt

    salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1c or G@ec2:placement:availability_zone:us-gov-east-1c ) not moose*' cmd.run 'sensu-agent version'

    salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1c or G@ec2:placement:availability_zone:us-gov-east-1c ) not moose*' cmd.run 'yum clean all && yum makecache fast'

    salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1c or G@ec2:placement:availability_zone:us-gov-east-1c ) not moose*' cmd.run 'yum --disablerepo="*" --enablerepo="msoc" list available'
    
    ```

7. Stop / Update / Reload daemon / Start agent on minions `systemctl stop sensu-agent && yum update -y sensu-go-agent && systemctl daemon-reload && systemctl start sensu-agent`
    ```
    # XDR Infrastructure 
    salt -C '* not ( afs* or nga* or ma-* or dc-c19* or la-c19* or bas-* or ca-c19* or frtib* or dgi* or vmray* or sensu* )' cmd.run 'sensu-agent version'
    
    date; salt -C '* not ( afs* or nga* or ma-* or dc-c19* or la-c19* or bas-* or ca-c19* or frtib* or dgi* or vmray* or sensu* )' cmd.run 'systemctl stop sensu-agent && yum update -y sensu-go-agent && systemctl daemon-reload && systemctl start sensu-agent'

    # LCPs
    salt -C '* not *.local not *.pvt.xdr.accenturefederalcyber.com' cmd.run 'systemctl stop sensu-agent && yum update -y sensu-go-agent && systemctl daemon-reload && systemctl start sensu-agent'
    
    date; salt -C '* not *.local not *.pvt.xdr.accenturefederalcyber.com' cmd.run 'systemctl stop sensu-agent'

    # Customer Slices
    salt -C 'afs*local or afs*com or ma-*com or la-*com or nga*com or nga*local or dc*com or bas-*com or frtib*com or ca-c19*com or dgi*com' cmd.run 'sensu-agent version'

    date; salt -C 'afs*local or afs*com or ma-*com or la-*com or nga*com or nga*local or dc*com or bas-*com or frtib*com or ca-c19*com or dgi*com' cmd.run 'systemctl stop sensu-agent && yum update -y sensu-go-agent && systemctl daemon-reload && systemctl start sensu-agent'

    # Customer Slices Search Heads Only
    date; salt -C '*-sh* and not *moose* and not fm-shared-search*' cmd.run 'systemctl stop sensu-agent && yum update -y sensu-go-agent && systemctl daemon-reload && systemctl start sensu-agent'

    # Customer Slices Cluster masters and Heavy Forwarders 
    date; salt -C '( *splunk-cm* or *splunk-hf* ) not moose*' cmd.run 'systemctl stop sensu-agent && yum update -y sensu-go-agent && systemctl daemon-reload && systemctl start sensu-agent'

    # Customer Slices Indexers
    
    # us-east-1a
    salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1a or G@ec2:placement:availability_zone:us-gov-east-1a ) not moose*' test.ping --out=txt

    date; salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1a or G@ec2:placement:availability_zone:us-gov-east-1a ) not moose*' cmd.run 'systemctl stop sensu-agent && yum update -y sensu-go-agent && systemctl daemon-reload && systemctl start sensu-agent'

    # us-gov-east-1b
    salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1b or G@ec2:placement:availability_zone:us-gov-east-1b ) not moose*' test.ping --out=txt
    
    date; salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1b or G@ec2:placement:availability_zone:us-gov-east-1b ) not moose*' cmd.run 'systemctl stop sensu-agent && yum update -y sensu-go-agent && systemctl daemon-reload && systemctl start sensu-agent'

    # us-gov-east-1c
    salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1c or G@ec2:placement:availability_zone:us-gov-east-1c ) not moose*' test.ping --out=txt

    date; salt -C '*splunk-i* and ( G@ec2:placement:availability_zone:us-east-1c or G@ec2:placement:availability_zone:us-gov-east-1c ) not moose*' cmd.run 'systemctl stop sensu-agent && yum update -y sensu-go-agent && systemctl daemon-reload && systemctl start sensu-agent'

    ```

8. Verify with this: 
    ```
    salt '*' cmd.run 'sensu-agent version'
    salt -C '* not salt* not sensu* not jira*' cmd.run 'sensu-agent version'
    ```

> :warning: Don't forget to un-silence Sensu. 

---
### Sensu Go caveats
---
In `version 5.16` the default password was removed in favor of a sensu-backend init with bash variables. 

Sen$uP@ssw0rd!

```
systemctl start sensu-backend
export SENSU_BACKEND_CLUSTER_ADMIN_USERNAME=YOUR_USERNAME
export SENSU_BACKEND_CLUSTER_ADMIN_PASSWORD=YOUR_PASSWORD
sensu-backend init
sensuctl create --file filename.json
```