Home Esplora Aiuto
Registrati Accedi
AFS
/
infrastructure-notes
2
0
Forka 0
File Problemi 0 Pull Requests 0 Wiki
Albero (Tree): 3393d9fc5d
Rami (Branch) Tag
infrastructure-...
/
Sensu Upgrade 6.3 -> 6.4.3 Notes.md

Sensu Upgrade 6.3 -> 6.4.3 Notes.md 7.3 KB
Cronologia Originale

Sensu Upgrade 6.3 -> 6.4.3 Notes

Places where code might need to be upgraded for a new version

:warning: We will use our XDR Internal Reposerver for all upgrade methods - See How to add a new package to the Reposerver

Sensu Go Upgrade to 6.4.3

Jira MSOCI-1908 ticket - Upgrade Sensu to 6.4.3

Initial Ticket:

The latest major release of Sensu Go is version 6.4.3.  This patch adds a backend configuration attribute that allows parallel event log encoding, as well as two summary metrics for the metrics API endpoint. 

Sensu Go version 6.4 includes a number of feature improvements and important bug fixes. They upgraded the embedded etcd from version 3.3 to 3.5 for improved stability and security.

Most notable changes / fixes since 6.3:
1. etcd security upgrade - 6.4.3 includes Upgraded etcd version from 3.3.22 to 3.5.0

      * etcd versions <= 3.4.9 have CVE-2020-15114 as the highest vulnerability:

          a. https://github.com/etcd-io/etcd/security/advisories/GHSA-2xhq-gv6c-p224

          b. https://nvd.nist.gov/vuln/detail/CVE-2020-15114

2. Golang security upgrade - Upgraded Go version from 1.13.15 to 1.16.5 which entails patching multiple vulns

    * Go versions <=1.16.5 have multiple HIGH CVEs - https://groups.google.com/g/golang-announce/c/RgCMkAEQjSI

         a. https://nvd.nist.gov/vuln/detail/CVE-2021-33198

         b. https://nvd.nist.gov/vuln/detail/CVE-2021-33198

         c. https://nvd.nist.gov/vuln/detail/CVE-2021-33197

         d. https://nvd.nist.gov/vuln/detail/CVE-2021-33196

Ticket conclusion: Sensu Go GC Test and Prod env had been upgraded 100% to 6.4.3

[Conclusion]:

    GC Test and Prod envs are both running Sensu Go 6.4.3 at both the 'backend', 'cli' and 'agents' from Sensu to 'entities'. 
    We have not observed any issues within the XDR env pertaining to this upgrade after break-fix was made.
    Sensu Go 6.4.3 has introduced some newer features within the GUI that we are currently exploring usage of.

Sensu Go 6.4 Upgrade Documentation.
We want to deploy the new code in iterations so that we can quickly abort deployment if we run in to any issues. Start with GC Test XDR Infrastructure first.

Starting with Moose and Internal infra within GC TEST. After deployment is verfied and functional, let it bake for 24-48 hrs before GC Prod deployment.

  1. Download latest packages for Sensu backend, Sensu agents, Sensuctl (Sensu CLI) to Repo server and run yum clean all on Sensu Backend server - See Reposerver notes.

  2. If needed, update Salt states to ensure they are up-to-date

  3. Stop Sensu services on Sensu Backend server

    systemctl stop sensu-agent 
systemctl stop sensu-backend

  4. Update Sensu Go software on the Sensu Backend server

    yum update sensu-go-backend
yum update sensu-go-cli
yum update sensu-go-agent
systemctl daemon-reload

  5. Restart the Sensu services

    systemctl start sensu-backend
systemctl start sensu-agent

  6. XDR Infrastructure first; Run yum clean all on Salt minions; then LCPs and customers

    salt -C '* not ( afs* or nga* or ma-* or dc-c19* or la-c19* or bp-ot-demo* or bas-* or doed* or ca-c19* or frtib* or dgi* or threatq* )' cmd.run 'yum clean all'
salt -C '* not *.local not *.pvt.xdr.accenturefederalcyber.com' cmd.run 'sensu-agent version'

  7. Verify and then Stop agent on minions systemctl stop sensu-agent

    #XDR Infrastructure
salt -C '* not ( afs* or nga* or ma-* or dc-c19* or la-c19* or bp-ot-demo* or bas-* or doed* or ca-c19* or frtib* or dgi* or threatq* )' cmd.run 'sensu-agent version'
salt -C '* not ( afs* or nga* or ma-* or dc-c19* or la-c19* or bp-ot-demo* or bas-* or doed* or ca-c19* or frtib* or dgi* or threatq* )' cmd.run 'systemctl stop sensu-agent'

#LCPS
salt -C '* not *.local not *.pvt.xdr.accenturefederalcyber.com' cmd.run 'sensu-agent version'
salt -C '* not *.local not *.pvt.xdr.accenturefederalcyber.com' cmd.run 'systemctl stop sensu-agent'

#Customer Slices
salt -C 'afs*local or afs*com or ma-*com or la-*com or nga*com or nga*local or dc*com or bp-ot-demo*com or bas-*com or doed*com or frtib*com or ca-c19*com or dgi*com' cmd.run 'sensu-agent version'

salt -C 'afs*local or afs*com or ma-*com or la-*com or nga*com or nga*local or dc*com or bp-ot-demo*com or bas-*com or doed*com or frtib*com or ca-c19*com or dgi*com' cmd.run 'systemctl stop sensu-agent'

  8. Update the agent on minion yum update sensu-go-agent -y

    #XDR Infrastructure
salt -C '* not ( afs* or nga* or ma-* or dc-c19* or la-c19* or bp-ot-demo* or bas-* or doed* or ca-c19* or frtib* or dgi* or threatq* )' cmd.run 'yum update sensu-go-agent -y'

#LCPs
salt -C '* not *.local not *.pvt.xdr.accenturefederalcyber.com' cmd.run 'yum update sensu-go-agent -y'

#Customer Slices
salt -C 'afs*local or afs*com or ma-*com or la-*com or nga*com or nga*local or dc*com or bp-ot-demo*com or bas-*com or doed*com or frtib*com or ca-c19*com or dgi*com' cmd.run 'yum update sensu-go-agent -y'

  9. Reload the daemon systemctl daemon-reload

    #XDR Infrastructure
salt -C '* not ( afs* or nga* or ma-* or dc-c19* or la-c19* or bp-ot-demo* or bas-* or doed* or ca-c19* or frtib* or dgi* or threatq* )' cmd.run 'systemctl daemon-reload'

#LCPs
salt -C '* not *.local not *.pvt.xdr.accenturefederalcyber.com' cmd.run 'systemctl daemon-reload'

#Customer Slices
salt -C 'afs*local or afs*com or ma-*com or la-*com or nga*com or nga*local or dc*com or bp-ot-demo*com or bas-*com or doed*com or frtib*com or ca-c19*com or dgi*com' cmd.run 'systemctl daemon-reload'

  10. Start agent systemctl start sensu-agent

    #XDR Infrastructure
salt -C '* not ( afs* or nga* or ma-* or dc-c19* or la-c19* or bp-ot-demo* or bas-* or doed* or ca-c19* or frtib* or dgi* or threatq* )' cmd.run 'systemctl start sensu-agent'

#LCPs
salt -C '* not *.local not *.pvt.xdr.accenturefederalcyber.com' cmd.run 'systemctl start sensu-agent'

#Customer Slices
salt -C 'afs*local or afs*com or ma-*com or la-*com or nga*com or nga*local or dc*com or bp-ot-demo*com or bas-*com or doed*com or frtib*com or ca-c19*com or dgi*com' cmd.run 'systemctl start sensu-agent'

  11. Verify with this:

    salt '*' cmd.run 'sensu-agent version'
salt -C '* not salt* not sensu* not jira*' cmd.run 'sensu-agent version'

In version 5.16 the default password was removed in favor of a sensu-backend init with bash variables.

Sen$uP@ssw0rd!

systemctl start sensu-backend
export SENSU_BACKEND_CLUSTER_ADMIN_USERNAME=YOUR_USERNAME
export SENSU_BACKEND_CLUSTER_ADMIN_PASSWORD=YOUR_PASSWORD
sensu-backend init
sensuctl create --file filename.json