# Sensu Notes.md See (Sensu Go Migration Notes.md) file for more details ## Sensu License 5/6/2021 Congrats, we are over 100 servers in Prod. We now need a license. These are the steps to apply the license. Manually write the license file to the Sensu server. Manually apply it with `sensuctl`. It is not handled by TF or Salt at the moment. Sorry I was busy! ``` sudo -i vim /etc/sensu/conf.d/sensu_license.json chown sensu: /etc/sensu/conf.d/sensu_license.json chmod 600 /etc/sensu/conf.d/sensu_license.json vim /root/sensu-password # see vault engineering/sensu/sensu-prod sensuctl configure -n --username 'admin' --password $( cat /root/sensu-password ) --namespace default --url 'https://127.0.0.1:8080' --insecure-skip-tls-verify sensuctl license info sensuctl create --file /etc/sensu/conf.d/sensu_license.json sensuctl license info --format tabular sensuctl logout echo "" > /root/sensu-password ``` ## Sensu Upgrade 08/03/2020 [Sensu Upgrade Documentation](https://docs.sensu.io/sensu-go/latest/operations/maintain-sensu/upgrade/) 1. Download latest packages for master and agents to Repo server and run `yum clean all` on Sensu Master 2. If needed, update Salt states to ensure they are up-to-date 3. Stop sensu services on sensu-master `systemctl stop sensu-agent` `systemctl stop sensu-backend` 4. Update sensu software on the sensu-master ``` yum update sensu-go-backend yum update sensu-go-cli yum update sensu-go-agent systemctl daemon-reload ``` 5. Restart the sensu services `systemctl start sensu-backend` and `systemctl start sensu-agent` 6. Run `yum clean all` on Salt minions 7. Stop agent on minion `systemctl stop sensu-agent` 8. Upgrade agent on minion `yum update sensu-go-agent -y` 9. Run this `systemctl daemon-reload` 10. Start agent `systemctl start sensu-agent` 11. Verify with this: ``` salt '*' cmd.run 'sensu-agent version' salt -C '* not salt* not sensu* not jira*' cmd.run 'sensu-agent version' ``` In version 5.16 the default password was removed in favor of a sensu-backend init with bash variables. Sen$uP@ssw0rd! ``` systemctl start sensu-backend export SENSU_BACKEND_CLUSTER_ADMIN_USERNAME=YOUR_USERNAME export SENSU_BACKEND_CLUSTER_ADMIN_PASSWORD=YOUR_PASSWORD sensu-backend init sensuctl create --file filename.json ``` ## OIDC Okta Auth --- type: oidc api_version: authentication/v2 metadata: name: oidc_okta spec: additional_scopes: - groups - email client_id: 0oa2qxyf3q0DC6Dj2297 client_secret: bpKGPQ9-ILxZI_UrcpGgT5jsmT7JT6X6LLMa3kF_ redirect_uri: https://sensu.msoc.defpoint.local:8000/api/enterprise/authentication/v2/oidc/callback server: https://mdr-multipass.okta.com groups_claim: groups groups_prefix: 'okta:' username_claim: email username_prefix: 'okta:' ``` #cluster role binding for okta sensuctl cluster-role-binding create okta --cluster-role=cluster-admin --group=okta:mdr-admins sensuctl cluster-role-binding create mdr-admin --cluster-role=mdr-admin --group=ldap:mdr-admins sensuctl cluster-role-binding list ``` type: ClusterRoleBinding api_version: core/v2 metadata: name: cluster-admin spec: role_ref: name: cluster-admin type: ClusterRole subjects: - name: okta:group type: Group Look for the JWT in the tokens URL parameter. ``` { "exp": 1596583324, "jti": "4d048b63688296361fd96b1ac7d77ef1", "sub": "okta:brad.poulton@accenturefederal.com", "groups": [ "okta:splunk-role-admin", "okta:Everyone", "okta:vault-admin", "okta:phantom-role-administrator", "okta:VPN", "okta:aws#afs-mdr-common-services#mdr_engineer_readonly#471284459109", "okta:aws-us-gov#afs-mdr-common-services-gov#mdr_engineer_readonly#701290387780", "okta:AWS - MDR_Engineer-Readonly Role", "okta:mdr-admins", "okta:VictorOps" ], "provider": { "provider_id": "oidc_okta", "provider_type": "oidc", "user_id": "00u22ymdgdKPTDyR5297" }, "api_key": false } ``` ### LDAP Auth ``` running ldap search with basedn \"ou=groups, dc=mdr-multipass, dc=okta, dc=com\" and filter \"(\u0026(objectclass=groupOfNames)(uniqueMember=uid=brad.poulton,ou=users,dc=mdr-multipass,dc=okta,dc=com))\" ldapsearch -x -H ldaps://mdr-multipass.ldap.okta.com -b dc=mdr-multipass,dc=okta,dc=com -D "uid=ldap.read@defpoint.com,dc=mdr-multipass,dc=okta,dc=com" -W ``` ``` # brad.poulton, users, mdr-multipass.okta.com dn: uid=brad.poulton,ou=users,dc=mdr-multipass,dc=okta,dc=com objectClass: top objectClass: person objectClass: organizationalPerson objectClass: inetOrgPerson uid: brad.poulton uniqueIdentifier: 00u22ymdgdKPTDyR5297 organizationalStatus: ACTIVE givenName: Brad sn: Poulton cn: Brad Poulton mail: brad.poulton@accenturefederal.com mobile: 4355126342 ``` ``` # mdr-admins, groups, mdr-multipass.okta.com dn: cn=mdr-admins,ou=groups,dc=mdr-multipass,dc=okta,dc=com objectClass: top objectClass: groupofUniqueNames cn: mdr-admins uniqueIdentifier: 00g1m5jakrmiDwISV297 uniqueMember: uid=chris.lynch,ou=users,dc=mdr-multipass,dc=okta,dc=com uniqueMember: uid=ryan.damour,ou=users,dc=mdr-multipass,dc=okta,dc=com uniqueMember: uid=duane.waddle,ou=users,dc=mdr-multipass,dc=okta,dc=com uniqueMember: uid=brad.poulton,ou=users,dc=mdr-multipass,dc=okta,dc=com ``` ``` Jan 14 23:48:51 sensu sensu-backend: {"component":"authentication/v2","level":"debug","msg":"running ldap search with basedn \"ou=groups, dc=mdr-multipass, dc=okta, dc=com\" and filter \"(\u0026(objectclass=groupOfNames)(uniqueMember=uid=brad.poulton,ou=users,dc=mdr-multipass,dc=okta,dc=com))\"","time":"2020-01-14T23:48:51Z"} ``` brad-test SensuA123 # Defragging the Database If /var starts filling up, a likely candidate is the etcd database. This can be compacted and defragged to free up space, but the tool to do so isn't installed by default. To defrag: (based off [this document](https://docs.sensu.io/sensu-go/latest/operations/maintain-sensu/troubleshoot/)) ``` sudo yum install -y etcd3 sudo bash export ETCDCTL_API=3 export ETCDCTL_CACERT=/etc/sensu/conf.d/ca.pem export ETCDCTL_ENDPOINTS="http://localhost:2379" etcdctl endpoint status # SHould outptu something like: http://localhost:2379, 8927110dc66458af, 3.3.17, 1.2 GB, true, 1122, 531838152 etcdctl defrag exit sudo yum remove etcd3 ``` If you get the error `No help topic for 'endpoint'`, then your environment variables aren't set (did you set them before you sudo'd?). If you get the error `Error: context deadline exceeded`, then your endpoint is wrong, probably `https` instead of `http`.