Merge pull request #428 from mdr-engineering/feature/bp_MSOCI-2013_sensu-vault-check

Adds New Sensu Vault service Check

base/sensu-configuration/README.md

@@ -0,0 +1,43 @@
+# Sensu Configuration
+
+To use the TF Sensu provider set the password via a env variable. The username is static in terragrunt.hcl. The password is located at engineering/sensu/sensu-prod/admin
+
+in one line to apply:
+```
+SENSU_PASSWORD=blah terragrunt apply
+```
+
+OR...
+```
+SENSU_PASSWORD=
+echo $SENSU_PASSWORD
+```
+
+If you are an idiot and add an ! to the password you can escape it like this:
+
+e!password
+
+`export SENSU_PASSWORD=e\!password`
+
+
+# AWS Secrets Manager Requirement
+
+To support email from sensu, we now need an AWS secrets manager secret
+available that has the SMTP host (mailrelay) information.
+
+In AWS secrets manager, a secret named "smtp/smtp_creds" needs to be
+IN THE C2 account.  The Secret value needs to be a key/value that
+encodes as JSON like:
+
+``
+{
+  "mail_from_domain": "xdrtest.accenturefederalcyber.com",
+  "mail_host": "mailrelay2.pvt.xdrtest.accenturefederalcyber.com",
+  "mail_user": "xdrtestpostfix@pvt.xdrtest.accenturefederalcyber.com",
+  "mail_password": "the_password_in_salt"
+}
+```
+
+If you don't set this up BEFORE trying to apply this terraform, a bad time
+will be had.
+

base/sensu-configuration/checks.tf

@@ -267,3 +267,14 @@ resource "sensu_check" "check_splunk_service" {
   publish        = "true"
   runtime_assets = ["sensu-plugins-process-checks", "sensu-ruby-runtime", ]
 }
+
+resource "sensu_check" "check_vault_service" {
+  name           = "vault_service"
+  command        = "check-process.rb -p vault"
+  namespace      = "default"
+  subscriptions  = ["vault_service", ]
+  handlers       = local.default_handlers
+  interval       = 60
+  publish        = "true"
+  runtime_assets = ["sensu-plugins-process-checks", "sensu-ruby-runtime", ]
+}

base/sensu/README.md

@@ -0,0 +1,32 @@
+# Sensu Setup Steps
+
+NOTE: If you are recreating the Sensu server, don't forget to destroy the sensu-configuration first THEN destroy the instance-sensu! This will clear out the terraform state. If you forget you can use these terragrunt commands to remove the configurations from the state and then reapply them. 
+```
+terragrunt state list
+terragrunt state list > deleteme
+for line in `cat deleteme`; do terragrunt state rm $line; done
+```
+
+- Run terraform in 095-instance-sensu to spin up the server and other resources
+- Ensure the RPMs are located on the reposerver
+- Run prep Salt states
+    - `salt sensu*com saltutil.sync_all`
+    - `salt sensu*com saltutil.pillar_refresh`
+    - `salt sensu*com state.sls os_modifications`
+    - `salt sensu*com state.highstate`
+- Run Salt state sensu_master to install sensu from reposerver
+    - `salt sensu*com state.sls sensu_master`
+- Run post-installation commands as root user on Sensu server to setup the backend
+    - Get the password from Vault to input
+    - Init the backend `sensu-backend init --interactive`
+    - prep to login `vim sensu-password`
+    - Log in via sensuctl `sensuctl configure -n --username 'admin' --password $( cat sensu-password ) --namespace default --url 'https://127.0.0.1:8080' --insecure-skip-tls-verify`
+    - Setup OIDC auth ( no TF support for this call ) `sensuctl create -f /etc/sensu/conf.d/oidc-okta.yml`
+    - Opt-out of stats `sensuctl tessen opt-out --skip-confirm`
+    - Apply License file ( see Sensu Notes.md )
+    - Clean up password `echo "" > sensu-password`
+    - Log out of sensuctl `sensuctl logout`
+- Run Terraform in 096-sensu-configuration to configure the software
+    - On your laptop, put the sensu password in the SENSU_PASSWORD environment var 
+    - `export SENSU_PASSWORD='password'`
+

base/vault-configuration/README.md

@@ -0,0 +1 @@
+../vault/README.md

base/vault/README.md

@@ -0,0 +1,51 @@
+# Vault Setup Steps
+
+- Run the 090-instance-vault module first
+  - `terragrunt apply`
+- Run these salt states on the Vault instances
+  - `salt vault*com saltutil.sync_all`
+  - `salt vault*com saltutil.refresh_pillar`
+  - `salt vault*com state.sls os_modifications test=true --state-output=changes`
+  - `salt vault*com state.highstate test=true --state-output=changes`
+  - `salt vault*com state.sls salt_minion.salt_minion_proxy test=true --state-output=changes`
+  - `salt vault*com state.sls vault test=true --state-output=changes`
+  - `salt vault*com state.sls vault pillar='{"kms_key_id": "<new-kms-key>"}' test=true --state-output=changes`
+- Ensure Vault is running and unseal the Vault via CLI on ONE of the Vault servers
+  - `export VAULT_ADDR=https://127.0.0.1`
+  - `export VAULT_SKIP_VERIFY=1`
+  - `vault status`
+  - `vault operator init -recovery-shares=5 -recovery-threshold=2`
+  - Copy root token and 5 recovery shares to safe place
+- On your Laptop, use the root Vault token to run the Terraform 099-vault-configuration module
+  - `vim ~/.vault-token`
+  - The DNS in the Terraform Vault provider will be used. To test connectivity be sure to run this command: `export VAULT_ADDR=https://internal-vault-alb-test-778772793.us-gov-east-1.elb.amazonaws.com`. Replace the ALB address with the current ALB DNS address. This is due to Golang DNS not updating when connected to XDR over VPN. Vault binary is written in Go.  
+  - Why not use the accenturefederal address? This is due to golang DNS issues.
+  - Download the vault binary ( might not be needed? Is the binary in the TF provider? )
+  - `terragrunt apply`
+  - Might need this command as well. See main.tf in module for more information.
+  - `vault write auth/aws/config/client sts_endpoint=https://sts.us-gov-east-1.amazonaws.com sts_region=us-gov-east-1`
+  - The Terraform Vault provider will look at ~/.vault-token for the token and the bash variables for the address to connect to.
+- Revoke the root token
+  - `vault token revoke <root-token>`
+- Distribute Vault shareds to the appriopriate individuals. 
+- Export/Import secrets
+  - https://github.com/adamdecaf/vault-backend-migrator 
+
+For additional Vault documentation see these locations:
+
+https://github.mdr.defpoint.com/mdr-engineering/infrastructure-notes/blob/master/Vault%20Notes.md
+
+https://github.mdr.defpoint.com/mdr-engineering/msoc-infrastructure/blob/develop/salt/fileroots/vault/README.md
+
+## To apply without putting your junk in the terragrunt.hcl
+
+THIS SECTION INCOMPLETE
+
+1. `VAULT_TOKEN`: Login to vault, and click your name dropdown, then copy secret. You'll need it more than once.
+1. `TF_VAR_okta_api_token`: Find this in the prod vault (even for test)
+1. `TF_VAR_okta_oidc_client_secret`: Find this in the prod vault (even for test)
+
+Then apply:
+```
+VAULT_TOKEN=<fromvault> TF_VAR_okta_api_token=YOURTOKENHERE TF_VAR_okta_oidc_client_secret=YOURSECRETHERE terragrunt apply
+```