Vault Notes.md 8.9 KB

Vault Notes.md

Vualt is setup with dynamoDB as the backend. Vault has 3 nodes in a cluster and an AWS ALB as the frontend. The vault is unsealed with AWS KMS instead of the usual master key.

the vault binary is located at /usr/local/bin/vault

  1. change made to the service file Unknown lvalue 'StartLimitIntervalSec' in section 'Service'

Failed to parse capability in bounding/ambient set, ignoring: CAP_IPC_LOCK,CAP_NET_BIND_SERVICE

Oct 30 13:31:32 vault-1 systemd: [/etc/systemd/system/vault.service:16] Failed to parse capability in bounding/ambient set, ignoring: CAP_IPC_LOCK,CAP_NET_BIND_SERVICE

TEST VAULT

https://github.mdr.defpoint.com/mdr-engineering/msoc-infrastructure/tree/master/salt/fileroots/vault

  1. stop vault service from salt on all vault instances 1.1 salt vault* cmd.run 'systemctl stop vault'
  2. wipe dynamoDB (select items-> actions -> delete) until there are no more items (BESURE to BACKUP FIRST!)
  3. start vault 3.1 run salt state to ensure it is in the correct state with all policies on disk. 3.2 salt vault* state.sls vault
  4. on vault-1, init vault RUN on the server not salt (avoid the recovery keys from getting into logs) 4.1 VAULT_ADDR=https://vault.mdr-test.defpoint.com vault operator init -tls-skip-verify=true -recovery-shares=5 -recovery-threshold=2
  5. login 5.1 VAULT_ADDR=https://vault.mdr-test.defpoint.com vault login -tls-skip-verify=true -method=token 5.2 Do yourself a favor and setup some Bash Variables or run commands from salt export VAULT_ADDR=https://vault.mdr-test.defpoint.com export VAULT_ADDR=https://127.0.0.1 export VAULT_ADDR=https://vault.mdr.defpoint.com export VAULT_SKIP_VERIFY=1

  6. setup okta auth 6.1 VAULT_ADDR=https://vault.mdr-test.defpoint.com vault auth enable okta 6.2 VAULT_ADDR=https://vault.mdr-test.defpoint.com vault write -tls-skip-verify=true auth/okta/config base_url="okta.com" organization="mdr-multipass" token="api_token_here" 6.2 VAULT_ADDR=https://vault.mdr-test.defpoint.com vault write -tls-skip-verify=true auth/okta/config base_url="okta.com" organization="mdr-multipass" token="$( cat ~/.okta-token )" 6.3 VAULT_ADDR=https://vault.mdr-test.defpoint.com vault auth list 6.4 set the TTL for the okta auth method 6.4.1 VAULT_ADDR=https://vault.mdr-test.defpoint.com vault auth tune -default-lease-ttl=3h -max-lease-ttl=3h okta/

  7. Enable/add Policies 7.1 vault policy write -tls-skip-verify=true admins /etc/vault/admins.hcl 7.2 vault policy write -tls-skip-verify=true engineers /etc/vault/engineers.hcl 7.2 vault policy write -tls-skip-verify=true clu /etc/vault/clu.hcl 7.2 vault policy write -tls-skip-verify=true onboarding /etc/vault/onboarding.hcl 7.2 vault policy write -tls-skip-verify=true portal /etc/vault/portal.hcl 7.2 vault policy write -tls-skip-verify=true soc /etc/vault/soc.hcl 7.2 vault policy write salt-master /etc/vault/salt-master.hcl 7.2 vault policy write saltstack/minions /etc/vault/salt-minions.hcl

8 Add external groups 8.1 vault write identity/group name="admins" policies="admins" type="external" 8.2 vault write identity/group name="mdr-engineers" policies="engineers" type="external" 8.3 vault write identity/group name="vault-admins" policies="admins" type="external" 8.4 vault write identity/group name="soc-lead" policies="soc" type="external" 8.5 vault write identity/group name="soc-tier-3" policies="soc" type="external"

9 add alias through the GUI. (use the root token to login or a temp root token (better)) 9.1 Access -> Groups -> admins -> Aliases -> Create alias -> mdr-admins 9.2 Access -> Groups -> mdr-engineers -> Aliases -> Create alias -> mdr-engineers 9.3 Access -> Groups -> vault-admins -> Aliases -> Create alias -> vault-admin 9.4 Access -> Groups -> soc-lead -> Aliases -> Create alias -> Analyst-Shift-Lead 9.5 Access -> Groups -> soc-tier-3 -> Aliases -> Create alias -> Analyst-Tier-3

groups alias policy admins mdr-admins admins mdr-engineers mdr-engineers engineers vault-admins vault-admin admins soc-lead Analyst-Shift-Lead soc soc-tier-3 Analyst-Tier-3 soc

10 enable the file audit 10.1 VAULT_ADDR=https://vault.mdr-test.defpoint.com vault audit enable -tls-skip-verify=true file file_path=/var/log/vault.log

11 enable the aws & approle auth 11.1 VAULT_ADDR=https://vault.mdr-test.defpoint.com vault auth enable -tls-skip-verify=true aws 11.2 setup approle auth using the salt-master policy 11.2.1 vault auth enable approle 11.2.2 vault write auth/approle/role/salt-master token_max_ttl=3h token_policies=salt-master

12 configure the aws policies on the role (clu and portal) UPDATE THE AWS ACCOUNT!!! 12.1 VAULT_ADDR=https://vault.mdr-test.defpoint.com vault write auth/aws/role/portal auth_type=iam bound_iam_principal_arn=arn:aws:iam::527700175026:role/portal-instance-role policies=portal max_ttl=24h 12.2 VAULT_ADDR=https://vault.mdr-test.defpoint.com vault write auth/aws/role/clu auth_type=iam bound_iam_principal_arn=arn:aws:iam::527700175026:role/clu-instance-role policies=clu max_ttl=24h

13 Create the kv V2 secret engines VAULT_ADDR=https://vault.mdr-test.defpoint.com ~/Documents/MDR/Vault/vault secrets enable -path=engineering kv-v2 vault secrets enable -path=engineering kv-v2 vault secrets enable -path=ghe-deploy-keys kv-v2 vault secrets enable -path=jenkins kv-v2 vault secrets enable -path=onboarding kv-v2 vault secrets enable -path=onboarding-afs kv-v2 vault secrets enable -path=onboarding-gallery kv-v2 vault secrets enable -path=onboarding-saf kv-v2 vault secrets enable -path=portal kv-v2 vault secrets enable -path=soc kv-v2 vault secrets enable -version=1 -path=salt kv

vault write salt/pillar_data auth="abc123"

14 export the secrets (be sure to export your bash variable for VAULT_TOKEN DON'T Use ROOT TOKEN!) /Users/bradpoulton/.go/src/vault-backend-migrator/vault-backend-migrator -export engineering/data/ -metadata engineering/metadata/ -file engineering-secrets.json -ver 2

/Users/bradpoulton/.go/src/vault-backend-migrator/vault-backend-migrator -export ghe-deploy-keys/data/ -metadata ghe-deploy-keys/metadata/ -file ghe-deploy-keys-secrets.json -ver 2

/Users/bradpoulton/.go/src/vault-backend-migrator/vault-backend-migrator -export jenkins/data/ -metadata jenkins/metadata/ -file jenkins-secrets.json -ver 2

/Users/bradpoulton/.go/src/vault-backend-migrator/vault-backend-migrator -export onboarding/data/ -metadata onboarding/metadata/ -file onboarding-secrets.json -ver 2

/Users/bradpoulton/.go/src/vault-backend-migrator/vault-backend-migrator -export onboarding-afs/data/ -metadata onboarding-afs/metadata/ -file onboarding-afs-secrets.json -ver 2

/Users/bradpoulton/.go/src/vault-backend-migrator/vault-backend-migrator -export onboarding-gallery/data/ -metadata onboarding-gallery/metadata/ -file onboarding-gallery-secrets.json -ver 2

/Users/bradpoulton/.go/src/vault-backend-migrator/vault-backend-migrator -export onboarding-saf/data/ -metadata onboarding-saf/metadata/ -file onboarding-saf-secrets.json -ver 2

/Users/bradpoulton/.go/src/vault-backend-migrator/vault-backend-migrator -export portal/data/ -metadata portal/metadata/ -file portal-secrets.json -ver 2

15 import the json secret files back into vault /Users/bradpoulton/.go/src/vault-backend-migrator/vault-backend-migrator -import engineering/ -file engineering-secrets.json -ver 2

/Users/bradpoulton/.go/src/vault-backend-migrator/vault-backend-migrator -import ghe-deploy-keys/ -file ghe-deploy-keys-secrets.json -ver 2

/Users/bradpoulton/.go/src/vault-backend-migrator/vault-backend-migrator -import jenkins/ -file jenkins-secrets.json -ver 2

/Users/bradpoulton/.go/src/vault-backend-migrator/vault-backend-migrator -import onboarding/ -file onboarding-secrets.json -ver 2

/Users/bradpoulton/.go/src/vault-backend-migrator/vault-backend-migrator -import onboarding-afs/ -file onboarding-afs-secrets.json -ver 2

/Users/bradpoulton/.go/src/vault-backend-migrator/vault-backend-migrator -import onboarding-gallery/ -file onboarding-gallery-secrets.json -ver 2

/Users/bradpoulton/.go/src/vault-backend-migrator/vault-backend-migrator -import onboarding-saf/ -file onboarding-saf-secrets.json -ver 2

/Users/bradpoulton/.go/src/vault-backend-migrator/vault-backend-migrator -import portal/ -file portal-secrets.json -ver 2

AWS auth the vault instances have access to AWS IAM Read.

curl -v --header "X-Vault-Token:$VAULT_TOKEN" --request LIST

https://vault.mdr.defpoint.com:443/v1/auth/aws/roles --insecure
  1. map okta to policies ( not needed ) 8.1 VAULT_ADDR=https://vault.mdr-test.defpoint.com vault policy write -tls-skip-verify=true auth/okta/groups/mdr-admins policies=admins

Vault Logs

cat 0c86fda6-1139-7914-fef5-6b7532e9fb | grep -v -F '"operation":"list"' | grep -v -F '"operation":"read"' cat c3c0b50b-9429-355d-8c8f-038e093c3e | grep -v -F '"operation":"list"' | grep -v -F '"operation":"read"'

entity_34d6c410 -< nothing in logs
"entity_id":"c3c0b50b-9429-355d-8c8f-038e093c3e entity_ba27bb07 < - nothing in logs 0c86fda6-1139-7914-fef5-6b7532e9fb