# CA Notes

Information learned / designed when creating our CA.

# Quick Reference

Dashboard: https://moose-splunk.pvt.xdr.accenturefederalcyber.com/en-US/app/splunk_app_aws/private_ca_status_dashboard

# Hierarchy

XDR uses a simple 2-level CA hierarchy: root CA and subordinate CA

| CA Type | Description | Path Length |
| ------- | ----------- | ----------- |
| Root CA | In mdr-prod-root-ca account that alerts on any and all activity. | Unspecified |
| Identity Subordinate CA | In mdr-common-services account that alerts on all kms activity. | 0 |
| WWW Subordinate CA | In mdr-common-services account that alerts on all kms activity. | 0 |

## Templates

ACM provides templates for basic constraint values. Of interest to us:
* `RootCACertificate/V1` - Root CA
* `SubordinateCACertificate_PathLen0/V1` - Our signing cert
* `EndEntityCertificate/V1` - End entities

## Parsing the CRLs

Get the CRL URIs:
```
# This will output the _root_ CRL
openssl x509 -in ~/infrastructure-notes/files/xdr_subordinate_ca.govcloud_for_idp.crt -noout -text  | grep crl
# This will output the subordinate CRL
yubico-piv-tool --slot 9a --action read-certificate | openssl x509 -noout -text  | grep crl
```

```
# Root CA's CRL:
curl http://xdr-root-crl.s3.us-gov-east-1.amazonaws.com/crl/f58ae001-7ba3-4fb2-935a-bf12aad34ec6crl | openssl crl -inform DER -text -noout
# Identity Subordinate CA's CRL:
curl http://xdr-subordinate-crl.s3.us-gov-east-1.amazonaws.com/crl/FILLTHISINFROMACLIENTCERT.crl | openssl crl -inform DER -text -noout
# WWW Subordinate CA's CRL:
curl http://xdr-subordinate-crl.s3.us-gov-east-1.amazonaws.com/crl/FILLTHISINFROMACLIENTCERT.crl | openssl crl -inform DER -text -noout
```

## Generate an audit report

The moose SH automatically generates these reports via `/opt/splunk/etc/apps/SA-moose/`.

```
# Root CA
aws --profile mdr-prod-root-ca-gov \
  acm-pca create-certificate-authority-audit-report \
  --certificate-authority-arn arn:aws-us-gov:acm-pca:us-gov-east-1:455637268483:certificate-authority/f58ae001-7ba3-4fb2-935a-bf12aad34ec6 \
  --s3-bucket-name xdr-ca-audit-reports \
  --audit-report-response-format CSV 
# Identity Secondary CA
aws --profile mdr-common-services-gov \
  acm-pca create-certificate-authority-audit-report \
  --certificate-authority-arn arn:aws-us-gov:acm-pca:us-gov-east-1:701290387780:certificate-authority/cb0ea325-a347-4297-9cb8-2134410c3889 \
  --s3-bucket-name xdr-ca-audit-reports \
  --audit-report-response-format CSV 
# WWW Secondary CA
aws --profile mdr-common-services-gov \
  acm-pca create-certificate-authority-audit-report \
  --certificate-authority-arn arn:aws-us-gov:acm-pca:us-gov-east-1:701290387780:certificate-authority/05b83e94-f625-43e4-a9f2-c3d8f04410f9 \
  --s3-bucket-name xdr-ca-audit-reports \
  --audit-report-response-format CSV 
```

### Audit Report Not Updating

If you get an email:
```
The alert condition for 'Private CA Audit Log - Not Downloaded or Generated' was triggered.

THIS MEANS WE ARENT AUDITING THE PRIVATE CA! Investigate SA-Moose on the Moose SH.

See FedRAMP SC-17
```

Splunk may have lost its permissions. Run:
```
cd ~/xdr-terraform-live/prod/aws-us-gov/mdr-prod-root-ca/011-root-CA
terragrunt apply
cd ~/xdr-terraform-live/prod/aws-us-gov/mdr-prod-root-ca/012-subordinate-cas
terragrunt apply
```

The audit log is triggered and download by an app on moose-splunk-sh. To troubleshoot, try running it manually:
```
tshp moose-splunk-sh
cd /opt/splunk/etc/apps/SA-moose/bin
sudo -u splunk ./download_audit_reports.sh
```


## Revoke a Certificate

Grab the serial number from [Moose Private CA Dashboard](https://moose-splunk.pvt.xdr.accenturefederalcyber.com/en-US/app/splunk_app_aws/private_ca_status_dashboard)

```
aws --profile mdr-common-services-gov \
  acm-pca revoke-certificate \
  --certificate-authority-arn arn:aws-us-gov:acm-pca:us-gov-east-1:701290387780:certificate-authority/cb0ea325-a347-4297-9cb8-2134410c3889 \
  --revocation-reason SUPERSEDED \
  --certificate-serial <FILLMEIN>
```
Reason can be one of:
`AFFILIATION_CHANGED, CESSATION_OF_OPERATION, A_A_COMPROMISE, PRIVILEGE_WITHDRAWN, SUPERSEDED, UNSPECIFIED, KEY_COMPROMISE, CERTIFICATE_AUTHORITY_COMPROMISE`

# 2022-08-09 Wildcard Cert Investigation

3 non-revoked Certs:
| Certificate                                     | Expiration                       | Where found                             | Alt Names | Notes |
+ ----------------------------------------------- + -------------------------------- + --------------------------------------- + --------- + ----- +
| 87:99:92:ac:31:77:2c:f0:9a:c4:ae:d0:03:f8:fa:8b | Expires 2023-10-18T18:17:05+0000 | Fred's Laptop                           | DNS:localhost, DNS:xdr.accenturefederalcyber.com, DNS:xdrtest.accenturefederalcyber.com, DNS:*.xdr.accenturefederalcyber.com, DNS:*.xdrtest.accenturefederalcyber.com, DNS:*.pvt.xdr.accenturefederalcyber.com, DNS:*.pvt.xdrtest.accenturefederalcyber.com | Notes |
| 69:fe:ae:64:d6:5c:b7:9c:72:28:b5:cc:d7:43:a2:0a | Expires 2022-08-20T19:34:44+0000 | Salt Pillar: xdr_govcloud_wildcard_2021 | DNS:*.xdr.accenturefederalcyber.com, DNS:*.xdrtest.accenturefederalcyber.com, DNS:*.pvt.xdr.accenturefederalcyber.com, DNS:*.pvt.xdrtest.accenturefederalcyber.com | Notes |
| e7:5b:b2:ba:5e:49:2f:8c:46:f3:53:20:ee:22:24:80 | Expires 2022-08-17T23:03:15+0000 |                                         |

# Wildcard Cert Replacement Process

The maximum expiration for SSL certificates is now 1 year.

Every 6 months, we need to replace the wildcard certificate on our systems. To do so:
0. Future todo: Revisit these steps to make it so nobody ever has to see the key nor store it locally.
1. Edit `msoc-infrastructure/salt/pillar/mdr_wildcard_cert.sls`
2. Under `xdr_wildcard`, rename `certificate` to `certificate_replaced_<date>`, and optionally remove any previous certificate backups.
3. Save a copy of the private key to `wildcard.key` by either (be sure to remove any indentation):
  a. Using gnupg to decrypt it, if it was encrypted with your key.
  b. Use `salt \* pillar.get certs:xdr_wildcard:private_key`. 
4. Generate a new CSR:
```
openssl req -new -key wildcard.key -nodes -out wildcard.csr \
  -subj '/C=US/ST=Virginia/L=Fairfax/O=Accenture Federal Services/OU=Extended Detection and Response/CN=*.xdr.accenturefederalcyber.com/' \
  -extensions san -config <( \
  echo '[req]'; \
  echo 'distinguished_name=req'; \
  echo '[san]'; \
  echo 'subjectAltName=DNS:*.xdr.accenturefederalcyber.com,DNS:*.xdrtest.accenturefederalcyber.com,DNS:*.pvt.xdr.accenturefederalcyber.com,DNS:*.pvt.xdrtest.accenturefederalcyber.com' \
  echo '[ext]'; \
  echo 'basicConstraints=CA:TRUE,pathlen:0')
```
5. Validate the request with: `openssl req -in wildcard.csr -noout -text -verify`
6. Generate the policy for SANs
```
cat <<EOF > tmp.policy
{
  "Extensions": {
    "SubjectAlternativeNames": [
      { "DnsName": "xdr.accenturefederalcyber.com" },
      { "DnsName": "xdrtest.accenturefederalcyber.com" },
      { "DnsName": "*.xdr.accenturefederalcyber.com" },
      { "DnsName": "*.xdrtest.accenturefederalcyber.com" },
      { "DnsName": "*.pvt.xdr.accenturefederalcyber.com" },
      { "DnsName": "*.pvt.xdrtest.accenturefederalcyber.com" }
    ]
  }
}
EOF
```
7. Sign the certificate and save the output
```
aws --profile mdr-common-services-gov --region us-gov-east-1 acm-pca issue-certificate \
  --api-passthrough file://./tmp.policy \
  --certificate-authority-arn arn:aws-us-gov:acm-pca:us-gov-east-1:701290387780:certificate-authority/05b83e94-f625-43e4-a9f2-c3d8f04410f9 \
  --csr fileb://./wildcard.csr \
  --signing-algorithm SHA512WITHECDSA \
  --template-arn arn:aws-us-gov:acm-pca:::template/EndEntityCertificate_APIPassthrough/V1 \
  --validity Value=365,Type=DAYS \
  --output=json \
  | jq -r .CertificateArn > wildcard.arn
```
arn:aws-us-gov:acm-pca:us-gov-east-1:701290387780:certificate-authority/05b83e94-f625-43e4-a9f2-c3d8f04410f9/certificate/8ae39f3463e5e3cdc1156a5f22d25ba3
8. Retrieve the certificate:
```
aws --profile mdr-common-services-gov --region us-gov-east-1 acm-pca get-certificate \
    --certificate-authority-arn arn:aws-us-gov:acm-pca:us-gov-east-1:701290387780:certificate-authority/05b83e94-f625-43e4-a9f2-c3d8f04410f9 \
    --certificate-arn $(cat wildcard.arn) \
    --output text | sed 's/\t/\n/' > wildcard.crt
```
9.  Validate the certificate
```
openssl x509 -in wildcard.crt -noout -text
```
10. Update the certificate in `msoc-infrastructure/salt/pillar/mdr_wildcard_cert.sls` and do a PR. Validate that the chain has not changed.
11. On the salt master, refresh pillars:
```
salt \* saltutil.pillar_refresh certs
```
12. In test, apply the following salt states:
```
salt \*-alsi-\* state.sls alsi --output-diff test=true
salt mailrelay\* state.sls mailrelay --output-diff test=true
salt rhsso\* state.sls rhsso --output-diff test=true
salt fm-shared-search\* state.sls splunk.certificate_update --output-diff test=true
salt threatq\* state.sls threatquotient --output-diff test=true
# The following will not automatically restart services, so are safe to run.
# You can either restart manually, or wait until patch day.
salt teleport\* state.sls teleport.config --output-diff test=true
salt \*splunk-\* state.sls splunk.certificate_update --output-diff exclude="[{'id': 'restart_splunk'}]" test=true
salt phantom\* state.sls phantom.certificate_update --output-diff exclude="[{'id': 'restart_nginx'}]" test=true
```
13. PR to main, and repeat steps 11 and 12 in prod.
14. Create a ticket to repeat in 6 months.
15. Clean up: `rm -v wildcard*`. Ensure that the key is deleted.
