CA Notes

Information learned / designed when creating our CA.

Quick Reference

Dashboard: https://moose-splunk.pvt.xdr.accenturefederalcyber.com/en-US/app/splunk_app_aws/private_ca_status_dashboard

Hierarchy

XDR uses a simple 2-level CA hierarchy: root CA and subordinate CA

CA Type	Description	Path Length
Root CA	In mdr-prod-root-ca account that alerts on any and all activity.	Unspecified
Identity Subordinate CA	In mdr-common-services account that alerts on all kms activity.	0
WWW Subordinate CA	In mdr-common-services account that alerts on all kms activity.	0

Templates

ACM provides templates for basic constraint values. Of interest to us:

RootCACertificate/V1 - Root CA
SubordinateCACertificate_PathLen0/V1 - Our signing cert
EndEntityCertificate/V1 - End entities

Parsing the CRLs

Get the CRL URIs:

# This will output the _root_ CRL
openssl x509 -in ~/infrastructure-notes/files/xdr_subordinate_ca.govcloud_for_idp.crt -noout -text  | grep crl
# This will output the subordinate CRL
yubico-piv-tool --slot 9a --action read-certificate | openssl x509 -noout -text  | grep crl

# Root CA's CRL:
curl http://xdr-root-crl.s3.us-gov-east-1.amazonaws.com/crl/f58ae001-7ba3-4fb2-935a-bf12aad34ec6crl | openssl crl -inform DER -text -noout
# Identity Subordinate CA's CRL:
curl http://xdr-subordinate-crl.s3.us-gov-east-1.amazonaws.com/crl/FILLTHISINFROMACLIENTCERT.crl | openssl crl -inform DER -text -noout
# WWW Subordinate CA's CRL:
curl http://xdr-subordinate-crl.s3.us-gov-east-1.amazonaws.com/crl/FILLTHISINFROMACLIENTCERT.crl | openssl crl -inform DER -text -noout

Generate an audit report

The moose SH automatically generates these reports via /opt/splunk/etc/apps/SA-moose/.

# Root CA
aws --profile mdr-prod-root-ca-gov \
  acm-pca create-certificate-authority-audit-report \
  --certificate-authority-arn arn:aws-us-gov:acm-pca:us-gov-east-1:455637268483:certificate-authority/f58ae001-7ba3-4fb2-935a-bf12aad34ec6 \
  --s3-bucket-name xdr-ca-audit-reports \
  --audit-report-response-format CSV 
# Identity Secondary CA
aws --profile mdr-common-services-gov \
  acm-pca create-certificate-authority-audit-report \
  --certificate-authority-arn arn:aws-us-gov:acm-pca:us-gov-east-1:701290387780:certificate-authority/cb0ea325-a347-4297-9cb8-2134410c3889 \
  --s3-bucket-name xdr-ca-audit-reports \
  --audit-report-response-format CSV 
# WWW Secondary CA
aws --profile mdr-common-services-gov \
  acm-pca create-certificate-authority-audit-report \
  --certificate-authority-arn arn:aws-us-gov:acm-pca:us-gov-east-1:701290387780:certificate-authority/05b83e94-f625-43e4-a9f2-c3d8f04410f9 \
  --s3-bucket-name xdr-ca-audit-reports \
  --audit-report-response-format CSV

Audit Report Not Updating

If you get an email:

The alert condition for 'Private CA Audit Log - Not Downloaded or Generated' was triggered.

THIS MEANS WE ARENT AUDITING THE PRIVATE CA! Investigate SA-Moose on the Moose SH.

See FedRAMP SC-17

Splunk may have lost its permissions. Run:

cd ~/xdr-terraform-live/prod/aws-us-gov/mdr-prod-root-ca/011-root-CA
terragrunt apply
cd ~/xdr-terraform-live/prod/aws-us-gov/mdr-prod-root-ca/012-subordinate-cas
terragrunt apply

The audit log is triggered and download by an app on moose-splunk-sh. To troubleshoot, try running it manually:

tshp moose-splunk-sh
cd /opt/splunk/etc/apps/SA-moose/bin
sudo -u splunk ./download_audit_reports.sh

Revoke a Certificate

Grab the serial number from Moose Private CA Dashboard

aws --profile mdr-common-services-gov \
  acm-pca revoke-certificate \
  --certificate-authority-arn arn:aws-us-gov:acm-pca:us-gov-east-1:701290387780:certificate-authority/cb0ea325-a347-4297-9cb8-2134410c3889 \
  --revocation-reason SUPERSEDED \
  --certificate-serial <FILLMEIN>

Reason can be one of: AFFILIATION_CHANGED, CESSATION_OF_OPERATION, A_A_COMPROMISE, PRIVILEGE_WITHDRAWN, SUPERSEDED, UNSPECIFIED, KEY_COMPROMISE, CERTIFICATE_AUTHORITY_COMPROMISE

2022-08-09 Wildcard Cert Investigation

----------------------------------------------- + -------------------------------- + --------------------------------------- + --------- + ----- + | 87:99:92:ac:31:77:2c:f0:9a:c4:ae:d0:03:f8:fa:8b | Expires 2023-10-18T18:17:05+0000 | Fred's Laptop | DNS:localhost, DNS:xdr.accenturefederalcyber.com, DNS:xdrtest.accenturefederalcyber.com, DNS:.xdr.accenturefederalcyber.com, DNS:.xdrtest.accenturefederalcyber.com, DNS:.pvt.xdr.accenturefederalcyber.com, DNS:.pvt.xdrtest.accenturefederalcyber.com | Notes | | 69:fe:ae:64:d6:5c:b7:9c:72:28:b5:cc:d7:43:a2:0a | Expires 2022-08-20T19:34:44+0000 | Salt Pillar: xdr_govcloud_wildcard_2021 | DNS:.xdr.accenturefederalcyber.com, DNS:.xdrtest.accenturefederalcyber.com, DNS:.pvt.xdr.accenturefederalcyber.com, DNS:.pvt.xdrtest.accenturefederalcyber.com | Notes | | e7:5b:b2:ba:5e:49:2f:8c:46:f3:53:20:ee:22:24:80 | Expires 2022-08-17T23:03:15+0000 | |

Wildcard Cert Replacement Process

The maximum expiration for SSL certificates is now 1 year.

Every 6 months, we need to replace the wildcard certificate on our systems. To do so:

Future todo: Revisit these steps to make it so nobody ever has to see the key nor store it locally.
Edit msoc-infrastructure/salt/pillar/mdr_wildcard_cert.sls
Under xdr_wildcard, rename certificate to certificate_replaced_<date>, and optionally remove any previous certificate backups.
Save a copy of the private key to wildcard.key by either (be sure to remove any indentation): a. Using gnupg to decrypt it, if it was encrypted with your key. b. Use salt \* pillar.get certs:xdr_wildcard:private_key.

Generate a new CSR:

openssl req -new -key wildcard.key -nodes -out wildcard.csr \
-subj '/C=US/ST=Virginia/L=Fairfax/O=Accenture Federal Services/OU=Extended Detection and Response/CN=*.xdr.accenturefederalcyber.com/' \
-extensions san -config <( \
echo '[req]'; \
echo 'distinguished_name=req'; \
echo '[san]'; \
echo 'subjectAltName=DNS:*.xdr.accenturefederalcyber.com,DNS:*.xdrtest.accenturefederalcyber.com,DNS:*.pvt.xdr.accenturefederalcyber.com,DNS:*.pvt.xdrtest.accenturefederalcyber.com' \
echo '[ext]'; \
echo 'basicConstraints=CA:TRUE,pathlen:0')

Validate the request with: openssl req -in wildcard.csr -noout -text -verify

Generate the policy for SANs

cat <<EOF > tmp.policy
{
"Extensions": {
"SubjectAlternativeNames": [
  { "DnsName": "xdr.accenturefederalcyber.com" },
  { "DnsName": "xdrtest.accenturefederalcyber.com" },
  { "DnsName": "*.xdr.accenturefederalcyber.com" },
  { "DnsName": "*.xdrtest.accenturefederalcyber.com" },
  { "DnsName": "*.pvt.xdr.accenturefederalcyber.com" },
  { "DnsName": "*.pvt.xdrtest.accenturefederalcyber.com" }
]
}
}
EOF

Sign the certificate and save the output

aws --profile mdr-common-services-gov --region us-gov-east-1 acm-pca issue-certificate \
--api-passthrough file://./tmp.policy \
--certificate-authority-arn arn:aws-us-gov:acm-pca:us-gov-east-1:701290387780:certificate-authority/05b83e94-f625-43e4-a9f2-c3d8f04410f9 \
--csr fileb://./wildcard.csr \
--signing-algorithm SHA512WITHECDSA \
--template-arn arn:aws-us-gov:acm-pca:::template/EndEntityCertificate_APIPassthrough/V1 \
--validity Value=365,Type=DAYS \
--output=json \
| jq -r .CertificateArn > wildcard.arn

arn:aws-us-gov:acm-pca:us-gov-east-1:701290387780:certificate-authority/05b83e94-f625-43e4-a9f2-c3d8f04410/certificate/8ae39f3463

Retrieve the certificate:

aws --profile mdr-common-services-gov --region us-gov-east-1 acm-pca get-certificate \
--certificate-authority-arn arn:aws-us-gov:acm-pca:us-gov-east-1:701290387780:certificate-authority/05b83e94-f625-43e4-a9f2-c3d8f04410f9 \
--certificate-arn $(cat wildcard.arn) \
--output text | sed 's/\t/\n/' > wildcard.crt

Validate the certificate

openssl x509 -in wildcard.crt -noout -text

Update the certificate in msoc-infrastructure/salt/pillar/mdr_wildcard_cert.sls and do a PR. Validate that the chain has not changed.
On the salt master, refresh pillars:
```
salt \* saltutil.pillar_refresh certs
```

In test, apply the following salt states:

salt \*-alsi-\* state.sls alsi --output-diff test=true
salt mailrelay\* state.sls mailrelay --output-diff test=true
salt rhsso\* state.sls rhsso --output-diff test=true
salt fm-shared-search\* state.sls splunk.certificate_update --output-diff test=true
salt threatq\* state.sls threatquotient --output-diff test=true
# The following will not automatically restart services, so are safe to run.
# You can either restart manually, or wait until patch day.
salt teleport\* state.sls teleport.config --output-diff test=true
salt \*splunk-\* state.sls splunk.certificate_update --output-diff exclude="[{'id': 'restart_splunk'}]" test=true
salt phantom\* state.sls phantom.certificate_update --output-diff exclude="[{'id': 'restart_nginx'}]" test=true

PR to main, and repeat steps 11 and 12 in prod.
Create a ticket to repeat in 6 months.
Clean up: rm -v wildcard*. Ensure that the key is deleted.

CA Notes.md 9.4 KB Histórico Raw