# CA Notes Information learned / designed when creating our CA. # Quick Reference Dashboard: https://moose-splunk.pvt.xdr.accenturefederalcyber.com/en-US/app/splunk_app_aws/private_ca_status_dashboard # Hierarchy XDR uses a simple 2-level CA hierarchy: root CA and subordinate CA | CA Type | Description | Path Length | | ------- | ----------- | ----------- | | Root CA | In mdr-prod-root-ca account that alerts on any and all activity. | Unspecified | | Identity Subordinate CA | In mdr-common-services account that alerts on all kms activity. | 0 | | WWW Subordinate CA | In mdr-common-services account that alerts on all kms activity. | 0 | ## Templates ACM provides templates for basic constraint values. Of interest to us: * `RootCACertificate/V1` - Root CA * `SubordinateCACertificate_PathLen0/V1` - Our signing cert * `EndEntityCertificate/V1` - End entities ## Parsing the CRLs Get the CRL URIs: ``` # This will output the _root_ CRL openssl x509 -in ~/infrastructure-notes/files/xdr_subordinate_ca.govcloud_for_idp.crt -noout -text | grep crl # This will output the subordinate CRL yubico-piv-tool --slot 9a --action read-certificate | openssl x509 -noout -text | grep crl ``` ``` # Root CA's CRL: curl http://xdr-root-crl.s3.us-gov-east-1.amazonaws.com/crl/f58ae001-7ba3-4fb2-935a-bf12aad34ec6crl | openssl crl -inform DER -text -noout # Identity Subordinate CA's CRL: curl http://xdr-subordinate-crl.s3.us-gov-east-1.amazonaws.com/crl/FILLTHISINFROMACLIENTCERT.crl | openssl crl -inform DER -text -noout # WWW Subordinate CA's CRL: curl http://xdr-subordinate-crl.s3.us-gov-east-1.amazonaws.com/crl/FILLTHISINFROMACLIENTCERT.crl | openssl crl -inform DER -text -noout ``` ## Generate an audit report The moose SH automatically generates these reports via `/opt/splunk/etc/apps/SA-moose/`. ``` # Root CA aws --profile mdr-prod-root-ca-gov \ acm-pca create-certificate-authority-audit-report \ --certificate-authority-arn arn:aws-us-gov:acm-pca:us-gov-east-1:455637268483:certificate-authority/f58ae001-7ba3-4fb2-935a-bf12aad34ec6 \ --s3-bucket-name xdr-ca-audit-reports \ --audit-report-response-format CSV # Identity Secondary CA aws --profile mdr-common-services-gov \ acm-pca create-certificate-authority-audit-report \ --certificate-authority-arn arn:aws-us-gov:acm-pca:us-gov-east-1:701290387780:certificate-authority/cb0ea325-a347-4297-9cb8-2134410c3889 \ --s3-bucket-name xdr-ca-audit-reports \ --audit-report-response-format CSV # WWW Secondary CA aws --profile mdr-common-services-gov \ acm-pca create-certificate-authority-audit-report \ --certificate-authority-arn arn:aws-us-gov:acm-pca:us-gov-east-1:701290387780:certificate-authority/05b83e94-f625-43e4-a9f2-c3d8f04410f9 \ --s3-bucket-name xdr-ca-audit-reports \ --audit-report-response-format CSV ``` ### Audit Report Not Updating If you get an email: ``` The alert condition for 'Private CA Audit Log - Not Downloaded or Generated' was triggered. THIS MEANS WE ARENT AUDITING THE PRIVATE CA! Investigate SA-Moose on the Moose SH. See FedRAMP SC-17 ``` Splunk may have lost its permissions. Run: ``` cd ~/xdr-terraform-live/prod/aws-us-gov/mdr-prod-root-ca/011-root-CA terragrunt apply cd ~/xdr-terraform-live/prod/aws-us-gov/mdr-prod-root-ca/012-subordinate-cas terragrunt apply ``` The audit log is triggered and download by an app on moose-splunk-sh. To troubleshoot, try running it manually: ``` tshp moose-splunk-sh cd /opt/splunk/etc/apps/SA-moose/bin sudo -u splunk ./download_audit_reports.sh ``` ## Revoke a Certificate Grab the serial number from [Moose Private CA Dashboard](https://moose-splunk.pvt.xdr.accenturefederalcyber.com/en-US/app/splunk_app_aws/private_ca_status_dashboard) ``` aws --profile mdr-common-services-gov \ acm-pca revoke-certificate \ --certificate-authority-arn arn:aws-us-gov:acm-pca:us-gov-east-1:701290387780:certificate-authority/cb0ea325-a347-4297-9cb8-2134410c3889 \ --revocation-reason SUPERSEDED \ --certificate-serial ``` Reason can be one of: `AFFILIATION_CHANGED, CESSATION_OF_OPERATION, A_A_COMPROMISE, PRIVILEGE_WITHDRAWN, SUPERSEDED, UNSPECIFIED, KEY_COMPROMISE, CERTIFICATE_AUTHORITY_COMPROMISE` # 2022-08-09 Wildcard Cert Investigation 3 non-revoked Certs: | Certificate | Expiration | Where found | Alt Names | Notes | + ----------------------------------------------- + -------------------------------- + --------------------------------------- + --------- + ----- + | 87:99:92:ac:31:77:2c:f0:9a:c4:ae:d0:03:f8:fa:8b | Expires 2023-10-18T18:17:05+0000 | Fred's Laptop | DNS:localhost, DNS:xdr.accenturefederalcyber.com, DNS:xdrtest.accenturefederalcyber.com, DNS:*.xdr.accenturefederalcyber.com, DNS:*.xdrtest.accenturefederalcyber.com, DNS:*.pvt.xdr.accenturefederalcyber.com, DNS:*.pvt.xdrtest.accenturefederalcyber.com | Notes | | 69:fe:ae:64:d6:5c:b7:9c:72:28:b5:cc:d7:43:a2:0a | Expires 2022-08-20T19:34:44+0000 | Salt Pillar: xdr_govcloud_wildcard_2021 | DNS:*.xdr.accenturefederalcyber.com, DNS:*.xdrtest.accenturefederalcyber.com, DNS:*.pvt.xdr.accenturefederalcyber.com, DNS:*.pvt.xdrtest.accenturefederalcyber.com | Notes | | e7:5b:b2:ba:5e:49:2f:8c:46:f3:53:20:ee:22:24:80 | Expires 2022-08-17T23:03:15+0000 | | # Wildcard Cert Replacement Process The maximum expiration for SSL certificates is now 1 year. Every 6 months, we need to replace the wildcard certificate on our systems. To do so: 0. Future todo: Revisit these steps to make it so nobody ever has to see the key nor store it locally. 1. Edit `msoc-infrastructure/salt/pillar/mdr_wildcard_cert.sls` 2. Under `xdr_wildcard`, rename `certificate` to `certificate_replaced_`, and optionally remove any previous certificate backups. 3. Save a copy of the private key to `wildcard.key` by either (be sure to remove any indentation): a. Using gnupg to decrypt it, if it was encrypted with your key. b. Use `salt \* pillar.get certs:xdr_wildcard:private_key`. 4. Generate a new CSR: ``` openssl req -new -key wildcard.key -nodes -out wildcard.csr \ -subj '/C=US/ST=Virginia/L=Fairfax/O=Accenture Federal Services/OU=Extended Detection and Response/CN=*.xdr.accenturefederalcyber.com/' \ -extensions san -config <( \ echo '[req]'; \ echo 'distinguished_name=req'; \ echo '[san]'; \ echo 'subjectAltName=DNS:*.xdr.accenturefederalcyber.com,DNS:*.xdrtest.accenturefederalcyber.com,DNS:*.pvt.xdr.accenturefederalcyber.com,DNS:*.pvt.xdrtest.accenturefederalcyber.com' \ echo '[ext]'; \ echo 'basicConstraints=CA:TRUE,pathlen:0') ``` 5. Validate the request with: `openssl req -in wildcard.csr -noout -text -verify` 6. Generate the policy for SANs ``` cat < tmp.policy { "Extensions": { "SubjectAlternativeNames": [ { "DnsName": "xdr.accenturefederalcyber.com" }, { "DnsName": "xdrtest.accenturefederalcyber.com" }, { "DnsName": "*.xdr.accenturefederalcyber.com" }, { "DnsName": "*.xdrtest.accenturefederalcyber.com" }, { "DnsName": "*.pvt.xdr.accenturefederalcyber.com" }, { "DnsName": "*.pvt.xdrtest.accenturefederalcyber.com" } ] } } EOF ``` 7. Sign the certificate and save the output ``` aws --profile mdr-common-services-gov --region us-gov-east-1 acm-pca issue-certificate \ --api-passthrough file://./tmp.policy \ --certificate-authority-arn arn:aws-us-gov:acm-pca:us-gov-east-1:701290387780:certificate-authority/05b83e94-f625-43e4-a9f2-c3d8f04410f9 \ --csr fileb://./wildcard.csr \ --signing-algorithm SHA512WITHECDSA \ --template-arn arn:aws-us-gov:acm-pca:::template/EndEntityCertificate_APIPassthrough/V1 \ --validity Value=365,Type=DAYS \ --output=json \ | jq -r .CertificateArn > wildcard.arn ``` arn:aws-us-gov:acm-pca:us-gov-east-1:701290387780:certificate-authority/05b83e94-f625-43e4-a9f2-c3d8f04410f9/certificate/8ae39f3463e5e3cdc1156a5f22d25ba3 8. Retrieve the certificate: ``` aws --profile mdr-common-services-gov --region us-gov-east-1 acm-pca get-certificate \ --certificate-authority-arn arn:aws-us-gov:acm-pca:us-gov-east-1:701290387780:certificate-authority/05b83e94-f625-43e4-a9f2-c3d8f04410f9 \ --certificate-arn $(cat wildcard.arn) \ --output text | sed 's/\t/\n/' > wildcard.crt ``` 9. Validate the certificate ``` openssl x509 -in wildcard.crt -noout -text ``` 10. Update the certificate in `msoc-infrastructure/salt/pillar/mdr_wildcard_cert.sls` and do a PR. Validate that the chain has not changed. 11. On the salt master, refresh pillars: ``` salt \* saltutil.pillar_refresh certs ``` 12. In test, apply the following salt states: ``` salt \*-alsi-\* state.sls alsi --output-diff test=true salt mailrelay\* state.sls mailrelay --output-diff test=true salt rhsso\* state.sls rhsso --output-diff test=true salt fm-shared-search\* state.sls splunk.certificate_update --output-diff test=true salt threatq\* state.sls threatquotient --output-diff test=true # The following will not automatically restart services, so are safe to run. # You can either restart manually, or wait until patch day. salt teleport\* state.sls teleport.config --output-diff test=true salt \*splunk-\* state.sls splunk.certificate_update --output-diff exclude="[{'id': 'restart_splunk'}]" test=true salt phantom\* state.sls phantom.certificate_update --output-diff exclude="[{'id': 'restart_nginx'}]" test=true ``` 13. PR to main, and repeat steps 11 and 12 in prod. 14. Create a ticket to repeat in 6 months. 15. Clean up: `rm -v wildcard*`. Ensure that the key is deleted.