# AWS Notes.md For the latest AWS account information see [Cloud Accounts Wiki](https://github.xdr.accenturefederalcyber.com/mdr-engineering/msoc-infrastructure/wiki/cloud-accounts) ### LEGACY AWS ACCOUNTS Root Account Alias (in AWS): defpoint-mdr-root Root Account Alias (for terraform/in `~/.aws/{config,credentials}`): mdr-root Root AWS Account ID: 350838957895 GovCloud Root Account: 701290387780 Test Account ID: 527700175026 Prod Account ID: 477548533976 Org Account ID: 228011623757 `assumeRole` to the test and prod accounts --- terraform has been setup to handle the cis checks for AWS. they are found in terraform/00-cis-hardening. Get a encoded error message from AWS? ``` AWS_PROFILE=mdr-test aws sts decode-authorization-message --encoded-message Q7h4sTOW_n_znBB7ojNotL ``` For prettier output: ``` aws --profile=mdr-test-c2-gov sts decode-authorization-message --encoded-message Q7h4sTOW_n_znBB7ojNotL... | \ jq '.["DecodedMessage"]' | \ sed 's/\\"/"/g' | \ sed 's/^"//' | \ sed 's/"$//' | \ jq ``` ------------------------------------------- Cloudtrail metric Alarms so .. cloudtrail writes a trail that trail is written into a cloudwatch logs log group in the log group, there are a number of metric filters the metric filters create metrics, upon which a metric alarm is set when events matching the metric filter arrive, the metric goes up, triggerting the alarm new messages the alarm has an SNS topic it writes to that emails me that the "metric was exceeded" ---------------------------------------------- AWS Systems Manager agent systemctl start amazon-ssm-agent ---------------------------------------------- ## AWS User Password Distribution ( legecy ) ``` https://gpgtools.org/ download and install use gpg keychain to generate pub/private keys something something terraform echo "wcFMA2sXDKYLpzaUbf6clQ043oDkHIrcWK509UIy5GUpEqBV/WLmuCMHkXUgnxy12HY8qBErF58vB7/VXs5pCKp4SDYWEtK73fKmYZ5wJDW6j6OHkpYI4USZXjVYb+Utt56Qprk4KiT6VlFNNPo00r2YDABDdtxPJS3N9REzHqp+7oR2SQkiyEhcF3ZwILk2fH4mc1VQUiFu68RCqbt+QfmDt3OHIRZVPvrS4AHkCbj2fdgkbAaRMJ/21TBn8OE8WuDR4NHh5w/gWeK5m6754DzkjVLxDpsvPG2UR9ErwANEo+BI4upil2vgT+S63PIVsAmTew/7QpPavttP4rUBM47h5cMA"|base64 -D | gpg -d ``` ``` #Export in one line and base64 gpg --export | base64 ``` ## saml2aws 2020-05-07 on OSX, install `saml2aws`. Follow the instructions in AFS Macbook Notes.md XDR uses a custom version modified by Duane. We can revert to official if this gets merged. Additional instructions on [Github](https://github.com/Versent/saml2aws). Create `~/.saml2aws` with the following: ``` [commercial] app_id = url = https://mdr-multipass.okta.com/home/amazon_aws/0oa3cn8m68DzMXNbc297/272 username = your.okta.username provider = Okta mfa = PUSH skip_verify = false timeout = 0 aws_urn = https://signin.aws.amazon.com/saml aws_session_duration = 28800 aws_profile = commercial resource_id = subdomain = role_arn = arn:aws:iam::471284459109:role/user/mdr_engineer_readonly http_attempts_count = http_retry_delay = [govcloud] app_id = url = https://mdr-multipass.okta.com/home/amazon_aws/0oa3e1gtozzfDFAel297/272 username = your.okta.username provider = Okta mfa = PUSH skip_verify = false timeout = 0 aws_urn = https://signin.amazonaws-us-gov.com/saml aws_session_duration = 28800 aws_profile = govcloud resource_id = subdomain = role_arn = arn:aws-us-gov:iam::701290387780:role/user/mdr_engineer_readonly http_attempts_count = http_retry_delay = region = us-gov-east-1 [cyber-range] app_id = url = https://mdr-multipass.okta.com/home/amazon_aws/0oa3cn8m68DzMXNbc297/272 username = your.okta.username provider = Okta mfa = PUSH skip_verify = false timeout = 0 aws_urn = https://signin.aws.amazon.com/saml aws_session_duration = 28800 aws_profile = cyber-range resource_id = subdomain = role_arn = arn:aws:iam::952430311316:role/user/mdr_engineer_readonly http_attempts_count = http_retry_delay = ``` Then you can sign in with: ``` saml2aws login --idp-account=govcloud ``` And use the profile via: ``` aws --profile govcloud ec2 describe-instances ``` Authentication not working? `saml2aws` uses the MAC keychain to store your password. When you update your password you need to update the password in keychain as well. Or resetup saml2aws. ## AssumeRole Extension and ~/.aws/config 08/03/2020 Install the plugin for chrome/firefox: Firefox [aws-extend-switch-roles](https://addons.mozilla.org/en-US/firefox/addon/aws-extend-switch-roles3/#&gid=1&pid=3) Chrome [aws-extend-switch-roles](https://chrome.google.com/webstore/detail/aws-extend-switch-roles/jpmkfafbacpgapdghgdpembnojdlgkdl/related?hl=en) Copy this into `~/.aws/credentials` ``` [govcloud] region = us-gov-east-1 [commercial] ``` Then, create a link from `files/config` to `~/.aws/config`: (your path may vary) ``` ln -s ~/infrastructure-notes/files/config ~/.aws/config ``` ## CloudWatch bit of code to search cloudwatch ``` fields @timestamp, @message | filter errorCode="AccessDenied" | fields coalesce(userIdentity.invokedBy,userIdentity.principalId) as whoo, coalesce(requestParameters.bucketName,errorMessage) as target | stats count() as count by bin(1d) as time, whoo, eventName, target | sort count desc ``` ### Using jq with EC2 API relevant links * [Reshaping JSON with jq](https://programminghistorian.org/en/lessons/json-and-jq) * [AWS, JQ and bash command cheat sheet](https://gist.github.com/lukeplausin/b64c10f8b524bb310e0083756c42caf6) #### dump a list of ec2 instances and some of their relevant fields into a CSV ``` AWS_PROFILE=dps_prod aws ec2 describe-instances | jq -r '.Reservations[].Instances[] | [ (.Tags[]?|select(.Key=="Name")|.Value?), .InstanceId, .LaunchTime, .StateTransitionReason, .State.Name ] | @csv' ``` ### AWS Instance Scheduler in TEST Test Ec2 instances get shutdown at night. This is accomplished via a cloudformation template that is configured in Terragrunt. To prevent an EC2 instance from getting shutdown, just remove the Schedule tag. ## Fixing bad volume tags The `volume_tags` field in `aws_instance` doesn't handle updates (due to a bug where it was updating other attached volumes). To add volume tags: 1. Generate a list of bad volumes: ``` aws --profile legacy-mdr-test ec2 describe-volumes --filters "Name=attachment.status,Values=attached" --output yaml | grep VolumeId | awk '{print $2}' | sort -u > badvolumes.legacy-mdr-test.txt ``` 2. Add the tag to all of them: ``` for i in `cat badvolumes.legacy-mdr-test.txt` do echo $i aws --profile legacy-mdr-test ec2 create-tags --resources $i --tags "Key=Snapshot,Value=Daily" done ``` ### AWS Automatic Snapshots Snapshot Daily In both test and prod the EC2 EBS Lifecycle Manager handles the automatic backups for EBS Volumes. The Lifecycle Manager has a policy that looks for the tag, Snapshot:Daily. If the volume has that tag, then a snapshot is taken. ### Cleaning Old AMIs The open source `amicleaner` is very useful for cleaning our old AMIs, but needs a patched version. To install: ``` # Get a fork of amicleaner with the asg fix: git clone git@github.com:ronaldosaheki/aws-amicleaner.git cd aws-amicleaner git checkout origin/fix-asg pip3 install --user -e . ln -s ~/Library/Python/3.9/bin/amicleaner /usr/local/bin/amicleaner ``` You now have amicleaner in your path, and can run `~/xdr-terraform-live/bin/clean_old_amis.sh`. ## Rescuing Systems using SSM For example, the disk space on sensu in test was full. ``` aws --profile mdr-test-c2-gov \ ssm send-command --document-name "AWS-RunShellScript" \ --instance-ids "i-0d5072669fb00c2fb" --comment "SystemRescue" \ --parameters commands='rm -rf /var/log/sudo-io/*' # Grab Command ID from the output aws --profile mdr-test-c2-gov \ ssm list-command-invocations \ --command-id ae86ec00-e356-4ad6-b162-6b5ec5260ed9 ``` # Instance Reservations Instance reservatrions are tracked in teh wiki at https://github.xdr.accenturefederalcyber.com/mdr-engineering/msoc-infrastructure/wiki/Reserved-Instances and also recorded in teh tags of the instances via tags in `xdr-terraform-live/.../0xx-instance-xxx/terragrunt.hcl`. To reserve an instance, you have to be confident that (a) you're going to need the instance for the next 12 months, and (b) that the instance type won't change. If you don't meet (b), you can still consider a convertible instance reservation. 1) Verify via the Monitoring Dashboard (https://moose-splunk.pvt.xdr.accenturefederalcyber.com/en-US/app/search/freds_monitoring_dashboard) that there are no concerns over utilization, especially with respect to CPU and Memory. 2) Verify the same via the AWS Console. 3) Carefully record the instance types that are going to be reserved. 4) Verify that all the instance have the Platform listed as "Red Hat (Inferred)" / "Red Hat Enterprise Linux" 5) Go to reserved instances: a) Change platform to "Red Hat Enterprise Linux" (Watch out for GitHub/VMRay instance!) - Tenancy = default - Offering class = Standard (Convertable instances can only be changed within the same class) b) Change instance type to the desired type c) Change term to 1 year d) Change Payment Option to "No Upfront" 6) Click "Search". You should see the standard and convertible options. 7) Be sure to update the terraform and Document reservations via tags in the `terragrunt.hcl` of the module in the `xdr-terraform-live` repository. Assignment of reservations to instances get assigned at billing time. The AWS Console does not show us which reservations are assigned to which instances. To find that out, you will need to look at the bill. For customers that renew, the cost savings is lost if they only pay for 9 months. Reserve for as close to 1 year as possible. # Rotating Keys Keys should be rotated regularly (TODO: What does SSP say?). Note: You can save time by doing the rotations all at once. They're separated out for clarity: IMPORTANT: The apply when rotating keys gives an error: `LimitExceeded: Cannot exceed quota for AccessKeysPerUser: 2`. This will go away on the second apply. YOU WILL NEED TO APPLY THE MODULES TWICE. ## SES 1) On the mailrelay servers in test and prod, `sudo cat /etc/postfix/sasl_passwd` and record the Key ID. You do not need the password. test: AKIA2YA7U4A5OCW3IQWA prod: AKIA2QD5QM6CB5IKPLEN 2) In `xdr-terraform-modules, edit `base/mailrelay/ses.tf`. a) Increment both values of `resource "aws_iam_access_key" "ses_access_key-v3"` (e.g. change 3 to 4, and 4 to 5) b) Increment the value of the two outputs to be the new highest key. 3) In `xdr-terraform-live`, run `update_module_refs --module 090-instance-mailrelay --newtag v4.3.11` (substituting appropriate tag) 4) Follow the normal PR procedure. 5) In another window, in `msoc-infrastructure`, edit `salt/pillar/mailrelay_sasl.sls` in preparation to record the updated keys. 6) Run `terragrunt apply` in the 090-instance-mailrelay modules in test and prod. During the apply, Verify that the current keys from step #1 _remain_ (i.e. the keys that are being destroyed do _not_ match the ones you recorded in step 1). This step should destroy the _previous_ keys. NOTE: This will not cause an interruption provided you verify the keys are not being destroyed. NOTE: You will likely need to run the apply twice, as the 'destroy' doesn't happen in time for the creation of the next key. 7) Run the following and update the pillar file: ``` terragrunt output ses_user_smtp_username; terragrunt output ses_user_smtp_password ``` 8) Do a PR to `develop`. 9) On the test salt master, run: ``` sudo salt-run fileserver.update salt mailrelay\* state.sls mailrelay --output-diff test=true salt mailrelay\* state.sls mailrelay --output-diff test=false salt mailrelay\* cmd.run 'echo Hi Fred | mail -s "Test" frederick.t.damstra@accenturefederal.com' ``` 10) Verify receipt of email 11) If all is well, pr to main, and repeat step 10 on the prod salt master. ## moose-hf 1. In `xdr-terraform-modules`, edit `base/account_standards_c2/iam.moose-hf.tf` a. Increment the values for `aws_iam_access_key` by 1 (e.g. change `moose-hf-v0` to `moose-hf-v1`, and `moose-hf-v1` to `moose-hf-v2`) b. Update the `output "access_keys"` to the correct revisions. 2. In `xdr-terraform-live`, run `update_module_refs --module 005-account-standards-c2 --newtag v4.4.1` (updating tag appropriately) 3. PR and tag as usual. 4. For the 4 modules (test and prod in govcloud and commercial), run the `terragrunt apply` (probably twice), followed by `terragrunt output access_keys`, and record the output. NOTE: Changing the keys should not interrupt the gathering of data, as you are removing the _previous_ key, not the in-use key. 5. Log into the moose HF in test, select 'Splunk TA AWS' from the left hand side, click 'configuration', and select 'edit' for `mdr-test-c2-gov` 6. Update the Key ID and Secret Key. NOTE: There is no account in test for commercial (future todo?) 7. Repeat step 5 for prod moose HF (prod will also have the `mdr-prod-c2` account for commercial) ## Salt-Master Note: Only the commercial c2 account gets a key. 1. In `xdr-terraform-modules`, edit `base/salt_master_inventory_role/user.tf` a. Increment both `aws_iam_access_key` versions b. Update the `output "access_keys"` section by incrementing the `v` values (*not* the `[0]` array offset). 2. In `xdr-terraform-live`, edit `test/aws/mdr-test-c2/072-salt-master-inventory-role/terragrunt.hcl`. *NOTE:* This module exists in every account, but only the commercial C2 accounts have keys. 3. In `xdr-terraform-live`, edit `prod/aws/mdr-prod-c2/072-salt-master-inventory-role/terragrunt.hcl`. 4. PR and tag as usual. 5. Run `terragrunt apply` in the two directories (you will likely need to apply twice due ot timing), followed by `terragrunt output access_keys`. Record the values temporarily. 6. For each of the `current` `aws_secret_access_keys`, encrypt via: ``` echo biglongrandomstringofcharacte/rswi/slashes | gpg --armor --encrypt -r salt --batch ``` Note: If you don't have gnupg encrypted, follow the directions in `infrastruture-notes/GnuPG (gpg) Notes.md`. 7. In `msoc-infrastructure`, edit `salt/pillar/xdr_asset_inventory.sls`. 8. Under `secrets: aws_profile: commercial:` for both prod and test, update the `aws_access_key_id` and the encrypted `aws_secret_access_key` Helpful tip: Before replacing the `aws_access_key_id`, compare it to the 'previous' value from your terragrunt output, and it should match. 9. Do a PR to develop. 10. On the test salt master, run: ``` sudo salt-run fileserver.update salt salt\* state.sls salt_master.xdr_asset_inventory --output-diff test=true salt salt\* state.sls salt_master.xdr_asset_inventory --output-diff test=false # In one window, monitor for errors in the program you're about to run: sudo tail -F /var/log/xdr_asset_inventory.log # check for errors # In a second window, run the program: sudo /opt/xdr_asset_inventory/xdr_asset_inventory.sh ``` 11. Do a PR from dev to master and repeat step 10. ## Assuming a Role from the CLI To assume a role from the CLI, as an example: ``` $ aws sts --region us-gov-east-1 assume-role --role-arn "arn:aws-us-gov:iam::738800754746:role/service/xdr-test-portal-shared-artifacts" --role-session-name "FredTesting" { "Credentials": { "AccessKeyId": "REDACTEDU4A5HERBM7VU", "SecretAccessKey": "REDACTEDOi6c8lTadmb6Y8pElptwAMtrduvOXG+K", "SessionToken": "REDACTEDREDACTEDEMj//////////REDACTEREDACTEREDACTEDiRzBFAiEAoKXpgbeEloQgCoBkEijeGfWe9FSLyeDx1PrZfGeMJokCIGBEltsgGGtQ4pmPbLd8k8bAia1dq5cmPY6wMsJIHnTLKpgCCBUQABoMNzM4ODAwNzU0NzQ2IgzllmPbx8zE4lhhBegq9QEEjXSvp+3fHvxE1YlLrkgblP3GvorpDynB/iXlDFZZR82KdNEYUwfkP/y40vqagAR/5fe3r3BhRPPWbeUvwLCGFIapDr3FiWk31b02i3mUsmPFc/Kct4zK8uj2mUlQYNw0dBDyxp82d/tLB2NBzntHl1etuC2ELdfAVF+J8HvdaFD/O9xEySjnwIO7h53NhO4dDALx0hsRrf3Efap0J+Ady1yzDRyrz987Dx1k3sfFJ9UoDkVMqa0VS3ROTa4nR57BUUO0VwF8FOahQbI/BG1FMc34e2imWBhmLsRV7wPChWoW4M8j4Yl6ik4ixHMltQCIUherojDnyPGVBjqdAVOFrRejiVMI6SrmljWkA587j6xnfb4UwZ5wi6p598zYQw7xb7NAH2twbLGlwggp+p4mDvqvDHiJzz8YLrW4pozQnE+FSTbwey6l2rK0OLUNnDJ4HyUXPASNNhtNrbQJUxibTn7NQtdbDDD7Yfq9WZAi49q1Qq5e5pMah0c43beEAsmFUUz2uR55VSGbdkTT938sXHMREDACTED+qNI=", "Expiration": "2022-06-29T15:40:39Z" }, "AssumedRoleUser": { "AssumedRoleId": "REDACTEDU4A5GQGXNWDG3:FredTesting", "Arn": "arn:aws-us-gov:sts::73REDACTED46:assumed-role/xdr-test-portal-shared-artifacts/FredTesting" } } $ export AWS_DEFAULT_REGION=us-gov-east-1 $ export AWS_ACCESS_KEY_ID= $ export AWS_SECRET_ACCESS_KEY= $ export AWS_SESSION_TOKEN= # To confirm it worked: $ aws sts get-caller-identity { "UserId": "REDACTEDU4A5GQGXNWDG3:FredTesting", "Account": "73REDACTED46", "Arn": "arn:aws-us-gov:sts::73REDACTED46:assumed-role/xdr-test-portal-shared-artifacts/FredTesting" } ``` # To unsubscribe an email address from SNS notifications: To remove an email address from all account SNS distributions: ``` for p in `egrep "\[profile" ~/.aws/config | awk '{ print $2 }' | sed "s/\]//" | egrep -v "default|commercial|govcloud"`; do for n in `aws --profile $p sns list-subscriptions --query "Subscriptions[?Endpoint == 'frederick.t.damstra@accenturefederal.com'].SubscriptionArn" | jq -r '.[]'`; do echo aws --profile $p sns unsubscribe --subscription-arn $n aws --profile $p sns unsubscribe --subscription-arn $n done done ```