Home Explore Help
Register Sign In
AFS
/
infrastructure-notes
2
0
Fork 0
Files Issues 0 Pull Requests 0 Wiki
Tree: cd2e3b87c4
Branches Tags
infrastructure-...
/
AWS Notes.md

AWS Notes.md 18 KB
History Raw

AWS Notes.md

For the latest AWS account information see Cloud Accounts Wiki

LEGACY AWS ACCOUNTS

Root Account Alias (in AWS): defpoint-mdr-root
Root Account Alias (for terraform/in ~/.aws/{config,credentials}): mdr-root
Root AWS Account ID: 350838957895
GovCloud Root Account: 701290387780

Test Account ID: 527700175026
Prod Account ID: 477548533976
Org Account ID: 228011623757

assumeRole to the test and prod accounts

terraform has been setup to handle the cis checks for AWS. they are found in terraform/00-cis-hardening.

Get a encoded error message from AWS?

AWS_PROFILE=mdr-test aws sts decode-authorization-message --encoded-message Q7h4sTOW_n_znBB7ojNotL

For prettier output:

aws --profile=mdr-test-c2-gov sts decode-authorization-message --encoded-message Q7h4sTOW_n_znBB7ojNotL... | \
        jq '.["DecodedMessage"]' | \
        sed 's/\\"/"/g' | \
        sed 's/^"//' | \
        sed 's/"$//' | \
        jq

Cloudtrail metric Alarms

so .. cloudtrail writes a trail

that trail is written into a cloudwatch logs log group

in the log group, there are a number of metric filters

the metric filters create metrics, upon which a metric alarm is set

when events matching the metric filter arrive, the metric goes up, triggerting the alarm new messages

the alarm has an SNS topic it writes to that emails me that the "metric was exceeded"

AWS Systems Manager agent

systemctl start amazon-ssm-agent

AWS User Password Distribution ( legecy )

https://gpgtools.org/ download and install
use gpg keychain to generate pub/private keys
something something terraform
echo "wcFMA2sXDKYLpzaU<redacted>bf6clQ043oDkHIrcWK509UIy5GUpEqBV/WLmuCMHkXUgnxy12HY8qBErF58vB7/VXs5pCKp4SDYWEtK73fKmYZ5wJDW6j6OHkpYI4USZXjVYb+Utt56Qprk4KiT6VlFNNPo00r2YDABDdtxPJS3N9REzHqp+7oR2SQkiyEhcF3ZwILk2fH4mc1VQUiFu68RCqbt+QfmDt3OHIRZVPvrS4AHkCbj2fdgkbAaRMJ/21TBn8OE8WuDR4NHh5w/gWeK5m6754DzkjVLxDpsvPG2UR9ErwANEo+BI4upil2vgT+S63PIVsAmTew/7QpPavttP4rUBM47h5cMA"|base64 -D  | gpg -d

#Export in one line and base64
gpg --export | base64

saml2aws

2020-05-07

on OSX, install saml2aws. Follow the instructions in AFS Macbook Notes.md

XDR uses a custom version modified by Duane. We can revert to official if this gets merged.

Additional instructions on Github.

Create ~/.saml2aws with the following:

[commercial]
app_id               =
url                  = https://mdr-multipass.okta.com/home/amazon_aws/0oa3cn8m68DzMXNbc297/272
username             = your.okta.username
provider             = Okta
mfa                  = PUSH
skip_verify          = false
timeout              = 0
aws_urn              = https://signin.aws.amazon.com/saml
aws_session_duration = 28800
aws_profile          = commercial
resource_id          =
subdomain            =
role_arn             = arn:aws:iam::471284459109:role/user/mdr_engineer_readonly
http_attempts_count  =
http_retry_delay     =

[govcloud]
app_id               =
url                  = https://mdr-multipass.okta.com/home/amazon_aws/0oa3e1gtozzfDFAel297/272
username             = your.okta.username
provider             = Okta
mfa                  = PUSH
skip_verify          = false
timeout              = 0
aws_urn              = https://signin.amazonaws-us-gov.com/saml
aws_session_duration = 28800
aws_profile          = govcloud
resource_id          =
subdomain            =
role_arn             = arn:aws-us-gov:iam::701290387780:role/user/mdr_engineer_readonly
http_attempts_count  =
http_retry_delay     =
region               = us-gov-east-1

[cyber-range]
app_id               =
url                  = https://mdr-multipass.okta.com/home/amazon_aws/0oa3cn8m68DzMXNbc297/272
username             = your.okta.username
provider             = Okta
mfa                  = PUSH
skip_verify          = false
timeout              = 0
aws_urn              = https://signin.aws.amazon.com/saml
aws_session_duration = 28800
aws_profile          = cyber-range
resource_id          =
subdomain            =
role_arn             = arn:aws:iam::952430311316:role/user/mdr_engineer_readonly
http_attempts_count  =
http_retry_delay     =

Then you can sign in with:

saml2aws login --idp-account=govcloud

And use the profile via:

aws --profile govcloud ec2 describe-instances

Authentication not working? saml2aws uses the MAC keychain to store your password. When you update your password you need to update the password in keychain as well. Or resetup saml2aws.

AssumeRole Extension and ~/.aws/config

08/03/2020

Install the plugin for chrome/firefox:
Firefox aws-extend-switch-roles

Chrome aws-extend-switch-roles

Copy this into ~/.aws/credentials

[govcloud]
region = us-gov-east-1

[commercial]

Then, create a link from files/config to ~/.aws/config: (your path may vary)

ln -s ~/infrastructure-notes/files/config ~/.aws/config

CloudWatch

bit of code to search cloudwatch

fields @timestamp, @message 
| filter errorCode="AccessDenied" 
| fields coalesce(userIdentity.invokedBy,userIdentity.principalId) as whoo,  coalesce(requestParameters.bucketName,errorMessage) as target
| stats count() as count by bin(1d) as time, whoo, eventName, target
| sort count desc

Using jq with EC2 API

relevant links

dump a list of ec2 instances and some of their relevant fields into a CSV

AWS_PROFILE=dps_prod aws ec2 describe-instances | 
	jq -r '.Reservations[].Instances[] | [ (.Tags[]?|select(.Key=="Name")|.Value?), .InstanceId, .LaunchTime, .StateTransitionReason, .State.Name  ] | @csv'

AWS Instance Scheduler in TEST

Test Ec2 instances get shutdown at night. This is accomplished via a cloudformation template that is configured in Terragrunt. To prevent an EC2 instance from getting shutdown, just remove the Schedule tag.

Fixing bad volume tags

The volume_tags field in aws_instance doesn't handle updates (due to a bug where it was updating other attached volumes).

To add volume tags:

  1. Generate a list of bad volumes:

    aws --profile legacy-mdr-test ec2 describe-volumes --filters "Name=attachment.status,Values=attached" --output yaml | grep VolumeId | awk '{print $2}' | sort -u > badvolumes.legacy-mdr-test.txt

  2. Add the tag to all of them:

    for i in `cat badvolumes.legacy-mdr-test.txt`
do
echo $i
aws --profile legacy-mdr-test ec2 create-tags --resources $i --tags "Key=Snapshot,Value=Daily"
done

AWS Automatic Snapshots

Snapshot Daily

In both test and prod the EC2 EBS Lifecycle Manager handles the automatic backups for EBS Volumes. The Lifecycle Manager has a policy that looks for the tag, Snapshot:Daily. If the volume has that tag, then a snapshot is taken.

Cleaning Old AMIs

The open source amicleaner is very useful for cleaning our old AMIs, but needs a patched version. To install:

# Get a fork of amicleaner with the asg fix:
git clone git@github.com:ronaldosaheki/aws-amicleaner.git
cd aws-amicleaner
git checkout origin/fix-asg
pip3 install --user -e .
ln -s ~/Library/Python/3.9/bin/amicleaner /usr/local/bin/amicleaner

You now have amicleaner in your path, and can run ~/xdr-terraform-live/bin/clean_old_amis.sh.

Rescuing Systems using SSM

For example, the disk space on sensu in test was full.

aws --profile mdr-test-c2-gov \
    ssm send-command --document-name "AWS-RunShellScript" \
    --instance-ids "i-0d5072669fb00c2fb" --comment "SystemRescue" \
    --parameters commands='rm -rf /var/log/sudo-io/*'
# Grab Command ID from the output
aws --profile mdr-test-c2-gov \
   ssm list-command-invocations \
   --command-id ae86ec00-e356-4ad6-b162-6b5ec5260ed9

Instance Reservations

Instance reservatrions are tracked in teh wiki at https://github.xdr.accenturefederalcyber.com/mdr-engineering/msoc-infrastructure/wiki/Reserved-Instances and also recorded in teh tags of the instances via tags in xdr-terraform-live/.../0xx-instance-xxx/terragrunt.hcl.

To reserve an instance, you have to be confident that (a) you're going to need the instance for the next 12 months, and (b) that the instance type won't change. If you don't meet (b), you can still consider a convertible instance reservation.

1) Verify via the Monitoring Dashboard (https://moose-splunk.pvt.xdr.accenturefederalcyber.com/en-US/app/search/freds_monitoring_dashboard) that there are no concerns over utilization, especially with respect to CPU and Memory. 2) Verify the same via the AWS Console. 3) Carefully record the instance types that are going to be reserved. 4) Verify that all the instance have the Platform listed as "Red Hat (Inferred)" / "Red Hat Enterprise Linux" 5) Go to reserved instances: a) Change platform to "Red Hat Enterprise Linux" (Watch out for GitHub/VMRay instance!)

  • Tenancy = default
  • Offering class = Standard (Convertable instances can only be changed within the same class) b) Change instance type to the desired type c) Change term to 1 year d) Change Payment Option to "No Upfront" 6) Click "Search". You should see the standard and convertible options. 7) Be sure to update the terraform and Document reservations via tags in the terragrunt.hcl of the module in the xdr-terraform-live repository.

Assignment of reservations to instances get assigned at billing time. The AWS Console does not show us which reservations are assigned to which instances. To find that out, you will need to look at the bill.

For customers that renew, the cost savings is lost if they only pay for 9 months. Reserve for as close to 1 year as possible.

Rotating Keys

Keys should be rotated regularly (TODO: What does SSP say?).

Note: You can save time by doing the rotations all at once. They're separated out for clarity:

IMPORTANT: The apply when rotating keys gives an error: LimitExceeded: Cannot exceed quota for AccessKeysPerUser: 2. This will go away on the second apply. YOU WILL NEED TO APPLY THE MODULES TWICE.

SES

1) On the mailrelay servers in test and prod, sudo cat /etc/postfix/sasl_passwd and record the Key ID. You do not need the password.

  test: AKIA2YA7U4A5OCW3IQWA
  prod: AKIA2QD5QM6CB5IKPLEN

2) In xdr-terraform-modules, editbase/mailrelay/ses.tf. a) Increment both values ofresource "aws_iam_access_key" "ses_access_key-v3"(e.g. change 3 to 4, and 4 to 5) b) Increment the value of the two outputs to be the new highest key. 3) Inxdr-terraform-live, runupdate_module_refs --module 090-instance-mailrelay --newtag v4.3.11(substituting appropriate tag) 4) Follow the normal PR procedure. 5) In another window, inmsoc-infrastructure, editsalt/pillar/mailrelay_sasl.slsin preparation to record the updated keys. 6) Runterragrunt apply` in the 090-instance-mailrelay modules in test and prod. During the apply, Verify that the current keys from step #1 remain (i.e. the keys that are being destroyed do not match the ones you recorded in step 1). This step should destroy the previous keys. NOTE: This will not cause an interruption provided you verify the keys are not being destroyed. NOTE: You will likely need to run the apply twice, as the 'destroy' doesn't happen in time for the creation of the next key. 7) Run the following and update the pillar file:

terragrunt output ses_user_smtp_username; terragrunt output ses_user_smtp_password

8) Do a PR to develop. 9) On the test salt master, run:

sudo salt-run fileserver.update
salt mailrelay\* state.sls mailrelay --output-diff test=true
salt mailrelay\* state.sls mailrelay --output-diff test=false
salt mailrelay\* cmd.run 'echo Hi Fred | mail -s "Test" frederick.t.damstra@accenturefederal.com'

10) Verify receipt of email 11) If all is well, pr to main, and repeat step 10 on the prod salt master.

moose-hf

  1. In xdr-terraform-modules, edit base/account_standards_c2/iam.moose-hf.tf a. Increment the values for aws_iam_access_key by 1 (e.g. change moose-hf-v0 to moose-hf-v1, and moose-hf-v1 to moose-hf-v2) b. Update the output "access_keys" to the correct revisions.
  2. In xdr-terraform-live, run update_module_refs --module 005-account-standards-c2 --newtag v4.4.1 (updating tag appropriately)
  3. PR and tag as usual.
  4. For the 4 modules (test and prod in govcloud and commercial), run the terragrunt apply (probably twice), followed by terragrunt output access_keys, and record the output. NOTE: Changing the keys should not interrupt the gathering of data, as you are removing the previous key, not the in-use key.
  5. Log into the moose HF in test, select 'Splunk TA AWS' from the left hand side, click 'configuration', and select 'edit' for mdr-test-c2-gov
  6. Update the Key ID and Secret Key. NOTE: There is no account in test for commercial (future todo?)
  7. Repeat step 5 for prod moose HF (prod will also have the mdr-prod-c2 account for commercial)

Salt-Master

Note: Only the commercial c2 account gets a key.

  1. In xdr-terraform-modules, edit base/salt_master_inventory_role/user.tf a. Increment both aws_iam_access_key versions b. Update the output "access_keys" section by incrementing the v values (not the [0] array offset).
  2. In xdr-terraform-live, edit test/aws/mdr-test-c2/072-salt-master-inventory-role/terragrunt.hcl. NOTE: This module exists in every account, but only the commercial C2 accounts have keys.
  3. In xdr-terraform-live, edit prod/aws/mdr-prod-c2/072-salt-master-inventory-role/terragrunt.hcl.
  4. PR and tag as usual.
  5. Run terragrunt apply in the two directories (you will likely need to apply twice due ot timing), followed by terragrunt output access_keys. Record the values temporarily.

  6. For each of the current aws_secret_access_keys, encrypt via:

    echo biglongrandomstringofcharacte/rswi/slashes | gpg --armor --encrypt -r salt --batch

    Note: If you don't have gnupg encrypted, follow the directions in infrastruture-notes/GnuPG (gpg) Notes.md.

  7. In msoc-infrastructure, edit salt/pillar/xdr_asset_inventory.sls.

  8. Under secrets: aws_profile: commercial: for both prod and test, update the aws_access_key_id and the encrypted aws_secret_access_key Helpful tip: Before replacing the aws_access_key_id, compare it to the 'previous' value from your terragrunt output, and it should match.

  9. Do a PR to develop.

  10. On the test salt master, run:

    sudo salt-run fileserver.update
salt salt\* state.sls salt_master.xdr_asset_inventory --output-diff test=true
salt salt\* state.sls salt_master.xdr_asset_inventory --output-diff test=false
# In one window, monitor for errors in the program you're about to run:
sudo tail -F /var/log/xdr_asset_inventory.log # check for errors
# In a second window, run the program:
sudo /opt/xdr_asset_inventory/xdr_asset_inventory.sh

  11. Do a PR from dev to master and repeat step 10.

Assuming a Role from the CLI

To assume a role from the CLI, as an example:

$ aws sts --region us-gov-east-1 assume-role --role-arn "arn:aws-us-gov:iam::738800754746:role/service/xdr-test-portal-shared-artifacts"  --role-session-name "FredTesting"
{
    "Credentials": {
        "AccessKeyId": "REDACTEDU4A5HERBM7VU",
        "SecretAccessKey": "REDACTEDOi6c8lTadmb6Y8pElptwAMtrduvOXG+K",
        "SessionToken": "REDACTEDREDACTEDEMj//////////REDACTEREDACTEREDACTEDiRzBFAiEAoKXpgbeEloQgCoBkEijeGfWe9FSLyeDx1PrZfGeMJokCIGBEltsgGGtQ4pmPbLd8k8bAia1dq5cmPY6wMsJIHnTLKpgCCBUQABoMNzM4ODAwNzU0NzQ2IgzllmPbx8zE4lhhBegq9QEEjXSvp+3fHvxE1YlLrkgblP3GvorpDynB/iXlDFZZR82KdNEYUwfkP/y40vqagAR/5fe3r3BhRPPWbeUvwLCGFIapDr3FiWk31b02i3mUsmPFc/Kct4zK8uj2mUlQYNw0dBDyxp82d/tLB2NBzntHl1etuC2ELdfAVF+J8HvdaFD/O9xEySjnwIO7h53NhO4dDALx0hsRrf3Efap0J+Ady1yzDRyrz987Dx1k3sfFJ9UoDkVMqa0VS3ROTa4nR57BUUO0VwF8FOahQbI/BG1FMc34e2imWBhmLsRV7wPChWoW4M8j4Yl6ik4ixHMltQCIUherojDnyPGVBjqdAVOFrRejiVMI6SrmljWkA587j6xnfb4UwZ5wi6p598zYQw7xb7NAH2twbLGlwggp+p4mDvqvDHiJzz8YLrW4pozQnE+FSTbwey6l2rK0OLUNnDJ4HyUXPASNNhtNrbQJUxibTn7NQtdbDDD7Yfq9WZAi49q1Qq5e5pMah0c43beEAsmFUUz2uR55VSGbdkTT938sXHMREDACTED+qNI=",
        "Expiration": "2022-06-29T15:40:39Z"
    },
    "AssumedRoleUser": {
        "AssumedRoleId": "REDACTEDU4A5GQGXNWDG3:FredTesting",
        "Arn": "arn:aws-us-gov:sts::73REDACTED46:assumed-role/xdr-test-portal-shared-artifacts/FredTesting"
    }
}
$ export AWS_DEFAULT_REGION=us-gov-east-1
$ export AWS_ACCESS_KEY_ID= <paste from above>
$ export AWS_SECRET_ACCESS_KEY= <paste from above>
$ export AWS_SESSION_TOKEN= <paste from above>
# To confirm it worked:
$  aws sts get-caller-identity
{
    "UserId": "REDACTEDU4A5GQGXNWDG3:FredTesting",
    "Account": "73REDACTED46",
    "Arn": "arn:aws-us-gov:sts::73REDACTED46:assumed-role/xdr-test-portal-shared-artifacts/FredTesting"
}

To unsubscribe an email address from SNS notifications:

To remove an email address from all account SNS distributions:

for p in `egrep "\[profile" ~/.aws/config | awk '{ print $2 }' | sed "s/\]//" | egrep -v "default|commercial|govcloud"`; do
  for n in `aws --profile $p sns list-subscriptions --query "Subscriptions[?Endpoint == 'frederick.t.damstra@accenturefederal.com'].SubscriptionArn" | jq -r '.[]'`; do
    echo aws --profile $p sns unsubscribe --subscription-arn $n
    aws --profile $p sns unsubscribe --subscription-arn $n
  done
done