AWS Notes.md

For the latest AWS account information see Cloud Accounts Wiki

LEGACY AWS ACCOUNTS

Root Account Alias (in AWS): defpoint-mdr-root
Root Account Alias (for terraform/in ~/.aws/{config,credentials}): mdr-root
Root AWS Account ID: 350838957895
GovCloud Root Account: 701290387780

Test Account ID: 527700175026
Prod Account ID: 477548533976
Org Account ID: 228011623757

assumeRole to the test and prod accounts

terraform has been setup to handle the cis checks for AWS. they are found in terraform/00-cis-hardening.

Get a encoded error message from AWS?

AWS_PROFILE=mdr-test aws sts decode-authorization-message --encoded-message Q7h4sTOW_n_znBB7ojNotL

For prettier output:

aws --profile=mdr-test-c2-gov sts decode-authorization-message --encoded-message Q7h4sTOW_n_znBB7ojNotL... | \
        jq '.["DecodedMessage"]' | \
        sed 's/\\"/"/g' | \
        sed 's/^"//' | \
        sed 's/"$//' | \
        jq

Cloudtrail metric Alarms

so .. cloudtrail writes a trail

that trail is written into a cloudwatch logs log group

in the log group, there are a number of metric filters

the metric filters create metrics, upon which a metric alarm is set

when events matching the metric filter arrive, the metric goes up, triggerting the alarm new messages

the alarm has an SNS topic it writes to that emails me that the "metric was exceeded"

AWS Systems Manager agent

systemctl start amazon-ssm-agent

AWS User Password Distribution ( legecy )

https://gpgtools.org/ download and install
use gpg keychain to generate pub/private keys
something something terraform
echo "wcFMA2sXDKYLpzaU<redacted>bf6clQ043oDkHIrcWK509UIy5GUpEqBV/WLmuCMHkXUgnxy12HY8qBErF58vB7/VXs5pCKp4SDYWEtK73fKmYZ5wJDW6j6OHkpYI4USZXjVYb+Utt56Qprk4KiT6VlFNNPo00r2YDABDdtxPJS3N9REzHqp+7oR2SQkiyEhcF3ZwILk2fH4mc1VQUiFu68RCqbt+QfmDt3OHIRZVPvrS4AHkCbj2fdgkbAaRMJ/21TBn8OE8WuDR4NHh5w/gWeK5m6754DzkjVLxDpsvPG2UR9ErwANEo+BI4upil2vgT+S63PIVsAmTew/7QpPavttP4rUBM47h5cMA"|base64 -D  | gpg -d

#Export in one line and base64
gpg --export | base64

saml2aws

2020-05-07

on OSX, install saml2aws via:

# 2021-04-21 - you should not need to install the tap any more
#brew tap versent/homebrew-taps
brew install saml2aws

otherwise follow instructions on Github.

Create ~/.saml2aws with the following:

[commercial]
app_id               =
url                  = https://mdr-multipass.okta.com/home/amazon_aws/0oa3cn8m68DzMXNbc297/272
username             = your.okta.username
provider             = Okta
mfa                  = PUSH
skip_verify          = false
timeout              = 0
aws_urn              = https://signin.aws.amazon.com/saml
aws_session_duration = 28800
aws_profile          = commercial
resource_id          =
subdomain            =
role_arn             = arn:aws:iam::471284459109:role/user/mdr_engineer_readonly
http_attempts_count  =
http_retry_delay     =

[govcloud]
app_id               =
url                  = https://mdr-multipass.okta.com/home/amazon_aws/0oa3e1gtozzfDFAel297/272
username             = your.okta.username
provider             = Okta
mfa                  = PUSH
skip_verify          = false
timeout              = 0
aws_urn              = https://signin.amazonaws-us-gov.com/saml
aws_session_duration = 28800
aws_profile          = govcloud
resource_id          =
subdomain            =
role_arn             = arn:aws-us-gov:iam::701290387780:role/user/mdr_engineer_readonly
http_attempts_count  =
http_retry_delay     =
region               = us-gov-east-1

[cyber-range]
app_id               =
url                  = https://mdr-multipass.okta.com/home/amazon_aws/0oa3cn8m68DzMXNbc297/272
username             = your.okta.username
provider             = Okta
mfa                  = PUSH
skip_verify          = false
timeout              = 0
aws_urn              = https://signin.aws.amazon.com/saml
aws_session_duration = 28800
aws_profile          = cyber-range
resource_id          =
subdomain            =
role_arn             = arn:aws:iam::952430311316:role/user/mdr_engineer_readonly
http_attempts_count  =
http_retry_delay     =

Then you can sign in with:

saml2aws login --idp-account=govcloud

And use the profile via:

aws --profile govcloud ec2 describe-instances

Authentication not working? saml2aws uses the MAC keychain to store your password. When you update your password you need to update the password in keychain as well. Or resetup saml2aws.

AssumeRole Extension and ~/.aws/config

08/03/2020

Install the plugin for chrome/firefox:
Firefox aws-extend-switch-roles

Chrome aws-extend-switch-roles

Copy this into ~/.aws/credentials

[govcloud]
region = us-gov-east-1

[commercial]

Then, create a link from files/config to ~/.aws/config: (your path may vary)

ln -s ~/infrastructure-notes/files/config ~/.aws/config

CloudWatch

bit of code to search cloudwatch

fields @timestamp, @message 
| filter errorCode="AccessDenied" 
| fields coalesce(userIdentity.invokedBy,userIdentity.principalId) as whoo,  coalesce(requestParameters.bucketName,errorMessage) as target
| stats count() as count by bin(1d) as time, whoo, eventName, target
| sort count desc

Using jq with EC2 API

relevant links

dump a list of ec2 instances and some of their relevant fields into a CSV

AWS_PROFILE=dps_prod aws ec2 describe-instances | 
	jq -r '.Reservations[].Instances[] | [ (.Tags[]?|select(.Key=="Name")|.Value?), .InstanceId, .LaunchTime, .StateTransitionReason, .State.Name  ] | @csv'

AWS Instance Scheduler in TEST

Test Ec2 instances get shutdown at night. This is accomplished via a lambda application. To prevent an EC2 instance from getting shutdown, just remove the Schedule tag.

Fixing bad volume tags

The volume_tags field in aws_instance doesn't handle updates (due to a bug where it was updating other attached volumes).

To add volume tags:

Generate a list of bad volumes:

aws --profile legacy-mdr-test ec2 describe-volumes --filters "Name=attachment.status,Values=attached" --output yaml | grep VolumeId | awk '{print $2}' | sort -u > badvolumes.legacy-mdr-test.txt

Add the tag to all of them:

for i in `cat badvolumes.legacy-mdr-test.txt`
do
echo $i
aws --profile legacy-mdr-test ec2 create-tags --resources $i --tags "Key=Snapshot,Value=Daily"
done

AWS Automatic Snapshots

Snapshot Daily

In both test and prod the EC2 EBS Lifecycle Manager handles the automatic backups for EBS Volumes. The Lifecycle Manager has a policy that looks for the tag, Snapshot:Daily. If the volume has that tag, then a snapshot is taken.

Cleaning Old AMIs

The open source amicleaner is very useful for cleaning our old AMIs, but needs a patched version. To install:

# Get a fork of amicleaner with the asg fix:
git clone git@github.com:ronaldosaheki/aws-amicleaner.git
cd aws-amicleaner
git checkout origin/fix-asg
pip3 install --user -e .
ln -s ~/Library/Python/3.9/bin/amicleaner /usr/local/bin/amicleaner

You now have amicleaner in your path, and can run ~/xdr-terraform-live/bin/clean_old_amis.sh.

AWS Notes.md 7.7 KB Historie Surový