# AWS Notes.md

For the latest AWS account information see [Cloud Accounts Wiki](https://github.xdr.accenturefederalcyber.com/mdr-engineering/msoc-infrastructure/wiki/cloud-accounts)


### LEGACY AWS ACCOUNTS
Root Account Alias (in AWS): defpoint-mdr-root     
Root Account Alias (for terraform/in `~/.aws/{config,credentials}`): mdr-root     
Root AWS Account ID:   350838957895     
GovCloud Root Account: 701290387780     

Test Account ID: 527700175026     
Prod Account ID: 477548533976     
Org  Account ID: 228011623757     

`assumeRole` to the test and prod accounts

---
terraform has been setup to handle the cis checks for AWS. they are found in terraform/00-cis-hardening.

Get a encoded error message from AWS?

```
AWS_PROFILE=mdr-test aws sts decode-authorization-message --encoded-message Q7h4sTOW_n_znBB7ojNotL
```

For prettier output:
```
aws --profile=mdr-test-c2-gov sts decode-authorization-message --encoded-message Q7h4sTOW_n_znBB7ojNotL... | \
        jq '.["DecodedMessage"]' | \
        sed 's/\\"/"/g' | \
        sed 's/^"//' | \
        sed 's/"$//' | \
        jq
```


-------------------------------------------
Cloudtrail metric Alarms


so .. cloudtrail writes a trail

that trail is written into a cloudwatch logs log group

in the log group, there are a number of metric filters

the metric filters create metrics, upon which a metric alarm is set

when events matching the metric filter arrive, the metric goes up, triggerting the alarm
new messages

the alarm has an SNS topic it writes to that emails me that the "metric was exceeded"



----------------------------------------------
AWS Systems Manager agent

systemctl start amazon-ssm-agent

----------------------------------------------

## AWS User Password Distribution ( legecy )

```
https://gpgtools.org/ download and install
use gpg keychain to generate pub/private keys
something something terraform
echo "wcFMA2sXDKYLpzaU<redacted>bf6clQ043oDkHIrcWK509UIy5GUpEqBV/WLmuCMHkXUgnxy12HY8qBErF58vB7/VXs5pCKp4SDYWEtK73fKmYZ5wJDW6j6OHkpYI4USZXjVYb+Utt56Qprk4KiT6VlFNNPo00r2YDABDdtxPJS3N9REzHqp+7oR2SQkiyEhcF3ZwILk2fH4mc1VQUiFu68RCqbt+QfmDt3OHIRZVPvrS4AHkCbj2fdgkbAaRMJ/21TBn8OE8WuDR4NHh5w/gWeK5m6754DzkjVLxDpsvPG2UR9ErwANEo+BI4upil2vgT+S63PIVsAmTew/7QpPavttP4rUBM47h5cMA"|base64 -D  | gpg -d
```

```
#Export in one line and base64
gpg --export | base64
```
## saml2aws
2020-05-07

on OSX, install `saml2aws`. Follow the instructions in AFS Macbook Notes.md

XDR uses a custom version modified by Duane. We can revert to official if this gets merged.

Additional instructions on [Github](https://github.com/Versent/saml2aws).

Create `~/.saml2aws` with the following:
```
[commercial]
app_id               =
url                  = https://mdr-multipass.okta.com/home/amazon_aws/0oa3cn8m68DzMXNbc297/272
username             = your.okta.username
provider             = Okta
mfa                  = PUSH
skip_verify          = false
timeout              = 0
aws_urn              = https://signin.aws.amazon.com/saml
aws_session_duration = 28800
aws_profile          = commercial
resource_id          =
subdomain            =
role_arn             = arn:aws:iam::471284459109:role/user/mdr_engineer_readonly
http_attempts_count  =
http_retry_delay     =

[govcloud]
app_id               =
url                  = https://mdr-multipass.okta.com/home/amazon_aws/0oa3e1gtozzfDFAel297/272
username             = your.okta.username
provider             = Okta
mfa                  = PUSH
skip_verify          = false
timeout              = 0
aws_urn              = https://signin.amazonaws-us-gov.com/saml
aws_session_duration = 28800
aws_profile          = govcloud
resource_id          =
subdomain            =
role_arn             = arn:aws-us-gov:iam::701290387780:role/user/mdr_engineer_readonly
http_attempts_count  =
http_retry_delay     =
region               = us-gov-east-1

[cyber-range]
app_id               =
url                  = https://mdr-multipass.okta.com/home/amazon_aws/0oa3cn8m68DzMXNbc297/272
username             = your.okta.username
provider             = Okta
mfa                  = PUSH
skip_verify          = false
timeout              = 0
aws_urn              = https://signin.aws.amazon.com/saml
aws_session_duration = 28800
aws_profile          = cyber-range
resource_id          =
subdomain            =
role_arn             = arn:aws:iam::952430311316:role/user/mdr_engineer_readonly
http_attempts_count  =
http_retry_delay     =
```

Then you can sign in with:
```
saml2aws login --idp-account=govcloud
```

And use the profile via:
```
aws --profile govcloud ec2 describe-instances
```

Authentication not working? `saml2aws` uses the MAC keychain to store your password. When you update your password you need to update the password in keychain as well. Or resetup saml2aws. 

## AssumeRole Extension and ~/.aws/config
08/03/2020

Install the plugin for chrome/firefox:     
Firefox [aws-extend-switch-roles](https://addons.mozilla.org/en-US/firefox/addon/aws-extend-switch-roles3/#&gid=1&pid=3) 

Chrome [aws-extend-switch-roles](https://chrome.google.com/webstore/detail/aws-extend-switch-roles/jpmkfafbacpgapdghgdpembnojdlgkdl/related?hl=en)

Copy this into `~/.aws/credentials`
```
[govcloud]
region = us-gov-east-1

[commercial]
```

Then, create a link from `files/config` to `~/.aws/config`: (your path may vary)
```
ln -s ~/infrastructure-notes/files/config ~/.aws/config
```

## CloudWatch 

bit of code to search cloudwatch
```
fields @timestamp, @message 
| filter errorCode="AccessDenied" 
| fields coalesce(userIdentity.invokedBy,userIdentity.principalId) as whoo,  coalesce(requestParameters.bucketName,errorMessage) as target
| stats count() as count by bin(1d) as time, whoo, eventName, target
| sort count desc
```


### Using jq with EC2 API

relevant links

* [Reshaping JSON with jq](https://programminghistorian.org/en/lessons/json-and-jq)
* [AWS, JQ and bash command cheat sheet](https://gist.github.com/lukeplausin/b64c10f8b524bb310e0083756c42caf6)

#### dump a list of ec2 instances and some of their relevant fields into a CSV

```
AWS_PROFILE=dps_prod aws ec2 describe-instances | 
	jq -r '.Reservations[].Instances[] | [ (.Tags[]?|select(.Key=="Name")|.Value?), .InstanceId, .LaunchTime, .StateTransitionReason, .State.Name  ] | @csv'
```

### AWS Instance Scheduler in TEST

Test Ec2 instances get shutdown at night. This is accomplished via a cloudformation template that is configured in Terragrunt. To prevent an EC2 instance from getting shutdown, just remove the Schedule tag.

## Fixing bad volume tags

The `volume_tags` field in `aws_instance` doesn't handle updates (due to a bug where it was updating other attached volumes).

To add volume tags:

1. Generate a list of bad volumes:
```
aws --profile legacy-mdr-test ec2 describe-volumes --filters "Name=attachment.status,Values=attached" --output yaml | grep VolumeId | awk '{print $2}' | sort -u > badvolumes.legacy-mdr-test.txt
```

2. Add the tag to all of them:
```
for i in `cat badvolumes.legacy-mdr-test.txt`
do
  echo $i
  aws --profile legacy-mdr-test ec2 create-tags --resources $i --tags "Key=Snapshot,Value=Daily"
done
```

### AWS Automatic Snapshots

Snapshot Daily

In both test and prod the EC2 EBS Lifecycle Manager handles the automatic backups for EBS Volumes. The Lifecycle Manager has a policy that looks for the tag, Snapshot:Daily. If the volume has that tag, then a snapshot is taken. 

### Cleaning Old AMIs

The open source `amicleaner` is very useful for cleaning our old AMIs, but needs a patched version. To install:
```
# Get a fork of amicleaner with the asg fix:
git clone git@github.com:ronaldosaheki/aws-amicleaner.git
cd aws-amicleaner
git checkout origin/fix-asg
pip3 install --user -e .
ln -s ~/Library/Python/3.9/bin/amicleaner /usr/local/bin/amicleaner
```

You now have amicleaner in your path, and can run `~/xdr-terraform-live/bin/clean_old_amis.sh`.

## Rescuing Systems using SSM

For example, the disk space on sensu in test was full.

```
aws --profile mdr-test-c2-gov \
    ssm send-command --document-name "AWS-RunShellScript" \
    --instance-ids "i-0d5072669fb00c2fb" --comment "SystemRescue" \
    --parameters commands='rm -rf /var/log/sudo-io/*'
# Grab Command ID from the output
aws --profile mdr-test-c2-gov \
   ssm list-command-invocations \
   --command-id ae86ec00-e356-4ad6-b162-6b5ec5260ed9
```

# Instance Reservations

To reserve an instance, you have to be confident that (a) you're going to need the instance for the next 12 months, and (b) that the instance type won't change. If you don't meet (b), you can still consider a convertible instance reservation.

1) Verify via the Monitoring Dashboard (https://moose-splunk.pvt.xdr.accenturefederalcyber.com/en-US/app/search/freds_monitoring_dashboard) that there are no concerns over utilization, especially with respect to CPU and Memory.
2) Verify the same via the AWS Console.
3) Carefully record the instance types that are going to be reserved.
4) Verify that all the instance have the Platform listed as "Red Hat (Inferred)"
5) Go to reserved instances:
  a) Change platform to "Red Hat Enterprise Linux"
  b) Change instance type to the desired type
  c) Change term to 1 year
  d) Change Payment Option to "No Upfront"
6) Click "Search". You should see the standard and convertible options.
7) Be sure to update the terraform and Document reservations via tags in the `terragrunt.hcl` of the module in the `xdr-terraform-live` repository.

# Rotating Keys

Keys should be rotated regularly (TODO: What does SSP say?).

Note: You can save time by doing the rotations all at once. They're separated out for clarity:

IMPORTANT: The apply when rotating keys gives an error:
`LimitExceeded: Cannot exceed quota for AccessKeysPerUser: 2`.
This will go away on the second apply. YOU WILL NEED TO APPLY THE MODULES TWICE.

## SES

1) On the mailrelay servers in test and prod, `sudo cat /etc/postfix/sasl_passwd` and record the Key ID. You do not need the password.
      test: AKIA2YA7U4A5OCW3IQWA
      prod: AKIA2QD5QM6CB5IKPLEN
2) In `xdr-terraform-modules, edit `base/mailrelay/ses.tf`. 
  a) Increment both values of `resource "aws_iam_access_key" "ses_access_key-v3"` (e.g. change 3 to 4, and 4 to 5)
  b) Increment the value of the two outputs to be the new highest key.
3) In `xdr-terraform-live`, run `update_module_refs --module 090-instance-mailrelay --newtag v4.3.11` (substituting appropriate tag)
4) Follow the normal PR procedure.
5) In another window, in `msoc-infrastructure`, edit `salt/pillar/mailrelay_sasl.sls` in preparation to record the updated keys.
6) Run `terragrunt apply` in the 090-instance-mailrelay modules in test and prod. During the apply, Verify that the current keys from step #1 _remain_ (i.e. the keys that are being destroyed do _not_ match the ones you recorded in step 1). This step should destroy the _previous_ keys.
NOTE: This will not cause an interruption provided you verify the keys are not being destroyed.
NOTE: You will likely need to run the apply twice, as the 'destroy' doesn't happen in time for the creation of the next key.
7) Run the following and update the pillar file:
```
terragrunt output ses_user_smtp_username; terragrunt output ses_user_smtp_password
```
8) Do a PR to `develop`.
9) On the test salt master, run:
```
sudo salt-run fileserver.update
salt mailrelay\* state.sls mailrelay --output-diff test=true
salt mailrelay\* state.sls mailrelay --output-diff test=false
salt mailrelay\* cmd.run 'echo Hi Fred | mail -s "Test" frederick.t.damstra@accenturefederal.com'
```
10) Verify receipt of email
11) If all is well, pr to main, and repeat step 10 on the prod salt master.

## moose-hf

1. In `xdr-terraform-modules`, edit `base/account_standards_c2/iam.moose-hf.tf`
  a. Increment the values for `aws_iam_access_key` by 1 (e.g. change `moose-hf-v0` to `moose-hf-v1`, and `moose-hf-v1` to `moose-hf-v2`)
  b. Update the `output "access_keys"` to the correct revisions.
2. In `xdr-terraform-live`, run `update_module_refs --module 005-account-standards-c2 --newtag v4.4.1` (updating tag appropriately)
3. PR and tag as usual.
4. For the 4 modules (test and prod in govcloud and commercial), run the `terragrunt apply` (probably twice), followed by `terragrunt output access_keys`, and record the output.
NOTE: Changing the keys should not interrupt the gathering of data, as you are removing the _previous_ key, not the in-use key.
5. Log into the moose HF in test, select 'Splunk TA AWS' from the left hand side, click 'configuration', and select 'edit' for `mdr-test-c2-gov`
6. Update the Key ID and Secret Key.
NOTE: There is no account in test for commercial (future todo?)
7. Repeat step 5 for prod moose HF (prod will also have the `mdr-prod-c2` account for commercial)

## Salt-Master

Note: Only the commercial c2 account gets a key.

1. In `xdr-terraform-modules`, edit `base/salt_master_inventory_role/user.tf`
  a. Increment both `aws_iam_access_key` versions
  b. Update the `output "access_keys"` section by incrementing the `v` values (*not* the `[0]` array offset).
2. In `xdr-terraform-live`, edit `test/aws/mdr-test-c2/072-salt-master-inventory-role/terragrunt.hcl`. *NOTE:* This module exists in every account, but only the commercial C2 accounts have keys.
3. In `xdr-terraform-live`, edit `prod/aws/mdr-prod-c2/072-salt-master-inventory-role/terragrunt.hcl`.
4. PR and tag as usual.
5. Run `terragrunt apply` in the two directories (you will likely need to apply twice due ot timing), followed by `terragrunt output access_keys`. Record the values temporarily.
6. For each of the `current` `aws_secret_access_keys`, encrypt via:
```
echo biglongrandomstringofcharacte/rswi/slashes | gpg --armor --encrypt -r salt --batch
```
Note: If you don't have gnupg encrypted, follow the directions in `infrastruture-notes/GnuPG (gpg) Notes.md`.
7. In `msoc-infrastructure`, edit `salt/pillar/xdr_asset_inventory.sls`.
8. Under `secrets: aws_profile: commercial:` for both prod and test, update the `aws_access_key_id` and the encrypted `aws_secret_access_key`
   Helpful tip: Before replacing the `aws_access_key_id`, compare it to the 'previous' value from your terragrunt output, and it should match.
9. Do a PR to develop.
10. On the test salt master, run:
```
sudo salt-run fileserver.update
salt salt\* state.sls salt_master.xdr_asset_inventory --output-diff test=true
salt salt\* state.sls salt_master.xdr_asset_inventory --output-diff test=false
# In one window, monitor for errors in the program you're about to run:
sudo tail -F /var/log/xdr_asset_inventory.log # check for errors
# In a second window, run the program:
sudo /opt/xdr_asset_inventory/xdr_asset_inventory.sh
```
11. Do a PR from dev to master and repeat step 10.

## Assuming a Role from the CLI

To assume a role from the CLI, as an example:
```
$ aws sts --region us-gov-east-1 assume-role --role-arn "arn:aws-us-gov:iam::738800754746:role/service/xdr-test-portal-shared-artifacts"  --role-session-name "FredTesting"
{
    "Credentials": {
        "AccessKeyId": "REDACTEDU4A5HERBM7VU",
        "SecretAccessKey": "REDACTEDOi6c8lTadmb6Y8pElptwAMtrduvOXG+K",
        "SessionToken": " ",
        "Expiration": "2022-06-29T15:40:39Z"
    },
    "AssumedRoleUser": {
        "AssumedRoleId": "REDACTEDU4A5GQGXNWDG3:FredTesting",
        "Arn": "arn:aws-us-gov:sts::73REDACTED46:assumed-role/xdr-test-portal-shared-artifacts/FredTesting"
    }
}
$ export AWS_DEFAULT_REGION=us-gov-east-1
$ export AWS_ACCESS_KEY_ID= <paste from above>
$ export AWS_SECRET_ACCESS_KEY= <paste from above>
$ export AWS_SESSION_TOKEN= <paste from above>
# To confirm it worked:
$  aws sts get-caller-identity
{
    "UserId": "REDACTEDU4A5GQGXNWDG3:FredTesting",
    "Account": "73REDACTED46",
    "Arn": "arn:aws-us-gov:sts::73REDACTED46:assumed-role/xdr-test-portal-shared-artifacts/FredTesting"
}
```
