# Splunk App and Add-on for AWS Notes

# Installation
2020-08-27

## Prepare the Accounts/roles:

1. Update '005-account-standards-c2' to latest
2. Update '006-account-standards' to latest
3. Repeat for other accounts

## Keys:

To get access keys:
```
cd xdr-terraform-live/test/aws-us-gov/mdr-test-c2/005-account-standards-c2/
terragrunt output
```

Returns something like:
```
access_keys = {
  "current" = {
    "aws_access_key_id" = "BLAHBLAHBLAH"
    "aws_secret_access_key" = "SecretBlahBlahBlah"
  }
  "previous" = {
    "aws_access_key_id" = "BLAHBLAHBLAH"
    "aws_secret_access_key" = "SecretBlahBlahBlah"
  }
}
```

Use the 'current' values. Previous values are for key rotation.

n.b. on 2020-09-01, actual keys were accidentally leaked here. They were all destroyed and regenerated.

## AWS Addon

The addon should be installed on teh search heads and heavy forwarders

```
scp ~/Downloads/Splunk/Apps/splunk-add-on-for-amazon-web-services_501.tgz dev-moose-splunk-hf:
```

* Note, may want to install on SH: *
You can install this add-on on a search head cluster for all search-time functionality, but configure inputs on forwarders to avoid duplicate data collection.
Before installing this add-on to a cluster, make the following changes to the add-on package:
1. Remove the eventgen.conf files and all files in the samples folder
2. Remove the inputs.conf file.

```
ssh dev-moose-splunk-hf
sudo tar cvzf aws_app.backups.`date "+%Y%m%d"`tar.gz /opt/splunk/etc/apps/{TA-AFS_aws,TA-AFS_aws_addon}
tar xvzf splunk-add-on-for-amazon-web-services_501.tgz
cd /opt/splunk/etc/apps/
sudo mv ~frederick_t_damstra/Splunk_TA_aws .
sudo chown -R splunk:splunk Splunk_TA_aws
sudo -u splunk /opt/splunk/bin/splunk restart
```

## Configure the add-on

### Proxy
1. Browse to the HF via mdr okta.
2. Click on the add-on
3. Click on configuration
4. Configure the proxy to proxy.
  * Enable: Yes
  * Host: proxy.pvt.xdrtest.accenturefederalcyber.com
  * Port: 80

### C2 Account in other partition
1. Go to configuration->Accounts
2. Configure the govcloud account:
  * mdr-test-c2-gov
  * access key
  * secret key
  * Region

### IAM Roles
1. Go to configuration->IAM Role

Skip:
2. Click on 'Add'
  * name: mdrtestc2gov # Doesn't accept dashes or spaces
  * arn: arn:aws-us-gov:iam::738800754746:role/instance/splunk-addon-for-aws
3. Click 'Add' again
  * name: mdrtestc2
  * arn: arn:aws:iam::816914342178:role/instance/splunk-addon-for-aws
4. Add another account for testing purposes
  * name: mdrtestmalwaregov
  * arn: arn:aws-us-gov:iam::876942499057:role/instance/splunk_addon_for_aws

Instead:
Edit `splunk_ta_aws_iam_roles.conf`:
```
[mdrtestc2]
#arn = arn:aws:iam::816914342178:role/instance/splunk-addon-for-aws
arn = arn:aws:iam::816914342178:role/instance/splunk-addon-for-aws
disabled = 0

[mdrtestc2gov]
#arn = arn:aws-us-gov:iam::738800754746:role/instance/splunk-addon-for-aws
arn = arn:aws-us-gov:iam::738800754746:role/instance/splunk-addon-for-aws
disabled = 0

[mdrtestmalwaregov]
#arn = arn:aws-us-gov:iam::876942499057:role/instance/splunk-addon-for-aws
arn = arn:aws-us-gov:iam::876942499057:role/instance/splunk-addon-for-aws
disabled = 0

... 
repeat for other accounts
```

### Cloudtrail Inputs
1. Go to configuration->inputs
2. Create new inputs -> cloud trail -> SQS based S3
3. Set:
  * Name: CloudTrail-GovCloud
  * AWS Account: mdr-test-c2-gov
  * AssumeRole: mdrtestc2gov
  * AWS Region: us-gov-east-1
  * SQS Queue Name: new objects for ...
  * Index: app_aws
  * ... and defaults * Probably want a separate index for production *

Repeat for CloudTrail-Commercial
Repeat for Config-{GovCloud,Commercial}

## CM
```
ssh dev-moose-splunk-cm
sudo su - splunk
cd /opt/splunk/etc/master-apps/
mkdir -p splunk_app_aws/local
vim splunk_app_aws/local/indexes.conf
```

Paste:
```
[aws_topology_history]
coldPath = volume:normal_primary/$_index_name/colddb
homePath = volume:normal_primary/$_index_name/db
thawedPath = $SPLUNK_DB/$_index_name/thaweddb
# frozen time is 7 days
frozenTimePeriodInSecs = 604800
maxHotIdleSecs = 3600
repFactor = auto

[aws_topology_daily_snapshot]
coldPath = volume:normal_primary/$_index_name/colddb
homePath = volume:normal_primary/$_index_name/db
thawedPath = $SPLUNK_DB/$_index_name/thaweddb
#frozen time is about 6 months
frozenTimePeriodInSecs = 15552000
maxHotIdleSecs = 3600
repFactor = auto

[aws_topology_monthly_snapshot]
coldPath = volume:normal_primary/$_index_name/colddb
homePath = volume:normal_primary/$_index_name/db
thawedPath = $SPLUNK_DB/$_index_name/thaweddb
# frozen time is 365 days
frozenTimePeriodInSecs = 31536000
maxHotIdleSecs = 86400
repFactor = auto

[aws_topology_playback]
coldPath = volume:normal_primary/$_index_name/colddb
homePath = volume:normal_primary/$_index_name/db
thawedPath = $SPLUNK_DB/$_index_name/thaweddb
#frozen time is about 6 months
frozenTimePeriodInSecs = 15552000
maxHotIdleSecs = 3600
repFactor = auto

[aws_vpc_flow_logs]
coldPath = volume:normal_primary/$_index_name/colddb
homePath = volume:normal_primary/$_index_name/db
thawedPath = $SPLUNK_DB/$_index_name/thaweddb
# frozen time is 7 days
frozenTimePeriodInSecs = 604800
maxHotIdleSecs = 3600
repFactor = auto

[aws_anomaly_detection]
coldPath = volume:normal_primary/$_index_name/colddb
homePath = volume:normal_primary/$_index_name/db
thawedPath = $SPLUNK_DB/$_index_name/thaweddb
repFactor = auto
```

validate and push cluster bundle

## SH

```
scp ~/Downloads/Splunk/Apps/{python-for-scientific-computing-for-linux-64-bit_202.tgz,splunk-app-for-aws_601.tgz,splunk-add-on-for-amazon-web-services_501.tgz} dev-moose-splunk-sh:
ssh dev-moose-splunk-sh
sudo tar cvzf aws_app.backups.`date "+%Y%m%d"`tar.gz /opt/splunk/etc/apps/{TA-AFS_aws,TA-AFS_aws_addon}
for i in *.tgz; do tar xvzf $i; done
# The add-on
sudo cp -r /opt/splunk/etc/apps/TA-AFS_aws_addon/local Splunk_TA_aws/local
# check this again, find which two of these three are actually 
sudo cp /opt/splunk/etc/apps/TA-AFS_aws_addon/lookups/{aws_console_login_signature.csv,aws_config_action_lookup.csv,cloudtrail_change_type.csv} Splunk_TA_aws/lookups/
sudo rm -rf /opt/splunk/etc/apps/TA-AFS_aws_addon
sudo mv Splunk_TA_aws /opt/splunk/etc/apps/
# The app:
sudo cp -r /opt/splunk/etc/apps/TA-AFS_aws/local splunk_app_aws/local
sudo cp /opt/splunk/etc/apps/TA-AFS_aws/lookups/{account_name.csv,billing_report_s3key.csv,tags_config.csv,tags_description.csv,topology_billing_metrics.csv,topology_playback_checkpoint.csv} splunk_app_aws/lookups/
sudo rm -rf /opt/splunk/etc/apps/TA-AFS_aws
sudo mv splunk_app_aws /opt/splunk/etc/apps/
# The scientific computing
sudo mv Splunk_SA_Scientific_Python_linux_x86_64 /opt/splunk/etc/apps/
# Fix perms
sudo chown -R splunk:splunk /opt/splunk/etc/apps
# validate
sudo -u splunk /opt/splunk/bin/splunk btool check
sudo -u splunk /opt/splunk/bin/splunk restart
```

Browse to app. It'll warn you about acclerations. Then:
1) In settings->Data Models, for each acceleration in splunk_app_aws, click edit->edit acclerations, check the box, and click 'save'
2) In settings->advanced search, click 'Search Macros', select `aws-data-model-acceleration`, set `summariesonly=t`
3) In Settings > Searches, reports, and alerts, select the app `Splunk App for AWS`:
  * Run the Addon Synchronization and App Upgrader
  * For each of those, click edit->edit schedule, enable, and click 'okay'
4) Hide the add-on on the search head. Go to apps->manage apps, find the app, click properties, change visible to 'no'

On the HF:
5) In settings->Searches, Reports, and Alerts, find the search `Addon Metadata - Summarize AWS Inputs`, run it, and then enable it.
6) Also run an schedule `Addon Metadata - Migrate AWS Accounts`

### DIY:

In the end, you need the following set up:

* Accounts:
  * The local instance profile ('splunk-aws-instance-role')
  * The opposing partition ('mdr-test-c2-gov')
* IAM Roles (splunk_ta_aws_iam_roles.conf):
  * mdrtestc2
  * mdrtestc2gov
  * One for each other account
e.g.:
```
[mdrtestc2]
arn = arn:aws:iam::816914342178:role/instance/splunk_addon_for_aws
```
* Inputs:
  * For the two c2 accounts:
    * Cloudtrail - sqs-based-s3
    * Config - sqs-based-s3
    * ELB Logs - sqs-based-s3
  * For the c2 and the remaining accounts:
    * cloudwatch
    * Description
    * Config Rules

### Log - Test
cloudwatch:
x  cloudwatch-mdr-test-c2
x  cloudwatch-mdr-test-c2-gov
x  cloudwatch-mdr-test-malware-gov
x  cloudwatch-mdr-test-modelclient-gov
x  cloudwatch-legacy-mdr-test

Description:
x  description-mdr-test-malware-gov
x  description-mdr-test-modelclient-gov
x  description-legacy-mdr-test

Config Rules
x  configrules-mdr-test-c2
x  configrules-mdr-test-c2-gov
x  configrules-mdr-test-malware-gov
x  configrules-mdr-test-modelclient-gov
x  configrules-legacy-mdr-test

### Log - prod
cloudwatch:
x  cloudwatch-mdr-prod-c2
x  cloudwatch-mdr-prod-c2-gov
x  cloudwatch-legacy-mdr-prod
x  cloudwatch-mdr-commonservices
x  cloudwatch-mdr-commonservices-gov
x  cloudwatch-mdr-service-root
x  cloudwatch-mdr-cyber-range
x  cloudwatch-mdr-prod-bas
x  cloudwatch-mdr-prod-ca-c19
x  cloudwatch-mdr-prod-dc-c19
x  cloudwatch-mdr-prod-doed
x  cloudwatch-mdr-prod-frtib
x  cloudwatch-mdr-prod-la-c19
x  cloudwatch-mdr-prod-ma-c19
x  cloudwatch-mdr-prod-nga
x  cloudwatch-mdr-tiaas
x  cloudwatch-mdr-prod-root-ca
x  cloudwatch-mdr-prod-afs-gov
x  cloudwatch-mdr-prod-dgi-gov

Description:
x  description-mdr-prod-c2
x  description-mdr-prod-c2-gov
x  description-legacy-mdr-prod
x  description-mdr-commonservices
x  description-mdr-commonservices-gov
x  description-mdr-service-root
x  description-mdr-cyber-range
x  description-mdr-prod-bas
x  description-mdr-prod-ca-c19
x  description-mdr-prod-dc-c19
x  description-mdr-prod-doed
x  description-mdr-prod-frtib
x  description-mdr-prod-la-c19
x  description-mdr-prod-ma-c19
x  description-mdr-prod-nga
x  description-mdr-tiaas
x  description-mdr-prod-root-ca
x  description-mdr-prod-afs-gov
x  description-mdr-prod-dgi-gov

Config Rules:
x  configrules-mdr-prod-c2
x  configrules-mdr-prod-c2-gov
x  configrules-legacy-mdr-prod
x  configrules-mdr-commonservices
x  configrules-mdr-commonservices-gov
x  configrules-mdr-service-root
x  configrules-mdr-cyber-range
x  configrules-mdr-prod-bas
x  configrules-mdr-prod-ca-c19
x  configrules-mdr-prod-dc-c19
x  configrules-mdr-prod-doed
x  configrules-mdr-prod-frtib
x  configrules-mdr-prod-la-c19
x  configrules-mdr-prod-ma-c19
x  configrules-mdr-prod-nga
x  configrules-mdr-tiaas
x  configrules-mdr-prod-root-ca
x  configrules-mdr-prod-afs-gov
x  configrules-mdr-prod-dgi-gov