Merge branch 'master' of github.mdr.defpoint.com:mdr-engineering/infrastructure-notes

AWS Web Application Firewall Add-on Notes.md

@@ -0,0 +1,50 @@
+# AWS Web Application Firewall Add-on
+
+Download the [AWS Web Application Firewall Add-on](https://splunkbase.splunk.com/app/4714/).
+
+Install onto CM and SH
+
+## Installing onto cluster master:
+
+```
+scp aws-web-application-firewall-add-on_101.tgz dev-moose-splunk-cm:
+ssh dev-moose-splunk-cm
+tar xvzf aws-web-application-firewall-add-on_101.tgz
+sudo mv TA-aws_waf /opt/splunk/etc/master-apps/
+sudo mkdir /opt/splunk/etc/master-apps/TA-aws_waf/local
+sudo vim /opt/splunk/etc/master-apps/TA-aws_waf/local/inputs.conf
+```
+
+Generate a token via `uuidgen`
+
+```
+[http://aws_waf_logs]
+disabled = 0
+index = test
+indexes = test
+sourcetype = aws:waf
+useACK = 1
+token = <TOKEN_HERE>
+```
+
+```
+sudo chown -R splunk:splunk /opt/splunk/etc/master-apps/TA-aws_waf/
+sudo -u splunk /opt/splunk/bin/splunk btool check
+sudo -u splunk /opt/splunk/bin/splunk validate cluster-bundle
+sudo -u splunk /opt/splunk/bin/splunk show cluster-bundle-status
+sudo -u splunk /opt/splunk/bin/splunk 
+```
+
+## Installing onto SH
+
+```
+scp aws-web-application-firewall-add-on_101.tgz dev-moose-splunk-sh:
+ssh dev-moose-splunk-sh
+tar xvzf aws-web-application-firewall-add-on_101.tgz
+sudo mv TA-aws_waf /opt/splunk/etc/apps/
+sudo chown -R splunk:splunk /opt/splunk/etc/apps/TA-aws_waf
+sudo -u splunk /opt/splunk/bin/splunk restart
+```
+
+
+

Splunk AWS App and Addon Notes.md

@@ -0,0 +1,308 @@
+# Splunk App and Add-on for AWS Notes
+
+# Installation
+2020-08-27
+
+## Prepare the Accounts/roles:
+
+1. Update '005-account-standards-c2' to latest
+2. Update '006-account-standards' to latest
+3. Repeat for other accounts
+
+## Keys:
+
+To get access keys:
+```
+cd xdr-terraform-live/test/aws-us-gov/mdr-test-c2/005-account-standards-c2/
+terragrunt output
+```
+
+Returns something like:
+```
+access_keys = {
+  "current" = {
+    "aws_access_key_id" = "BLAHBLAHBLAH"
+    "aws_secret_access_key" = "SecretBlahBlahBlah"
+  }
+  "previous" = {
+    "aws_access_key_id" = "BLAHBLAHBLAH"
+    "aws_secret_access_key" = "SecretBlahBlahBlah"
+  }
+}
+```
+
+Use the 'current' values. Previous values are for key rotation.
+
+n.b. on 2020-09-01, actual keys were accidentally leaked here. They were all destroyed and regenerated.
+
+## AWS Addon
+
+The addon should be installed on teh search heads and heavy forwarders
+
+```
+scp ~/Downloads/Splunk/Apps/splunk-add-on-for-amazon-web-services_501.tgz dev-moose-splunk-hf:
+```
+
+* Note, may want to install on SH: *
+You can install this add-on on a search head cluster for all search-time functionality, but configure inputs on forwarders to avoid duplicate data collection.
+Before installing this add-on to a cluster, make the following changes to the add-on package:
+1. Remove the eventgen.conf files and all files in the samples folder
+2. Remove the inputs.conf file.
+
+```
+ssh dev-moose-splunk-hf
+sudo tar cvzf aws_app.backups.`date "+%Y%m%d"`tar.gz /opt/splunk/etc/apps/{TA-AFS_aws,TA-AFS_aws_addon}
+tar xvzf splunk-add-on-for-amazon-web-services_501.tgz
+cd /opt/splunk/etc/apps/
+sudo mv ~frederick_t_damstra/Splunk_TA_aws .
+sudo chown -R splunk:splunk Splunk_TA_aws
+sudo -u splunk /opt/splunk/bin/splunk restart
+```
+
+## Configure the add-on
+
+### Proxy
+1. Browse to the HF via mdr okta.
+2. Click on the add-on
+3. Click on configuration
+4. Configure the proxy to proxy.
+  * Enable: Yes
+  * Host: proxy.pvt.xdrtest.accenturefederalcyber.com
+  * Port: 80
+
+### C2 Account in other partition
+1. Go to configuration->Accounts
+2. Configure the govcloud account:
+  * mdr-test-c2-gov
+  * access key
+  * secret key
+  * Region
+
+### IAM Roles
+1. Go to configuration->IAM Role
+
+Skip:
+2. Click on 'Add'
+  * name: mdrtestc2gov # Doesn't accept dashes or spaces
+  * arn: arn:aws-us-gov:iam::738800754746:role/instance/splunk-addon-for-aws
+3. Click 'Add' again
+  * name: mdrtestc2
+  * arn: arn:aws:iam::816914342178:role/instance/splunk-addon-for-aws
+4. Add another account for testing purposes
+  * name: mdrtestmalwaregov
+  * arn: arn:aws-us-gov:iam::876942499057:role/instance/splunk_addon_for_aws
+
+Instead:
+Edit `splunk_ta_aws_iam_roles.conf`:
+```
+[mdrtestc2]
+#arn = arn:aws:iam::816914342178:role/instance/splunk-addon-for-aws
+arn = arn:aws:iam::816914342178:role/instance/splunk-addon-for-aws
+disabled = 0
+
+[mdrtestc2gov]
+#arn = arn:aws-us-gov:iam::738800754746:role/instance/splunk-addon-for-aws
+arn = arn:aws-us-gov:iam::738800754746:role/instance/splunk-addon-for-aws
+disabled = 0
+
+[mdrtestmalwaregov]
+#arn = arn:aws-us-gov:iam::876942499057:role/instance/splunk-addon-for-aws
+arn = arn:aws-us-gov:iam::876942499057:role/instance/splunk-addon-for-aws
+disabled = 0
+
+... 
+repeat for other accounts
+```
+
+### Cloudtrail Inputs
+1. Go to configuration->inputs
+2. Create new inputs -> cloud trail -> SQS based S3
+3. Set:
+  * Name: CloudTrail-GovCloud
+  * AWS Account: mdr-test-c2-gov
+  * AssumeRole: mdrtestc2gov
+  * AWS Region: us-gov-east-1
+  * SQS Queue Name: new objects for ...
+  * Index: app_aws
+  * ... and defaults * Probably want a separate index for production *
+
+Repeat for CloudTrail-Commercial
+Repeat for Config-{GovCloud,Commercial}
+
+## CM
+```
+ssh dev-moose-splunk-cm
+sudo su - splunk
+cd /opt/splunk/etc/master-apps/
+mkdir -p splunk_app_aws/local
+vim splunk_app_aws/local/indexes.conf
+```
+
+Paste:
+```
+[aws_topology_history]
+coldPath = volume:normal_primary/$_index_name/colddb
+homePath = volume:normal_primary/$_index_name/db
+thawedPath = $SPLUNK_DB/$_index_name/thaweddb
+# frozen time is 7 days
+frozenTimePeriodInSecs = 604800
+maxHotIdleSecs = 3600
+repFactor = auto
+
+[aws_topology_daily_snapshot]
+coldPath = volume:normal_primary/$_index_name/colddb
+homePath = volume:normal_primary/$_index_name/db
+thawedPath = $SPLUNK_DB/$_index_name/thaweddb
+#frozen time is about 6 months
+frozenTimePeriodInSecs = 15552000
+maxHotIdleSecs = 3600
+repFactor = auto
+
+[aws_topology_monthly_snapshot]
+coldPath = volume:normal_primary/$_index_name/colddb
+homePath = volume:normal_primary/$_index_name/db
+thawedPath = $SPLUNK_DB/$_index_name/thaweddb
+# frozen time is 365 days
+frozenTimePeriodInSecs = 31536000
+maxHotIdleSecs = 86400
+repFactor = auto
+
+[aws_topology_playback]
+coldPath = volume:normal_primary/$_index_name/colddb
+homePath = volume:normal_primary/$_index_name/db
+thawedPath = $SPLUNK_DB/$_index_name/thaweddb
+#frozen time is about 6 months
+frozenTimePeriodInSecs = 15552000
+maxHotIdleSecs = 3600
+repFactor = auto
+
+[aws_vpc_flow_logs]
+coldPath = volume:normal_primary/$_index_name/colddb
+homePath = volume:normal_primary/$_index_name/db
+thawedPath = $SPLUNK_DB/$_index_name/thaweddb
+# frozen time is 7 days
+frozenTimePeriodInSecs = 604800
+maxHotIdleSecs = 3600
+repFactor = auto
+
+[aws_anomaly_detection]
+coldPath = volume:normal_primary/$_index_name/colddb
+homePath = volume:normal_primary/$_index_name/db
+thawedPath = $SPLUNK_DB/$_index_name/thaweddb
+repFactor = auto
+```
+
+validate and push cluster bundle
+
+## SH
+
+```
+scp ~/Downloads/Splunk/Apps/{python-for-scientific-computing-for-linux-64-bit_202.tgz,splunk-app-for-aws_601.tgz,splunk-add-on-for-amazon-web-services_501.tgz} dev-moose-splunk-sh:
+ssh dev-moose-splunk-sh
+sudo tar cvzf aws_app.backups.`date "+%Y%m%d"`tar.gz /opt/splunk/etc/apps/{TA-AFS_aws,TA-AFS_aws_addon}
+for i in *.tgz; do tar xvzf $i; done
+# The add-on
+sudo cp -r /opt/splunk/etc/apps/TA-AFS_aws_addon/local Splunk_TA_aws/local
+# check this again, find which two of these three are actually 
+sudo cp /opt/splunk/etc/apps/TA-AFS_aws_addon/lookups/{aws_console_login_signature.csv,aws_config_action_lookup.csv,cloudtrail_change_type.csv} Splunk_TA_aws/lookups/
+sudo rm -rf /opt/splunk/etc/apps/TA-AFS_aws_addon
+sudo mv Splunk_TA_aws /opt/splunk/etc/apps/
+# The app:
+sudo cp -r /opt/splunk/etc/apps/TA-AFS_aws/local splunk_app_aws/local
+sudo cp /opt/splunk/etc/apps/TA-AFS_aws/lookups/{account_name.csv,billing_report_s3key.csv,tags_config.csv,tags_description.csv,topology_billing_metrics.csv,topology_playback_checkpoint.csv} splunk_app_aws/lookups/
+sudo rm -rf /opt/splunk/etc/apps/TA-AFS_aws
+sudo mv splunk_app_aws /opt/splunk/etc/apps/
+# The scientific computing
+sudo mv Splunk_SA_Scientific_Python_linux_x86_64 /opt/splunk/etc/apps/
+# Fix perms
+sudo chown -R splunk:splunk /opt/splunk/etc/apps
+# validate
+sudo -u splunk /opt/splunk/bin/splunk btool check
+sudo -u splunk /opt/splunk/bin/splunk restart
+```
+
+Browse to app. It'll warn you about acclerations. Then:
+1) In settings->Data Models, for each acceleration in splunk_app_aws, click edit->edit acclerations, check the box, and click 'save'
+2) In settings->advanced search, click 'Search Macros', select `aws-data-model-acceleration`, set `summariesonly=t`
+3) In Settings > Searches, reports, and alerts, select the app `Splunk App for AWS`:
+  * Run the Addon Synchronization and App Upgrader
+  * For each of those, click edit->edit schedule, enable, and click 'okay'
+4) Hide the add-on on the search head. Go to apps->manage apps, find the app, click properties, change visible to 'no'
+
+On the HF:
+5) In settings->Searches, Reports, and Alerts, find the search `Addon Metadata - Summarize AWS Inputs`, run it, and then enable it.
+6) Also run an schedule `Addon Metadata - Migrate AWS Accounts`
+
+### DIY:
+
+In the end, you need the following set up:
+
+* Accounts:
+  * The local instance profile ('splunk-aws-instance-role')
+  * The opposing partition ('mdr-test-c2-gov')
+* IAM Roles (splunk_ta_aws_iam_roles.conf):
+  * mdrtestc2
+  * mdrtestc2gov
+  * One for each other account
+e.g.:
+```
+[mdrtestc2]
+arn = arn:aws:iam::816914342178:role/instance/splunk_addon_for_aws
+```
+* Inputs:
+  * For the two c2 accounts:
+    * Cloudtrail - sqs-based-s3
+    * Config - sqs-based-s3
+    * ELB Logs - sqs-based-s3
+  * For the c2 and the remaining accounts:
+    * cloudwatch
+    * Description
+    * Config Rules
+
+### Log - Test
+cloudwatch:
+x  cloudwatch-mdr-test-c2
+x  cloudwatch-mdr-test-c2-gov
+x  cloudwatch-mdr-test-malware-gov
+x  cloudwatch-mdr-test-modelclient-gov
+x  cloudwatch-legacy-mdr-test
+
+Description:
+x  description-mdr-test-malware-gov
+x  description-mdr-test-modelclient-gov
+x  description-legacy-mdr-test
+
+Config Rules
+x  configrules-mdr-test-c2
+x  configrules-mdr-test-c2-gov
+x  configrules-mdr-test-malware-gov
+x  configrules-mdr-test-modelclient-gov
+x  configrules-legacy-mdr-test
+
+### Log - prod
+cloudwatch:
+x  cloudwatch-mdr-prod-c2
+x  cloudwatch-mdr-prod-c2-gov
+x  cloudwatch-legacy-mdr-prod
+x  cloudwatch-mdr-commonservices
+x  cloudwatch-mdr-commonservices-gov
+x  cloudwatch-mdr-service-root
+x  cloudwatch-mdr-cyber-range
+
+Description:
+x  description-mdr-prod-c2
+x  description-mdr-prod-c2-gov
+x  description-legacy-mdr-prod
+x  description-mdr-commonservices
+x  description-mdr-commonservices-gov
+x  description-mdr-service-root
+x  description-mdr-cyber-range
+
+Config Rules:
+x  configrules-mdr-prod-c2
+x  configrules-mdr-prod-c2-gov
+x  configrules-legacy-mdr-prod
+x  configrules-mdr-commonservices
+x  configrules-mdr-commonservices-gov
+x  configrules-mdr-service-root
+x  configrules-mdr-cyber-range