|
|
@@ -0,0 +1,350 @@
|
|
|
+# Splunk AFS Thaw Request Notes
|
|
|
+
|
|
|
+This documents the process for searching the frozen data that is stored in S3.
|
|
|
+
|
|
|
+Plan:
|
|
|
+- charge time to CIRT Ops Support SPROJ.061
|
|
|
+- Don't use TF to manage the servers.
|
|
|
+- stand up muliple ( minimum 3? ) centos7 servers with large EBS disks
|
|
|
+- stand up one SH for the indexers
|
|
|
+- add EC2 instance policies with access to the S3 buckets
|
|
|
+- use aws s3 cp to pull the data down
|
|
|
+ - Use zztop.sh ( see below ) script to pull down data faster
|
|
|
+ - Data going back to Jan 1, 2020 (1577836861)
|
|
|
+- Thaw data ( no license needed since it is not ingestion )
|
|
|
+ - no need to thaw the data if it has all the needed parts in the bucket ( tsidx files )
|
|
|
+- Install AFS splunk apps on SH
|
|
|
+ - Zip them up and upload them to S3
|
|
|
+ - Download them from S3 on the new SH
|
|
|
+- Hand over to SOC for searching
|
|
|
+
|
|
|
+Assumptions:
|
|
|
+- The AWS account will be deleted when we are done.
|
|
|
+- No cluster master needed
|
|
|
+- Note: Data does not get replicated ( in the cluster ) from the thawed directory. So, if you thaw just a single copy of some bucket, instead of all the copies, only that single copy will reside in the cluster, in the thawed directory of the peer node where you placed it.
|
|
|
+
|
|
|
+Build the VPC in the same region as the data is located in S3!
|
|
|
+
|
|
|
+VPC info ( pick a CIDR that has not been used just in case you need to use transit gateway )
|
|
|
+afs-data-thaw
|
|
|
+10.199.0.0/22
|
|
|
+
|
|
|
+indexers: c5.4xlarge on-demand $7/day m5d.xlarge or larger
|
|
|
+1 TB EBS storage attached to instances
|
|
|
+search head: m5a.xlarge
|
|
|
+centos7 AMI
|
|
|
+key: msoc-build
|
|
|
+instance role: default-instance-role
|
|
|
+naming scheme: afs-splunk-sh
|
|
|
+encrypt EBS with default key
|
|
|
+give AWS IAM user both Administrator and IAMfullaccess to be able to launch instances!
|
|
|
+
|
|
|
+## Indexes
|
|
|
+
|
|
|
+```
|
|
|
+Needed indexes:
|
|
|
+app_mscas 967.3 GB done
|
|
|
+app_o365 1.3 TB done afs-splunk-idx-1
|
|
|
+av 86.8 MB done
|
|
|
+azure 149.4 GB done
|
|
|
+ids 17.4 GB done
|
|
|
+network_firewall 8.2 TB done
|
|
|
+network 99.4 GB done
|
|
|
+threat_metrics 8.0 GB done afs-splunk-idx-1
|
|
|
+websense 771.1 GB done
|
|
|
+wineventlog 5.0 TB done
|
|
|
+zscaler 87.3 GB done
|
|
|
+Total 16.6 TB
|
|
|
+```
|
|
|
+
|
|
|
+Use AWS console to calculate total size of folders in S3 bucket. This will help to see how many indexers are needed.
|
|
|
+
|
|
|
+## Permissions for S3 bucket
|
|
|
+
|
|
|
+Steps in New AWS Account
|
|
|
+- add role and attach to EC2 instance
|
|
|
+- add policy to the role allowing KMS* and S3* permissions
|
|
|
+
|
|
|
+Steps in Old AWS Account
|
|
|
+- modify KMS role to allow for role in New Account
|
|
|
+- modify S3 bucket policy to allow for role in New Account
|
|
|
+
|
|
|
+New Account policy for role
|
|
|
+```
|
|
|
+{
|
|
|
+ "Version": "2012-10-17",
|
|
|
+ "Statement": [
|
|
|
+ {
|
|
|
+ "Sid": "Stmt1610588870140",
|
|
|
+ "Action": "kms:*",
|
|
|
+ "Effect": "Allow",
|
|
|
+ "Resource": "*"
|
|
|
+ },
|
|
|
+ {
|
|
|
+ "Sid": "Stmt1610588903413",
|
|
|
+ "Action": "s3:*",
|
|
|
+ "Effect": "Allow",
|
|
|
+ "Resource": "*"
|
|
|
+ }
|
|
|
+ ]
|
|
|
+}
|
|
|
+```
|
|
|
+
|
|
|
+Changes for the Old Accout KMS key policy
|
|
|
+```
|
|
|
+ {
|
|
|
+ "Sid": "Allow use of the key",
|
|
|
+ "Effect": "Allow",
|
|
|
+ "Principal": {
|
|
|
+ "AWS": [
|
|
|
+ "arn:aws:iam::948010823789:role/default-instance-role",
|
|
|
+ "arn:aws:iam::477548533976:role/mdr_powerusers",
|
|
|
+ "arn:aws:iam::477548533976:role/msoc-default-instance-role"
|
|
|
+ ]
|
|
|
+ },
|
|
|
+ "Action": [
|
|
|
+ "kms:ReEncrypt*",
|
|
|
+ "kms:GenerateDataKey*",
|
|
|
+ "kms:Encrypt",
|
|
|
+ "kms:DescribeKey",
|
|
|
+ "kms:Decrypt"
|
|
|
+ ],
|
|
|
+ "Resource": "*"
|
|
|
+ }
|
|
|
+```
|
|
|
+
|
|
|
+Old Account S3 Bucket Policy
|
|
|
+```
|
|
|
+{
|
|
|
+ "Version": "2012-10-17",
|
|
|
+ "Id": "Policy1584399307003",
|
|
|
+ "Statement": [
|
|
|
+ {
|
|
|
+ "Sid": "DownloadandUpload",
|
|
|
+ "Effect": "Allow",
|
|
|
+ "Principal": {
|
|
|
+ "AWS": "arn:aws:iam::948010823789:role/default-instance-role"
|
|
|
+ },
|
|
|
+ "Action": [
|
|
|
+ "s3:GetObject",
|
|
|
+ "s3:GetObjectVersion",
|
|
|
+ "s3:RestoreObject"
|
|
|
+ ],
|
|
|
+ "Resource": "arn:aws:s3:::mdr-afs-prod-splunk-frozen/*"
|
|
|
+ },
|
|
|
+ {
|
|
|
+ "Sid": "ListBucket",
|
|
|
+ "Effect": "Allow",
|
|
|
+ "Principal": {
|
|
|
+ "AWS": "arn:aws:iam::948010823789:role/default-instance-role"
|
|
|
+ },
|
|
|
+ "Action": "s3:ListBucket",
|
|
|
+ "Resource": "arn:aws:s3:::mdr-afs-prod-splunk-frozen"
|
|
|
+ }
|
|
|
+ ]
|
|
|
+}
|
|
|
+```
|
|
|
+
|
|
|
+test permissions/access from second account
|
|
|
+`aws s3 ls s3://mdr-afs-prod-splunk-frozen`
|
|
|
+
|
|
|
+after the objects have been restored from Glacier try to download some objects
|
|
|
+`aws s3 cp s3://mdr-afs-prod-splunk-frozen/junk/frozendb/db_1598282957_1598264767_511_50F6EC26-9620-4CAA-802C-857CD78386CE/ . --recursive --force-glacier-transfer`
|
|
|
+
|
|
|
+
|
|
|
+
|
|
|
+## Restore Glacier objects
|
|
|
+
|
|
|
+https://infinityworks.com/insights/restoring-2-million-objects-from-glacier/
|
|
|
+https://github.com/s3tools/s3cmd
|
|
|
+
|
|
|
+***s3cmd was the best option for restoring because it can pull the list of files for you and restore an entire directory recursivly***
|
|
|
+s3cmd at this time, does not work with assumeRole STS credentials ( Don't run the command from laptop, just run the command from a new instance with the permissions )
|
|
|
+
|
|
|
+Steps
|
|
|
+- ensure your awscli can access the S3 buckets
|
|
|
+- use s3cmd to restore objects
|
|
|
+
|
|
|
+Plan
|
|
|
+- Restore for 30 days
|
|
|
+- Standard restore ( cheaper than Expedited )
|
|
|
+
|
|
|
+test access for file restore from glacier
|
|
|
+`aws s3 ls s3://mdr-afs-prod-splunk-frozen`
|
|
|
+
|
|
|
+warning: Skipping file s3://mdr-afs-prod-splunk-frozen/junk/frozendb/db_1598282957_1598264767_511_50F6EC26-9620-4CAA-802C-857CD78386CE/rawdata/slicesv2.dat. Object is of storage class GLACIER. Unable to perform download operations on GLACIER objects. You must restore the object to be able to perform the operation. See aws s3 download help for additional parameter options to ignore or force these transfers.
|
|
|
+
|
|
|
+Need to restore the data from glacier for a period of time.
|
|
|
+
|
|
|
+Restore TIER
|
|
|
+Expedited $$$ 1-5 minutes less than 250MB
|
|
|
+Standard $$ 3-5 hrs (Default if not given)
|
|
|
+Bulk $ 5-12 hrs
|
|
|
+
|
|
|
+
|
|
|
+See s3cmd command down below! It is better than s3api command.
|
|
|
+
|
|
|
+List objects
|
|
|
+```
|
|
|
+aws s3api list-objects-v2 --bucket mdr-afs-prod-splunk-frozen --prefix junk/frozendb/db_1598282957_1598264767_511_50F6EC26-9620-4CAA-802C-857CD78386CE --query "Contents[?StorageClass=='GLACIER']" --output text
|
|
|
+```
|
|
|
+
|
|
|
+Output the results to a file
|
|
|
+```
|
|
|
+aws s3api list-objects-v2 --bucket mdr-afs-prod-splunk-frozen --prefix junk/frozendb/db_1598282957_1598264767_511_50F6EC26-9620-4CAA-802C-857CD78386CE --query "Contents[?StorageClass=='GLACIER']" --output text | awk '{print $2}' > file.txt
|
|
|
+```
|
|
|
+
|
|
|
+Test access
|
|
|
+`aws s3api restore-object –restore-request Days=2 --bucket mdr-afs-prod-splunk-frozen --key junk/frozendb/db_1598282957_1598264767_511_50F6EC26-9620-4CAA-802C-857CD78386CE/splunk-autogen-params.dat`
|
|
|
+
|
|
|
+`aws s3api restore-object --restore-request Days=2 --bucket mdr-afs-prod-splunk-frozen --key junk/frozendb/db_1598282957_1598264767_511_50F6EC26-9620-4CAA-802C-857CD78386CE/rawdata/slicesv2.dat`
|
|
|
+
|
|
|
+All in one command
|
|
|
+```
|
|
|
+aws s3api list-objects-v2 --bucket mdr-afs-prod-splunk-frozen --prefix junk/frozendb/db_1594951175_1594844814_53_94F7BD8A-9043-487B-8BD5-41AA54D7A925 --query "Contents[?StorageClass=='GLACIER']" --output text | awk '{print $2}' | xargs -L 1 aws s3api restore-object --restore-request '{ "Days" : 2, "GlacierJobParameters" : { "Tier":"Expedited" } }' --bucket mdr-afs-prod-splunk-frozen --key
|
|
|
+```
|
|
|
+
|
|
|
+This just means, we are kinda busy right now try again later. Not an error in your code, but your code needs to retry the request to ensure the request gets processed. This only happened with expedited requests.
|
|
|
+```
|
|
|
+An error occurred (GlacierExpeditedRetrievalNotAvailable) when calling the RestoreObject operation (reached max retries: 2): Glacier expedited retrievals are currently not available, please try again later
|
|
|
+```
|
|
|
+
|
|
|
+```
|
|
|
+#!bin/sh
|
|
|
+for x in $(cat file.txt); do
|
|
|
+echo "Start restoring the file $x"
|
|
|
+aws s3api restore-object restore-request Days=2 "$x"
|
|
|
+echo "Completed restoring the file $x"
|
|
|
+done
|
|
|
+```
|
|
|
+
|
|
|
+Expedite that mother
|
|
|
+```
|
|
|
+#!bin/sh
|
|
|
+TIER=Expedited
|
|
|
+#TIER=Standard
|
|
|
+#TIER=Bulk
|
|
|
+DAYS=2
|
|
|
+for x in $(cat file.txt); do
|
|
|
+echo "Start restoring the file $x"
|
|
|
+aws s3api restore-object --restore-request '{ "Days" : 2, "GlacierJobParameters" : { "Tier":"Expedited" } }' --bucket mdr-afs-prod-splunk-frozen --key $x
|
|
|
+echo "Completed restoring the file $x"
|
|
|
+done
|
|
|
+```
|
|
|
+
|
|
|
+With s3cmd! Be sure to use the exclude rb_* in the commmand, no need to restore the replicated buckets.
|
|
|
+
|
|
|
+Just a bucket
|
|
|
+`./s3cmd restore --restore-priority=expedited --restore-days=2 --recursive s3://mdr-afs-prod-splunk-frozen/junk/frozendb/db_1566830011_1562776263_316_BBE343D5-D0D2-4120-A307-8B35B5E48D95/`
|
|
|
+
|
|
|
+Whole index
|
|
|
+`./s3cmd restore --restore-priority=standard --restore-days=30 --recursive s3://mdr-afs-prod-splunk-frozen/av/`
|
|
|
+
|
|
|
+Exclude rb_*
|
|
|
+
|
|
|
+`time ./s3cmd restore --restore-priority=standard --restore-days=30 --recursive --exclude="frozendb/rb_*" s3://mdr-afs-prod-splunk-frozen/av/`
|
|
|
+
|
|
|
+Distribute load to all servers via salt
|
|
|
+`salt afs-splunk-idx-8 cmd.run '/root/s3cmd-2.1.0/s3cmd restore --restore-priority=standard --restore-days=30 --recursive s3://mdr-afs-prod-splunk-frozen/zscaler/' --async`
|
|
|
+
|
|
|
+## Splunk
|
|
|
+
|
|
|
+### Server Prep
|
|
|
+- hostnamectl set-hostname afs-splunk-idx-2
|
|
|
+- install salt-master ( SH only), salt-minion (which includes python3)
|
|
|
+`rpm --import https://repo.saltstack.com/py3/redhat/7/x86_64/archive/3002.2/SALTSTACK-GPG-KEY.pub`
|
|
|
+`vi /etc/yum.repos.d/saltstack.repo`
|
|
|
+```
|
|
|
+[saltstack-repo]
|
|
|
+name=SaltStack repo for RHEL/CentOS 7 PY3
|
|
|
+baseurl=https://repo.saltstack.com/py3/redhat/7/$basearch/archive/3002.2
|
|
|
+enabled=1
|
|
|
+gpgcheck=1
|
|
|
+gpgkey=https://repo.saltstack.com/py3/redhat/7/$basearch/archive/3002.2/SALTSTACK-GPG-KEY.pub
|
|
|
+```
|
|
|
+yum clean expire-cache
|
|
|
+yum install salt-minion -y
|
|
|
+sed -i 's/#master: salt/master: 10.199.0.83/' /etc/salt/minion
|
|
|
+systemctl start salt-minion
|
|
|
+systemctl enable salt-minion
|
|
|
+- run salt states
|
|
|
+
|
|
|
+stupid chrome hates the TLS certificate. Type this to bypass Chrome block for self-signed cert.
|
|
|
+thisisunsafe
|
|
|
+
|
|
|
+### Installation
|
|
|
+
|
|
|
+run salt states
|
|
|
+
|
|
|
+### Data Pull with Salt
|
|
|
+
|
|
|
+Use Duane's zztop.sh!!!!
|
|
|
+
|
|
|
+`cmd.run '/root/s3cmd-2.1.0/s3cmd get --recursive s3://mdr-afs-prod-splunk-frozen/threat_metrics/frozendb/ /opt/splunk/var/lib/splunk/threat_metrics/thaweddb/'`
|
|
|
+
|
|
|
+
|
|
|
+/root/s3cmd-2.1.0/s3cmd get --recursive s3://mdr-afs-prod-splunk-frozen/app_o365/frozendb/ /opt/splunk/var/lib/splunk/app_o365/thaweddb/
|
|
|
+
|
|
|
+### Thaw it out!
|
|
|
+
|
|
|
+No need to thaw it out! The data was not fully frozen and the data does not need to be rebuilt.
|
|
|
+
|
|
|
+## GetObject
|
|
|
+
|
|
|
+Pull the file after it has been restored
|
|
|
+`aws s3 cp s3://mdr-afs-prod-splunk-frozen/junk/frozendb/db_1598282957_1598264767_511_50F6EC26-9620-4CAA-802C-857CD78386CE/splunk-autogen-params.dat here.dat`
|
|
|
+
|
|
|
+With S3cmd
|
|
|
+`./s3cmd get --recursive s3://mdr-afs-prod-splunk-frozen/junk/frozendb/db_1598282957_1598264767_511_50F6EC26-9620-4CAA-802C-857CD78386CE/ /home/centos/test-dir-2/`
|
|
|
+
|
|
|
+BEST
|
|
|
+
|
|
|
+STEPS
|
|
|
+- get file of all S3 objects
|
|
|
+- split file into 10 different files ( or number of indexers )
|
|
|
+- run file with zztop to download the files
|
|
|
+
|
|
|
+
|
|
|
+Make list of indexes
|
|
|
+`aws s3 ls s3://mdr-afs-prod-splunk-frozen | awk '{ print $2 }' > foo1`
|
|
|
+
|
|
|
+Make list of ALL buckets in each index
|
|
|
+`for i in $(cat foo1| egrep -v ^_); do aws s3 ls s3://mdr-afs-prod-splunk-frozen/${i}frozendb/ | egrep "db" | awk -v dir=$i '{ printf("s3://mdr-afs-prod-splunk-frozen/%sfrozendb/%s\n",dir,$2)}' ; done > bucketlist`
|
|
|
+
|
|
|
+break up list ( 10 indexers in this case )
|
|
|
+`cat bucketlist | awk '{ x=NR%10 }{print >> "indexerlist"x}'`
|
|
|
+
|
|
|
+create zztop.sh
|
|
|
+$ cat zztop.sh
|
|
|
+```
|
|
|
+#!/bin/bash
|
|
|
+DEST=$( echo $1 | awk -F/ '{ print "/opt/splunk/var/lib/splunk/"$4"/thaweddb/"$6 }' )
|
|
|
+mkdir -p $DEST
|
|
|
+/usr/local/aws-cli/v2/current/bin/aws s3 cp $1 $DEST --recursive --force-glacier-transfer --no-progress
|
|
|
+```
|
|
|
+
|
|
|
+Distribute files using salt
|
|
|
+```
|
|
|
+salt '*idx-2' cmd.run 'mkdir /root/s3cp'
|
|
|
+salt '*idx-2' cp.get_file salt://s3cp/indexerlist0 /root/s3cp/indexerlist0
|
|
|
+salt '*idx-2' cp.get_file salt://s3cp/zztop.sh /root/s3cp/zztop.sh
|
|
|
+salt '*idx-2' cmd.run 'chmod +x /root/s3cp/zztop.sh'
|
|
|
+```
|
|
|
+
|
|
|
+idx-2 indexerlist2 needs restart
|
|
|
+idx-3 indexerlist1 needs restart
|
|
|
+idx-4 indexerlist2 running
|
|
|
+idx-5 indexerlist3 running
|
|
|
+idx-6 indexerlist4 running
|
|
|
+idx-7 indexerlist5 running
|
|
|
+idx-8 indexerlist6 running
|
|
|
+idx-9 indexerlist7 running
|
|
|
+idx-10 indexerlist8 running
|
|
|
+idx-11 indexerlist9 running
|
|
|
+...
|
|
|
+
|
|
|
+distribute each list to an indexer and use zztop script with egrep and xargs to download all the buckets.
|
|
|
+
|
|
|
+tmux ( process multiple lines at the same time with -P flag )
|
|
|
+`egrep -h "*" indexerlist* | xargs -P 15 -n 1 ./zztop.sh`
|
|
|
+`egrep -h "*" indexerlist* | head -1 | xargs -P 10 -n 1 ./zztop.sh`
|
Fred Damstra