# Splunk AFS Thaw Request Notes

This documents the process for searching the frozen data that is stored in S3.

Plan:
- charge time to CIRT Ops Support     SPROJ.061 S&ID CIRT Ops Support_A
- Don't use TF to manage the servers. 
- stand up muliple ( minimum 3? ) centos7 servers with large EBS disks
- stand up one SH for the indexers
- add EC2 instance policies with access to the S3 buckets
- use aws s3 cp to pull the data down
    - Use zztop.sh ( see below ) script to pull down data faster
    - Data going back to Jan 1, 2020 (1577836861)
- Thaw data ( no license needed since it is not ingestion )
    - no need to thaw the data if it has all the needed parts in the bucket ( tsidx files )
- Install AFS splunk apps on SH
    - Zip them up and upload them to S3
    - Download them from S3 on the new SH
- Hand over to SOC for searching

Assumptions:
- The AWS account will be deleted when we are done. 
- No cluster master needed
- Note: Data does not get replicated ( in the cluster ) from the thawed directory. So, if you thaw just a single copy of some bucket, instead of all the copies, only that single copy will reside in the cluster, in the thawed directory of the peer node where you placed it.

Build the VPC in the same region as the data is located in S3!

VPC info ( pick a CIDR that has not been used just in case you need to use transit gateway )
afs-data-thaw
10.199.0.0/22

indexers: c5.4xlarge on-demand $7/day m5d.xlarge or larger
1 TB EBS storage attached to instances
search head: m5a.xlarge 
centos7 AMI
key: msoc-build
instance role: default-instance-role
naming scheme: afs-splunk-sh
encrypt EBS with default key
give AWS IAM user both Administrator and IAMfullaccess to be able to launch instances!

## Indexes

```
Needed indexes:
app_mscas           967.3 GB        done
app_o365            1.3 TB          done    afs-splunk-idx-1
av                  86.8 MB         done
azure               149.4 GB        done
ids                 17.4 GB         done
network_firewall    8.2 TB          done
network             99.4 GB         done
threat_metrics      8.0 GB          done    afs-splunk-idx-1
websense            771.1 GB        done
wineventlog         5.0 TB          done
zscaler             87.3 GB         done
Total               16.6 TB
```

Use AWS console to calculate total size of folders in S3 bucket. This will help to see how many indexers are needed. 

## Permissions for S3 bucket

Steps in New AWS Account
- add role and attach to EC2 instance
- add policy to the role allowing KMS* and S3* permissions

Steps in Old AWS Account
- modify KMS role to allow for role in New Account
- modify S3 bucket policy to allow for role in New Account

New Account policy for role
```
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1610588870140",
      "Action": "kms:*",
      "Effect": "Allow",
      "Resource": "*"
    },
    {
      "Sid": "Stmt1610588903413",
      "Action": "s3:*",
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}
```

Changes for the Old Accout KMS key policy
```
        {
            "Sid": "Allow use of the key",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::948010823789:role/default-instance-role",
                    "arn:aws:iam::477548533976:role/mdr_powerusers",
                    "arn:aws:iam::477548533976:role/msoc-default-instance-role"
                ]
            },
            "Action": [
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:Encrypt",
                "kms:DescribeKey",
                "kms:Decrypt"
            ],
            "Resource": "*"
        }
```

Old Account S3 Bucket Policy
```
{
    "Version": "2012-10-17",
    "Id": "Policy1584399307003",
    "Statement": [
        {
            "Sid": "DownloadandUpload",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::948010823789:role/default-instance-role"
            },
            "Action": [
                "s3:GetObject",
                "s3:GetObjectVersion",
                "s3:RestoreObject"
            ],
            "Resource": "arn:aws:s3:::mdr-afs-prod-splunk-frozen/*"
        },
        {
            "Sid": "ListBucket",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::948010823789:role/default-instance-role"
            },
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::mdr-afs-prod-splunk-frozen"
        }
    ]
}
```

test permissions/access from second account
`aws s3 ls s3://mdr-afs-prod-splunk-frozen`

after the objects have been restored from Glacier try to download some objects
`aws s3 cp s3://mdr-afs-prod-splunk-frozen/junk/frozendb/db_1598282957_1598264767_511_50F6EC26-9620-4CAA-802C-857CD78386CE/ . --recursive --force-glacier-transfer`



## Restore Glacier objects

https://infinityworks.com/insights/restoring-2-million-objects-from-glacier/
https://github.com/s3tools/s3cmd

***s3cmd was the best option for restoring because it can pull the list of files for you and restore an entire directory recursivly***
s3cmd at this time, does not work with assumeRole STS credentials ( Don't run the command from laptop, just run the command from a new instance with the permissions )

Steps
- ensure your awscli can access the S3 buckets
- use s3cmd to restore objects

Plan
- Restore for 30 days
- Standard restore ( cheaper than Expedited )

test access for file restore from glacier
`aws s3 ls s3://mdr-afs-prod-splunk-frozen`

warning: Skipping file s3://mdr-afs-prod-splunk-frozen/junk/frozendb/db_1598282957_1598264767_511_50F6EC26-9620-4CAA-802C-857CD78386CE/rawdata/slicesv2.dat. Object is of storage class GLACIER. Unable to perform download operations on GLACIER objects. You must restore the object to be able to perform the operation. See aws s3 download help for additional parameter options to ignore or force these transfers.

Need to restore the data from glacier for a period of time.

Restore TIER 
Expedited   $$$     1-5 minutes less than 250MB
Standard    $$      3-5 hrs (Default if not given)
Bulk        $       5-12 hrs


See s3cmd command down below! It is better than s3api command. 

List objects
```
aws s3api list-objects-v2 --bucket mdr-afs-prod-splunk-frozen --prefix junk/frozendb/db_1598282957_1598264767_511_50F6EC26-9620-4CAA-802C-857CD78386CE --query "Contents[?StorageClass=='GLACIER']" --output text
```

Output the results to a file
```
aws s3api list-objects-v2 --bucket mdr-afs-prod-splunk-frozen --prefix junk/frozendb/db_1598282957_1598264767_511_50F6EC26-9620-4CAA-802C-857CD78386CE --query "Contents[?StorageClass=='GLACIER']" --output text | awk '{print $2}' > file.txt
```

Test access
`aws s3api restore-object –restore-request Days=2 --bucket mdr-afs-prod-splunk-frozen --key junk/frozendb/db_1598282957_1598264767_511_50F6EC26-9620-4CAA-802C-857CD78386CE/splunk-autogen-params.dat`

`aws s3api restore-object --restore-request Days=2 --bucket mdr-afs-prod-splunk-frozen --key junk/frozendb/db_1598282957_1598264767_511_50F6EC26-9620-4CAA-802C-857CD78386CE/rawdata/slicesv2.dat`

All in one command
```
aws s3api list-objects-v2 --bucket mdr-afs-prod-splunk-frozen --prefix junk/frozendb/db_1594951175_1594844814_53_94F7BD8A-9043-487B-8BD5-41AA54D7A925 --query "Contents[?StorageClass=='GLACIER']" --output text | awk '{print $2}' | xargs -L 1 aws s3api restore-object --restore-request '{ "Days" : 2, "GlacierJobParameters" : { "Tier":"Expedited" } }' --bucket mdr-afs-prod-splunk-frozen --key
```

This just means, we are kinda busy right now try again later. Not an error in your code, but your code needs to retry the request to ensure the request gets processed. This only happened with expedited requests.
```
An error occurred (GlacierExpeditedRetrievalNotAvailable) when calling the RestoreObject operation (reached max retries: 2): Glacier expedited retrievals are currently not available, please try again later
```

```
#!bin/sh
for x in $(cat file.txt); do
echo "Start restoring the file $x"
aws s3api restore-object restore-request Days=2  "$x"
echo "Completed restoring the file $x"
done
```

Expedite that mother 
```
#!bin/sh
TIER=Expedited
#TIER=Standard
#TIER=Bulk
DAYS=2
for x in $(cat file.txt); do
echo "Start restoring the file $x"
aws s3api restore-object --restore-request '{ "Days" : 2, "GlacierJobParameters" : { "Tier":"Expedited" } }' --bucket mdr-afs-prod-splunk-frozen --key $x
echo "Completed restoring the file $x"
done
```

With s3cmd! Be sure to use the exclude rb_* in the command, no need to restore the replicated buckets. 

Just a bucket
`./s3cmd restore --restore-priority=expedited --restore-days=2 --recursive s3://mdr-afs-prod-splunk-frozen/junk/frozendb/db_1566830011_1562776263_316_BBE343D5-D0D2-4120-A307-8B35B5E48D95/`

Whole index
`./s3cmd restore --restore-priority=standard --restore-days=30 --recursive s3://mdr-afs-prod-splunk-frozen/av/`

Exclude rb_* 

`time ./s3cmd restore --restore-priority=standard --restore-days=30 --recursive --exclude="frozendb/rb_*" s3://mdr-afs-prod-splunk-frozen/av/`

Distribute load to all servers via salt
`salt afs-splunk-idx-8 cmd.run '/root/s3cmd-2.1.0/s3cmd restore --restore-priority=standard --restore-days=30 --recursive s3://mdr-afs-prod-splunk-frozen/zscaler/' --async`

## Splunk

### Server Prep
- hostnamectl set-hostname afs-splunk-idx-2
- install salt-master ( SH only), salt-minion (which includes python3)
`rpm --import https://repo.saltstack.com/py3/redhat/7/x86_64/archive/3002.2/SALTSTACK-GPG-KEY.pub`
`vi /etc/yum.repos.d/saltstack.repo`
```
[saltstack-repo]
name=SaltStack repo for RHEL/CentOS 7 PY3
baseurl=https://repo.saltstack.com/py3/redhat/7/$basearch/archive/3002.2
enabled=1
gpgcheck=1
gpgkey=https://repo.saltstack.com/py3/redhat/7/$basearch/archive/3002.2/SALTSTACK-GPG-KEY.pub
```
yum clean expire-cache
yum install salt-minion -y 
sed -i 's/#master: salt/master: 10.199.0.83/' /etc/salt/minion
systemctl start salt-minion
systemctl enable salt-minion
- run salt states

stupid chrome hates the TLS certificate. Type this to bypass Chrome block for self-signed cert. 
thisisunsafe

### Installation

run salt states

### Data Pull with Salt

Use Duane's zztop.sh!!!!

`cmd.run '/root/s3cmd-2.1.0/s3cmd get --recursive s3://mdr-afs-prod-splunk-frozen/threat_metrics/frozendb/ /opt/splunk/var/lib/splunk/threat_metrics/thaweddb/'`


/root/s3cmd-2.1.0/s3cmd get --recursive s3://mdr-afs-prod-splunk-frozen/app_o365/frozendb/ /opt/splunk/var/lib/splunk/app_o365/thaweddb/

### Thaw it out!

No need to thaw it out! The data was not fully frozen and the data does not need to be rebuilt. 

## GetObject

Pull the file after it has been restored
`aws s3 cp s3://mdr-afs-prod-splunk-frozen/junk/frozendb/db_1598282957_1598264767_511_50F6EC26-9620-4CAA-802C-857CD78386CE/splunk-autogen-params.dat here.dat`

With S3cmd
`./s3cmd get --recursive s3://mdr-afs-prod-splunk-frozen/junk/frozendb/db_1598282957_1598264767_511_50F6EC26-9620-4CAA-802C-857CD78386CE/ /home/centos/test-dir-2/` 

BEST

STEPS
- get file of all S3 objects
- split file into 10 different files ( or number of indexers )
- run file with zztop to download the files


Make list of indexes
`aws s3 ls s3://mdr-afs-prod-splunk-frozen | awk '{ print $2 }' > foo1`

Make list of ALL buckets in each index
`for i in $(cat foo1| egrep -v ^_); do aws s3 ls s3://mdr-afs-prod-splunk-frozen/${i}frozendb/ | egrep "db" | awk -v dir=$i '{ printf("s3://mdr-afs-prod-splunk-frozen/%sfrozendb/%s\n",dir,$2)}' ; done > bucketlist`

break up list ( 10 indexers in this case )
`cat bucketlist | awk '{ x=NR%10 }{print >> "indexerlist"x}'`

create zztop.sh
$ cat zztop.sh
```
#!/bin/bash
DEST=$( echo $1 | awk -F/ '{ print "/opt/splunk/var/lib/splunk/"$4"/thaweddb/"$6 }' )
mkdir -p $DEST
/usr/local/aws-cli/v2/current/bin/aws s3 cp $1 $DEST --recursive --force-glacier-transfer --no-progress
```

Distribute files using salt
```
salt '*idx-2' cmd.run 'mkdir /root/s3cp'
salt '*idx-2' cp.get_file salt://s3cp/indexerlist0 /root/s3cp/indexerlist0
salt '*idx-2' cp.get_file salt://s3cp/zztop.sh /root/s3cp/zztop.sh
salt '*idx-2' cmd.run 'chmod +x /root/s3cp/zztop.sh'
```

idx-2   indexerlist2    needs restart
idx-3   indexerlist1    needs restart
idx-4   indexerlist2    running
idx-5   indexerlist3    running
idx-6   indexerlist4    running
idx-7   indexerlist5    running
idx-8   indexerlist6    running
idx-9   indexerlist7    running
idx-10   indexerlist8   running
idx-11   indexerlist9   running
...

distribute each list to an indexer and use zztop script with egrep and xargs to download all the buckets.

tmux ( process multiple lines at the same time with -P flag )
`egrep -h "*" indexerlist* | xargs -P 15 -n 1 ./zztop.sh`
`egrep -h "*" indexerlist* | head -1 | xargs -P 10 -n 1 ./zztop.sh`
