# Splunk MA-c19 Offboarding Notes

## Test Data
Customer asked for test data prior to roll off. 

Steps
- tar up a bucket
- test the bucket on a standalone splunk install
    Standalone splunk
    centos7
    m5a.xlarge
    vpc-038e00d0478425598
    subnet-035fc7b980229db6a
    xdr-indexer-instance-role
    ma-c19-splunk-test-standalone
    msoc-build
- install a modern version of the aws cli. 
    - `/usr/local/bin/aws --version`
        - aws-cli/1.21.4 is too old. 
    
- set it to the customer via aws s3 presign url
    - use the xdr-ma-c19-prod-splunk-frozen bucket
        `/usr/local/bin/aws s3 cp azure_bucket.tar.gz s3://xdr-ma-c19-prod-splunk-frozen`
    - Create presigned URL
        `/usr/local/bin/aws s3 presign s3://xdr-ma-c19-prod-splunk-frozen/azure_bucket.tar.gz --expires-in 604800`


## PREP

Customer is OK with Splunk .tsidx files. No need to convert them to text files. 
Customer does need smaller file sizes. Index tar.gz files will need to be broken up. 

- Document which indexes will be moved
- Adjust CM to set data retention ( if, needed )
- Allow indexers to prune old data ( if, needed )
- Ensure all data is replicated to all indexers
    - Check the indexer clustering webpage and ensure 3 copies are green for each index
    - This is configured via the site_replication_factor setting
- Stop splunk on one indexer
- Tar up index directories
- Upload to s3
- Download from s3 to temp indexers and extract to ensure data is readable
- Repeat for all indexes
- Break up tar.gz files into smaller chunks for customer to consume
- Upload smaller chunks to S3 for final customer delivery

How big and old are the indexes?
```
| rest /services/data/indexes/  
| search title=* NOT title=_* NOT title=junk
| eval indexSizeGB = if(currentDBSizeMB >= 1 AND totalEventCount >=1, currentDBSizeMB/1024, null())     
| eval elapsedTime = now() - strptime(minTime,"%Y-%m-%dT%H:%M:%S%z")    
| eval dataAge = ceiling(elapsedTime / 86400)   
| stats sum(indexSizeGB) AS totalSize max(dataAge) as oldestDataAge by title    
| eval totalSize = if(isnotnull(totalSize), round(totalSize, 2), 0)     
| eval oldestDataAge = if(isNum(oldestDataAge), oldestDataAge, "N/A")       
| rename title as "Index" totalSize as "Total Size (GB)" oldestDataAge as "Oldest Data Age (days)"
```

What is the current license usage? Is data is coming in?
```
index=_internal host=ma-c19-splunk-cm* source="/opt/splunk/var/log/splunk/license_usage.log*" type=Usage 
| eval mb = round(b/1024/1024,2) 
| timechart span=1d useother=f sum(mb) AS gbytes BY h
```


## Indexes to be Copied

/opt/splunkdata/hot/normal_primary
            Uncompressed    Compressed
(5 parts uploaded)app_aws     368GB     246G   
(3 parts uploaded)salesforce  185GB           112GB
(4 parts uploaded)junk        223GB     163G  
(done)azure       47GB            36GB
(done)app_o365    4.8GB           3.3GB
(done)defaultdb   12MB            7.1MB
(done)audit       30GB            21GB
total: 635GB
total Compressed: 581 GB
Progress Bar: 12/12
total progress bar: 16/16

File sizes `du -sh * | sort -h`


## Prep Indexes

- Stop Splunk
- Use tmux to avoid ssh session timeout and incase it takes a long time to create the .tar.gz. 
- Tar index folder
    - tar cvzf myindex_index.tar.gz myindex/


## Upload to S3

Ensure the aws cli is installed. If not, use a python venv to setup aws cli. This assumes python3 is already installed. Why not use awscliv2? Because it is not in pip! These instructions need to be updated to use awscliv2. 

```
cd ~
python3 -m venv awscli
source awscli/bin/activate
cd awscli/bin
pip install awscli
chmod +x aws
aws --version

#bash oneliner
cd ~ && python3 -m venv awscli && source awscli/bin/activate && cd awscli/bin && pip install awscli && chmod +x aws && aws --version
```

The aws cli should be able to use the IAM instance role to connect to S3. No need to add AWS keys but you will need to configure the region (us-gov-east-1). 

```
aws configure
aws s3 ls
```

Actually upload the file with this command:
```
~/awscli/bin/aws s3 cp /opt/splunkdata/hot/normal_primary/app_o365_index.tar.gz s3://xdr-ma-c19-prod-splunk-frozen/app_o365_index.tar.gz
```
Run this some where you have awscliv2 ( sorry super annoying i know! If you don't use awscliv2 you will not be able to download the file. )
`aws --profile=mdr-prod-ma-c19-gov s3 presign s3://xdr-ma-c19-prod-splunk-frozen/app_o365_index.tar.gz --expires-in 86400`

## Split files before uploading 

split -b 50G salesforce_index.tar.gz salesforce_index.tar.gz.part-