Splunk MA-c19 Offboarding Notes

Test Data

Customer asked for test data prior to roll off.

Steps

tar up a bucket
test the bucket on a standalone splunk install Standalone splunk centos7 m5a.xlarge vpc-038e00d047 subnet-035fc7b980 xdr-indexer-instance-role ma-c19-splunk-test-standalone msoc-build
install a modern version of the aws cli.
- /usr/local/bin/aws --version
  - aws-cli/1.21.4 is too old.
set it to the customer via aws s3 presign url
- use the xdr-ma-c19-prod-splunk-frozen bucket /usr/local/bin/aws s3 cp azure_bucket.tar.gz s3://xdr-ma-c19-prod-splunk-frozen
- Create presigned URL /usr/local/bin/aws s3 presign s3://xdr-ma-c19-prod-splunk-frozen/azure_bucket.tar.gz --expires-in 604800

PREP

Customer is OK with Splunk .tsidx files. No need to convert them to text files. Customer does need smaller file sizes. Index tar.gz files will need to be broken up.

Document which indexes will be moved
Adjust CM to set data retention ( if, needed )
Allow indexers to prune old data ( if, needed )
Ensure all data is replicated to all indexers
- Check the indexer clustering webpage and ensure 3 copies are green for each index
- This is configured via the site_replication_factor setting
Stop splunk on one indexer
Tar up index directories
Upload to s3
Download from s3 to temp indexers and extract to ensure data is readable
Repeat for all indexes
Break up tar.gz files into smaller chunks for customer to consume
Upload smaller chunks to S3 for final customer delivery

How big and old are the indexes?

| rest /services/data/indexes/  
| search title=* NOT title=_* NOT title=junk
| eval indexSizeGB = if(currentDBSizeMB >= 1 AND totalEventCount >=1, currentDBSizeMB/1024, null())     
| eval elapsedTime = now() - strptime(minTime,"%Y-%m-%dT%H:%M:%S%z")    
| eval dataAge = ceiling(elapsedTime / 86400)   
| stats sum(indexSizeGB) AS totalSize max(dataAge) as oldestDataAge by title    
| eval totalSize = if(isnotnull(totalSize), round(totalSize, 2), 0)     
| eval oldestDataAge = if(isNum(oldestDataAge), oldestDataAge, "N/A")       
| rename title as "Index" totalSize as "Total Size (GB)" oldestDataAge as "Oldest Data Age (days)"

What is the current license usage? Is data is coming in?

index=_internal host=ma-c19-splunk-cm* source="/opt/splunk/var/log/splunk/license_usage.log*" type=Usage 
| eval mb = round(b/1024/1024,2) 
| timechart span=1d useother=f sum(mb) AS gbytes BY h

Indexes to be Copied

/opt/splunkdata/hot/normal_primary

        Uncompressed    Compressed

(5 parts uploaded)app_aws 368GB 246G
(3 parts uploaded)salesforce 185GB 112GB (4 parts uploaded)junk 223GB 163G
(done)azure 47GB 36GB (done)app_o365 4.8GB 3.3GB (done)defaultdb 12MB 7.1MB (done)audit 30GB 21GB total: 635GB total Compressed: 581 GB Progress Bar: 12/12 total progress bar: 16/16

File sizes du -sh * | sort -h

Prep Indexes

Stop Splunk
Use tmux to avoid ssh session timeout and incase it takes a long time to create the .tar.gz.
Tar index folder
- tar cvzf myindex_index.tar.gz myindex/

Upload to S3

Ensure the aws cli is installed. If not, use a python venv to setup aws cli. This assumes python3 is already installed. Why not use awscliv2? Because it is not in pip! These instructions need to be updated to use awscliv2.

cd ~
python3 -m venv awscli
source awscli/bin/activate
cd awscli/bin
pip install awscli
chmod +x aws
aws --version

#bash oneliner
cd ~ && python3 -m venv awscli && source awscli/bin/activate && cd awscli/bin && pip install awscli && chmod +x aws && aws --version

The aws cli should be able to use the IAM instance role to connect to S3. No need to add AWS keys but you will need to configure the region (us-gov-east-1).

aws configure
aws s3 ls

Actually upload the file with this command:

~/awscli/bin/aws s3 cp /opt/splunkdata/hot/normal_primary/app_o365_index.tar.gz s3://xdr-ma-c19-prod-splunk-frozen/app_o365_index.tar.gz

Run this some where you have awscliv2 ( sorry super annoying i know! If you don't use awscliv2 you will not be able to download the file. ) aws --profile=mdr-prod-ma-c19-gov s3 presign s3://xdr-ma-c19-prod-splunk-frozen/app_o365_index.tar.gz --expires-in 86400

Split files before uploading

split -b 50G salesforce_index.tar.gz salesforce_index.tar.gz.part-

Splunk ma-c19 Offboarding Notes.md 4.4 KB Histórico Raw