# Splunk SAF Data Offboarding.txt Currently a 3 node multi-site cluster. Possible solution, set search and rep factor to 3 and 3 then pull the index files off one of the indexers to a new instance. On the new instance, setup multi-site cluster with one site and see if you can read the indexed files. https://docs.splunk.com/Documentation/Splunk/8.0.2/Indexer/Decommissionasite https://docs.splunk.com/Documentation/Splunk/7.0.3/Indexer/Multisitedeploymentoverview 1 - cluster master 1 - indexer with search /opt/splunkdata/hot/normal_primary/ indexes: app_mscas app_o365 dns forescout network security te File paths #Where in salt are the search / rep factors salt/pillar/saf_variables.sls splunk: cluster_name: saf license_master: saf-splunk-cm idxc: label: "saf_index_cluster" pass4SymmKey: "$1$ekY601SK1y5wfbd2ogCNRIhn+gPeQ+UYKzY3MMAnPmmz" rep_factor: 2 search_factor: 2 #where in splunk are the configs written to? /opt/splunk/etc/system/local/server.conf [clustering] mode = master multisite = true replication_factor = 2 search_factor = 2 max_peer_sum_rep_load = 15 max_peer_rep_load = 15 max_peer_build_load = 6 summary_replication = true site_search_factor = origin:1, total:2 site_replication_factor = origin:1,site1:1,site2:1,site3:1,total:3 available_sites = site1,site2,site3 cluster_label = afs_index_cluster Steps 1. change /opt/splunk/etc/system/local/server.conf site_search_factor to origin:1,site1:1,site2:1,site3:1,total:3 This will ensure we have a searchable copy of all the buckets on all the sites. Should I change site_replication_factor to origin:1, total:1? this would reduce the size of the index. 2. restart CM ( this will apply the site_search_factor ) 3. send data to junk index (oneshot) 3.1 /opt/splunk/bin/splunk add oneshot /opt/splunk/var/log/splunk/splunkd.log -sourcetype splunkd -index junk 4. stop one indexer and copy index to new cluster. 5. on new cluster, setup CM and 1 indexer in multisite cluster. the clustermaster will be a search head in the same site 6. setup new cluster to have site_mappings = default:site1 7. attempt to search on new cluster made the new junk index on test saf number of events: 64675 latest = 02/21/20 9:32:01 PM UTC earlest = 02/19/20 2:32:57 PM UTC Before copying the buckets, ensure they are ALL WARM buckets, HOT buckets maybe be deleted on startup. #check on the buckets | dbinspect index=junk uploaded brad_LAN key pair to AWS for new instances. vpc-041edac5e3ca49e4d subnet-0ca93c00ac57c9ebf sg-0d78af22d0afd0334 saf-offboarding-cm-deleteme saf-offboarding-indexer-1-deleteme CentOS 7 (x86_64) - with Updates HVM t2.medium (2 CPU 4 GB RAM) 100 GB drive msoc-default-instance-role saf-offboarding-ssh Security group <- delete this not needed just SSH from Bastion host splunk version 7.0.3 setup proxy for yum and wget vi /etc/yum.conf proxy=http://proxy.msoc.defpoint.local:80 yum install vim wget vim /etc/wgetrc http_proxy = http://proxy.msoc.defpoint.local:80 https_proxy = http://proxy.msoc.defpoint.local:80 Download Splunk wget -O splunk-7.0.3-fa31da744b51-linux-2.6-x86_64.rpm 'https://www.splunk.com/page/download_track?file=7.0.3/linux/splunk-7.0.3-fa31da744b51-linux-2.6-x86_64.rpm&ac=&wget=true&name=wget&platform=Linux&architecture=x86_64&version=7.0.3&product=splunk&typed=release' install it yum localinstall splunk-7.0.3-fa31da744b51-linux-2.6-x86_64.rpm #setup https vim /opt/splunk/etc/system/local/web.conf [settings] enableSplunkWebSSL = 1 #start it /opt/splunk/bin/splunk start --accept-license #CM https://10.1.2.170:8000/en-US/app/launcher/home #Indexer https://10.1.2.236:8000/en-US/app/launcher/home Change password for admin user /opt/splunk/bin/splunk edit user admin -password Jtg0BS0nrAyD -auth admin:changeme Turn on distributed search in the GUI #on CM /opt/splunk/etc/system/local/server.conf [general] site = site1 [clustering] mode = master multisite = true replication_factor = 2 search_factor = 2 max_peer_sum_rep_load = 15 max_peer_rep_load = 15 max_peer_build_load = 6 summary_replication = true site_search_factor = origin:1,site1:1,site2:1,site3:1,total:3 site_replication_factor = origin:1,site1:1,site2:1,site3:1,total:3 available_sites = site1,site2,site3 cluster_label = saf_index_cluster pass4SymmKey = password site_mappings = default:site1 #on IDX /opt/splunk/etc/system/local/server.conf [general] site = site1 [clustering] master_uri = https://10.1.2.170:8089 mode = slave pass4SymmKey = password [replication_port://9887] ***ensure networking is allowed between the hosts*** The indexer will show up in the Cluster master #create this file on the indexer /opt/splunk/etc/apps/saf_all_indexes/local/indexes.conf [junk] homePath = $SPLUNK_DB/junk/db coldPath = $SPLUNK_DB/junk/colddb thawedPath = $SPLUNK_DB/junk/thaweddb #copy the index over to the indexer cp junk_index.targz /opt/splunk/var/lib/splunk/ tar -xzvf junk_index.targz ################################################################################### PROD testing Notes SAF PROD Cluster testing with the te index. The indexers do not have the space to move to search/rep factor 3/3. Duane suggests keeping the current 2/3 and letting the temp splunk cluster make the buckets searchable. according to the monitoring console: te index gathered on Feb 26 total index size: 3.1 GB total raw data size uncompressed: 10.37 GB total events: 12,138,739 earliest event: 2019-05-17 20:40:00 latest event: 2020-02-26 16:43:32 | dbinspect index=te | stats count by splunk_server count of buckets indexer1: 105 indexer2: 103 indexer3: 104 | dbinspect index=te | search state=hot currently 6 hot buckets index=te | stats count ALL TIME fast mode 6069419 size on disk 1.1 GB size of tarball 490 MB Allow instance to write to S3 bucket { "Id": "Policy1582738262834", "Version": "2012-10-17", "Statement": [ { "Sid": "Stmt1582738229969", "Action": [ "s3:PutObject" ], "Effect": "Allow", "Resource": "arn:aws:s3:::mdr-saf-off-boarding/*", "Principal": { "AWS": [ "arn:aws:iam::477548533976:role/msoc-default-instance-role" ] } } ] } ./aws s3 cp rst2odt.py s3://mdr-saf-off-boarding ./aws s3 cp /opt/splunkdata/hot/normal_primary/saf_te_index.tar.gz s3://mdr-saf-off-boarding aws --profile=mdr-prod s3 presign s3://mdr-saf-off-boarding/saf_te_index.tar.gz --expires-in 604800 uploaded brad_LAN key pair to AWS for new instances. vpc-0202aedf3d0417cd3 subnet-01bc9f77742ff132d sg-03dcc0ecde42fc8c2, sg-077ca2baaca3d8d97 saf-offboarding-splunk-cm saf-offboarding-splunk-indexer CentOS 7 (x86_64) - with Updates HVM t2.medium (2 CPU 4 GB RAM) 100 GB drive for te index test msoc-default-instance-role tag instances Client saf use the msoc_build key #CM ip-10-1-3-72 #indexer-1 ip-10-1-3-21 #indexer-2 ip-10-1-3-24 #indexer-3 ip-10-1-3-40 use virtualenv to grab awscli export https_proxy=http://proxy.msoc.defpoint.local:80 sudo -E ./pip install awscli ./aws s3 cp s3://mdr-saf-off-boarding/saf_te_index.tar.gz /opt/splunk/var/lib/splunk/saf_te_index.tar.gz move index to CM rep buckets are not expanding to Search buckets 1. rm -rf saf_all_indexes 2. create it on the CM 2.1 mkdir -p /opt/splunk/etc/master-apps/saf_all_indexes/local/ 2.2 vim /opt/splunk/etc/master-apps/saf_all_indexes/local/indexes.conf [te] homePath = $SPLUNK_DB/te/db coldPath = $SPLUNK_DB/te/colddb thawedPath = $SPLUNK_DB/te/thaweddb repFactor=auto 2.3 cluster bundle push 2.3.1 /opt/splunk/bin/splunk list cluster-peers 2.3.1 splunk validate cluster-bundle 2.3.2 splunk apply cluster-bundle ################### # # Actual PROD offboarding! # ################## #estimate size and age | rest /services/data/indexes/ | search title=app_mscas OR title = app_o365 OR title=dns OR title=forescout OR title=network OR title=security OR title=Te | eval indexSizeGB = if(currentDBSizeMB >= 1 AND totalEventCount >=1, currentDBSizeMB/1024, null()) | eval elapsedTime = now() - strptime(minTime,"%Y-%m-%dT%H:%M:%S%z") | eval dataAge = ceiling(elapsedTime / 86400) | stats sum(indexSizeGB) AS totalSize max(dataAge) as oldestDataAge by title | eval totalSize = if(isnotnull(totalSize), round(totalSize, 2), 0) | eval oldestDataAge = if(isNum(oldestDataAge), oldestDataAge, "N/A") | rename title as "Index" totalSize as "Total Size (GB)" oldestDataAge as "Oldest Data Age (days)" 1. adjust CM and push out new data retention limits per customer email 2. allow indexers to prune old data 3. stop splunk on one indexer 4. tar up splunk directory 5. upload to s3 6. download from s3 to temp indexers and extract to ensure data is readable 7. repeat for all indexes prune data based on time Updated Time 1/6/2020, 1:59:50 PM Active Bundle ID? 73462849B9E88F1DB2B9C60643A06F67 Latest Bundle ID? 73462849B9E88F1DB2B9C60643A06F67 Previous Bundle ID? FF9104B61366E1841FEDB1AF2DE901C2 4 With encryption tar cvzf saf_myindex_index.tar.gz myindex/ without encryption tar cvf /hubble.tar hubble/ trying this: https://github.com/jeremyn/s3-multipart-uploader use virtualenv bin/python s3-multipart-uploader-master/s3_multipart_uploader.py -h bucket name mdr-saf-off-boarding bin/aws s3 cp /opt/splunkdata/hot/saf_te_index.tar.gz s3://mdr-saf-off-boarding/saf_te_index.tar.gz DID NOT NEED TO USE THE MULTIPART uploader! aws --profile=mdr-prod s3 presign s3://mdr-saf-off-boarding/saf_app_mscas_index.tar.gz --expires-in 86400 aws --profile=mdr-prod s3 presign s3://mdr-saf-off-boarding/saf_app_o365_index.tar.gz --expires-in 86400 aws --profile=mdr-prod s3 presign s3://mdr-saf-off-boarding/saf_dns_index.tar.gz --expires-in 86400 aws --profile=mdr-prod s3 presign s3://mdr-saf-off-boarding/saf_forescout_index.tar.gz --expires-in 86400 aws --profile=mdr-prod s3 presign s3://mdr-saf-off-boarding/saf_network_index.tar --expires-in 86400 aws --profile=mdr-prod s3 presign s3://mdr-saf-off-boarding/saf_security_index.tar.gz --expires-in 86400 aws --profile=mdr-prod s3 presign s3://mdr-saf-off-boarding/saf_te_index.tar.gz --expires-in 86400