Splunk NGA Data Pull Request Notes.md 2.8 KB
Splunk NGA Data Pull Request Notes
Stand up a new "search head" that just has Splunk installed on it, no need to configure the Splunk instance. The Splunk instance will query the actual search head and pull the data out. See Hurricane Labs python script. The Best Guide for Exporting Massive Amounts of Data From Splunk
Jira MSOCI-1013 ticket - SPIKE: NGA CheckPoint Log Export Request
vpc-05e0cf38982e048db
subnet-0a2384bce743cf303
MSOC_RedHat_Minion_201807250350 (ami-01c2c25dc719d3546) USED CENTOS 7 AWS AMI
m4.large
generated SSH key pair bradp.pem
nga-splunk-searches
username is centos
delete key pair when done from AWS and the bastion host! bradp
delete svc-searches from nga splunk SH when done
delete 1TB EBS volume when done
search "index=network sourcetype=qos_syslog CA98C333-F830-0B45-A543-4450CDFDA84A 1571414560 Accept 47048" -output rawdata -maxout 0 -max_time 0 -uri https://10.2.2.122:8089
start fail
1019_1020export.raw
1018_1019 times:
head - 2019-09-15T09:14:59
tail - 2019-09-15T09:09:31
end fail
1091_1092export.raw
1093_1094 times:
head - 2019-09-14T14:14:59
tail - 2019-09-14T14:00:00
i=5000
start time 2019-09-15T09:14:59
stop time 2019-09-14T14:00:00
start fail
784_785export.raw
783_784 times:
head - 2019-09-17T19:59:59
tail 2019-09-17T19:46:54
end fail
857_858export.raw
859_860 times:
head 2019-09-17T00:29:59
tail 2019-09-17T00:15:00
i=6000
start time 2019-09-17T20:00:00
stop time 2019-09-17T00:15:00
start fail
909_910export.raw
907_908 times:
head - 2019-09-16T12:59:59
tail - 2019-09-16T12:45:00
end fail
982_983export.raw
985_986 times:
head - 2019-09-15T17:29:59
tail - 2019-09-15T17:15:00
i=7000
start time 2019-09-15T17:30:00
stop time 2019-09-16T12:45:00
#from my mac
aws s3 ls s3://nga-mdr-data-pull
aws s3 cp nga-splunk-pull.zip s3://nga-mdr-data-pull
aws --profile=mdr-prod s3 presign s3://nga-mdr-data-pull/nga-splunk-pull.zip --expires-in 86400
aws --profile=mdr-prod s3 presign s3://nga-mdr-data-pull/nga-splunk-pull.zip --expires-in 604800
https://nga-mdr-data-pull.s3.amazonaws.com/nga-splunk-pull.zip?AWSAccessKeyId=ASIAW6MA4LDMBGUOE7Q6&Signature=6WZ9KdHfH4rj28Ey5hrTib8HcHM%3D&x-amz-security-token=FQoGZXIvYXdzEFIaDCbQsc24x7kkQnhLQSL%2FAV4UBSVowGvhyMyS41rQtbtnmznvrbIu5Y9CCrxJ65RP%2BMeHz7Jkwu8BFEzNeeIT5M6Dfcd1NdFkqXBjE54y6G6HujSSLPk8gp2UqGDKkqMDE3qzrXfHRKaIlMInkACQi6VPpRDjFYGnnILS8vO5gjzqr9HUAsIgfVwpEuVf%2FPBbEcuUH87kZS6FqyQHTBc%2BcPk8KetsX2IuLmpOVAysip3IGgx2duVETNqKH0uXOM%2FUBygyJ7gD3DLoQWqCHQvxG0AfO0vEkRAZxgLKSDm6E2c8d9mJ5I6yXl2xBK7ii5bKWmhWtnPGYrErVFTxhfqeI6SHwzJOsLlNdkAC6nSKRyi1wMztBQ%3D%3D&Expires=1572625186
tail -1 1018_1019export.raw