# Splunk NGA Data Pull Request Notes

Stand up a new "search head" that just has Splunk installed on it, no need to configure the Splunk instance. The Splunk instance will query the actual search head and pull the data out. See Hurricane Labs python script.  [The Best Guide for Exporting Massive Amounts of Data From Splunk](https://hurricanelabs.com/splunk-tutorials/the-best-guide-for-exporting-massive-amounts-of-data-from-splunk/)

[Jira MSOCI-1013 ticket - SPIKE: NGA CheckPoint Log Export Request](https://jira.xdr.accenturefederalcyber.com/browse/MSOCI-1013)

```
vpc-05e0cf38982e048db

subnet-0a2384bce743cf303

MSOC_RedHat_Minion_201807250350 (ami-01c2c25dc719d3546) USED CENTOS 7 AWS AMI 

m4.large

generated SSH key pair bradp.pem
 
nga-splunk-searches

username is centos

delete key pair when done from AWS and the bastion host! bradp

delete svc-searches from nga splunk SH when done

delete 1TB EBS volume when done
```


`search "index=network sourcetype=qos_syslog CA98C333-F830-0B45-A543-4450CDFDA84A 1571414560 Accept 47048" -output rawdata -maxout 0 -max_time 0 -uri https://10.2.2.122:8089`


```
start fail
1019_1020export.raw
1018_1019 times:
head - 2019-09-15T09:14:59
tail - 2019-09-15T09:09:31

end fail
1091_1092export.raw
1093_1094 times:
head - 2019-09-14T14:14:59
tail - 2019-09-14T14:00:00


i=5000
start time 2019-09-15T09:14:59
stop time 2019-09-14T14:00:00


start fail
784_785export.raw
783_784 times:
head - 2019-09-17T19:59:59
tail 2019-09-17T19:46:54

end fail
857_858export.raw
859_860 times:
head  2019-09-17T00:29:59
tail 2019-09-17T00:15:00

i=6000
start time 2019-09-17T20:00:00
stop time 2019-09-17T00:15:00

start fail
909_910export.raw
907_908 times:
head - 2019-09-16T12:59:59
tail - 2019-09-16T12:45:00

end fail
982_983export.raw
985_986 times:
head - 2019-09-15T17:29:59
tail - 2019-09-15T17:15:00

i=7000
start time 2019-09-15T17:30:00
stop time 2019-09-16T12:45:00

```


```
#from my mac
aws s3 ls s3://nga-mdr-data-pull
aws s3 cp nga-splunk-pull.zip s3://nga-mdr-data-pull
aws --profile=mdr-prod s3 presign s3://nga-mdr-data-pull/nga-splunk-pull.zip --expires-in 86400

aws --profile=mdr-prod s3 presign s3://nga-mdr-data-pull/nga-splunk-pull.zip --expires-in 604800

https://nga-mdr-data-pull.s3.amazonaws.com/nga-splunk-pull.zip?AWSAccessKeyId=ASIAW6MA4LDMBGUOE7Q6&Signature=6WZ9KdHfH4rj28Ey5hrTib8HcHM%3D&x-amz-security-token=FQoGZXIvYXdzEFIaDCbQsc24x7kkQnhLQSL%2FAV4UBSVowGvhyMyS41rQtbtnmznvrbIu5Y9CCrxJ65RP%2BMeHz7Jkwu8BFEzNeeIT5M6Dfcd1NdFkqXBjE54y6G6HujSSLPk8gp2UqGDKkqMDE3qzrXfHRKaIlMInkACQi6VPpRDjFYGnnILS8vO5gjzqr9HUAsIgfVwpEuVf%2FPBbEcuUH87kZS6FqyQHTBc%2BcPk8KetsX2IuLmpOVAysip3IGgx2duVETNqKH0uXOM%2FUBygyJ7gD3DLoQWqCHQvxG0AfO0vEkRAZxgLKSDm6E2c8d9mJ5I6yXl2xBK7ii5bKWmhWtnPGYrErVFTxhfqeI6SHwzJOsLlNdkAC6nSKRyi1wMztBQ%3D%3D&Expires=1572625186


tail -1 1018_1019export.raw
```