infrastructure-notes/Splunk Notes.md

Splunk Notes

---

Change user to Splunk

sudo -iu splunk

---

How to apply the git changes to the CM or customer DS. Be patient, it is splunk. Review logs in salt

Chris broke Jenkins.but he moved the splunk git repo to gitfs

1. add your changes to the appropriate git repo (msoc-moose-cm)
2. then use the salt state to push the changes and apply the new bundle salt 'moose-splunk-cm' state.sls splunk.master.apply_bundle_master salt 'afs-splunk-cm' state.sls splunk.master.apply_bundle_master

Apply the git changes to the splunk UFs (Salt Deployment Server)

Moose DS has a salt file for pushing apps out directly to UFs.

Customer DS salt 'afs-splunk-ds*' state.sls splunk.deployment_server.reload_ds

to view the splunk command output look at the logs in splunk under the return.cmd_...changes.stdout or stderr index=salt sourcetype=salt_json fun="state.sls"

---

Splunk CM is the license master and the salt master is used to push out a new license. Each customer has its own license.

---

TEST SPLUNK CM admin password admin 6VB^8V3CFjbaiZ4Q#hLjNW3a1

############## # #SEARCHES # ############### | tstats values(sourcetype) where index=* group by index

#aws cloudtrail index=app_aws sourcetype=aws:cloudtrail

#proxy index=web sourcetype=squid:access:json

CLI search /opt/splunk/bin/splunk search 'index=bro' -earliest_time '-5m' output=raw > test.text

#NGA data request for checkpoint logs index=network sourcetype=qos_syslog (service=443 OR service=80) NOT src=172.20.109.16 NOT src=172.20.109.17 NOT dst=172.20.109.16 NOT dst=172.20.109.17 NOT (action=Drop src=172.20.8.3)

updated index=network sourcetype=qos_syslog (service=443 OR service=80) NOT (action=Drop src=172.20.8.3)

#Vault index=app_vault

| rest /services/data/indexes/ | search title=app_mscas OR title = app_o365 OR title=dns OR title=forescout OR title=network OR title=security OR title=Te