infrastructure-notes/Splunk Notes.md

Splunk Notes

---

Change user to Splunk

sudo -iu splunk

---

How to apply the git changes to the CM or customer DS. Be patient, it is splunk. Review logs in salt

Chris broke Jenkins.but he moved the splunk git repo to gitfs

1. add your changes to the appropriate git repo (msoc-moose-cm)
2. then use the salt state to push the changes and apply the new bundle salt 'moose-splunk-cm' state.sls splunk.master.apply_bundle_master salt 'afs-splunk-cm' state.sls splunk.master.apply_bundle_master

Apply the git changes to the splunk UFs (Salt Deployment Server)

Moose DS has a salt file for pushing apps out directly to UFs.

Customer DS salt 'afs-splunk-ds*' state.sls splunk.deployment_server.reload_ds

to view the splunk command output look at the logs in splunk under the return.cmd_...changes.stdout or stderr index=salt sourcetype=salt_json fun="state.sls"

---

Splunk CM is the license master and the salt master is used to push out a new license. Each customer has its own license.

---

TEST SPLUNK CM admin password admin 6VB^8V3CFjbaiZ4Q#hLjNW3a1

TEST SPLUNK indexer-* admin password 6VB8V3CFjbaiZ4QhLjNW3a1

SEARCHES

| tstats values(sourcetype) where index=* group by index

#collectd | mstats count WHERE index=collectd metric_name=* by host, metric_name

#aws cloudtrail index=app_aws sourcetype=aws:cloudtrail

#proxy index=web sourcetype=squid:access:json

CLI search /opt/splunk/bin/splunk search 'index=bro' -earliest_time '-5m' output=raw > test.text

#NGA data request for checkpoint logs index=network sourcetype=qos_syslog (service=443 OR service=80) NOT src=172.20.109.16 NOT src=172.20.109.17 NOT dst=172.20.109.16 NOT dst=172.20.109.17 NOT (action=Drop src=172.20.8.3)

updated index=network sourcetype=qos_syslog (service=443 OR service=80) NOT (action=Drop src=172.20.8.3)

#Vault index=app_vault

| rest /services/data/indexes/ | search title=app_mscas OR title = app_o365 OR title=dns OR title=forescout OR title=network OR title=security OR title=Te

coldToFrozenScript

Yes, this is a mess. Moose is running a version of splunk that breaks with the coldToFrozen script being pushed from the CM in an app. To get around this, i moved it to /usr/local/bin. The other customers have the script in the app.

ERROR: runcoldToFrozen and get SyntaxError. SOLUTION: upgrade the awscli with pip3 ( run the splunk.indexer state. )