infrastructure-notes/Collectd Notes.md

Collectd Notes.md

Collectd is used to tracking hard drive space and cpu usage. The data is collectd on the servers and sent to the Moose indexers via HEC.

Service

systemctl restart collectd

How do i search the data in Splunk?

05/08/2020

Basic

| mstats count WHERE index=collectd metric_name=* by host, metric_name

CPU idle with timechart

| mstats avg(_value) as "Avg" WHERE index=collectd metric_name=cpu.idle.value host=bastion* span=1m 
| timechart max("Avg") span=5m

CPU Usage with timechart

| mstats avg(_value) as "Avg"  WHERE index=collectd host=bastion* metric_name=cpu.system.value OR metric_name=cpu.user.value by metric_name span=1m
| timechart first("Avg") by metric_name span=1h

Junk in syslog

Currently a bug in collectd where it writes the response from HEC into the system log /var/log/messages. There's a github issue, Collectd Issue 3105. Duane has a PR in to fix it, in theory - PR 3263

Duane's PR has been merged.