xdr-terraform-modules/submodules/splunk/splunk_indexer_asg

Splunk Indexer ASG TF Module

If you are reading this, welcome. You must be in some kind of trouble to have made it all the way down here.

Why an Auto Scaling Group (ASG)?

Sometimes Amazon instances fail. It is rare, but if an Amazon instance fails, the ASG will detect it, launch a new instance, and termiante the failed instance.

Why three ASGs?

Three ensures that the indexers are spread across the availablity zones.

How do I increase the number of indexers?

Go to the "parent" terraform file ( e.g. moose.tf ) and increase the variable asg_size_x variable to the desired number for the ASG. Then apply the terraform state.

How do I decrease the number of indexers?

Go to the "parent" terraform file ( e.g. moose.tf ) and decrease the variable asg_size_x variable to the desired number for the ASG. Then apply the terraform state. This should be done 1 number at a time do ensure no data is lost. Indexers using an older launch template version will be choosen for termination first. If two indexers were launched at the same time with the same launch template they are choosen from at random.

https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-instance-termination.html

Will the ASG safely terminate a Splunk indexer?

No, if you want to run splunk offline --enforce-counts please run that manually prior to reducing the size of the ASG. What about the cluster master? The termianted instances will be cleaned up on the CM when monthly reboots occur.

What will cause the ASG to terminate the instances?

Will changing the launch template cause the instances to be terminated and recreated? No. Changing the launch template will not cause the instances to be terminated. The ASG should point to the latest verison of the launch template.

What should I do after an instance is terminated?

Manually remove it from Salt, ScaleFT, RedHat licensing, and Sensu.

Block device mappings

/dev/xvdf /opt/splunk /dev/xvdg LVM for /opt/splunkdata/hot (if no NVME devices attached) /dev/xvdh LVM for /opt/splunkdata/hot (if no NVME devices attached)

terraform/modules/splunk_cluster/cloud-init/nvme-setup.sh terraform/modules/splunk_cluster/cloud-init/opt_splunk.boothook