infrastructure-notes/Customer Decommision Notes.md

Customer decommision Notes.md

Follow these steps to permently decommision a customer.

These steps are to remove a customer POP

5/18/2020

Shutdown Splunk and disable to prevent new data going to the cluster.

salt saf-splunk-syslog-* cmd.run 'systemctl stop splunk'
salt saf-splunk-syslog-* cmd.run 'systemctl disable splunk'

salt -C 'saf-splunk-* not *.local' cmd.run 'systemctl stop splunk'
salt -C 'saf-splunk-* not *.local' cmd.run 'rm -rf /opt/*'

salt -C 'saf-splunk-* not *.local' cmd.run 'rm -rf /var/log/*'
salt -C 'saf-splunk-* not *.local' cmd.run 'rm -rf /etc/salt/minion && shutdown now'

salt saf-splunk-syslog-* cmd.run 'systemctl stop syslog-ng'
salt saf-splunk-syslog-* cmd.run 'systemctl disable syslog-ng'
salt saf-splunk-dcn-* cmd.run 'docker stop mdr-syslog-ng'

Update TF code and remove whitelisted SG IPs and/or rules to remove access from POP to C&C, Salt master, and splunk indexers.

Follow these steps to terminate a customer slice

05/18/2020

See Splunk SAF Offboarding Notes.md for notes on pulled data off an indexer to give to the customer.

Don't just terminate the instance, run terraform destroy in the appropriate folder!

1. Once sensu starts alerting, delete the sensu entities and resolve the alerts
2. On the salt master, delete the salt minion keys
3. On ScaleFT website, delete the servers and project
4. In the redhat website, remove the entitlements
5. Ensure the customer vpc is fully deleted and no dependencies remain
6. Delete the customer folder from the TF and update develop and master branches
7. Email Asha (Compliance/ISSO) and inform her that the servers can be removed from the FedRAMP inventory

Remove IPs SAF: 12.42.184.208

Remove the Customer from the Code

Remove references of the customer from these places:

1. Atlantis configs ( atlantis.yaml )
2. Splunk Monitoring Console ( salt/fileroots/splunk/monitoring_console/init.sls - salt/fileroots/splunk/search_head/init.sls )
3. Salt master configs ( default_acl.conf )
4. Salt Splunk files (salt/fileroots/splunk/files/saf_variables.jinja)
5. Salt top.sls and pillar/top.sls ( salt/fileroots/top.sls - salt/pillar/top.sls )
6. Salt global_variables.sls, os_settings.sls (salt/pillar/global_variables.sls - salt/pillar/os_settings.sls )
7. Salt Customer specific Pillars ( salt/pillar/saf_pop_settings.sls - salt/pillar/saf_variables.sls )
8. Salt gitfs pillar ( salt/pillar/salt_master.sls )
9. Terraform salt provision references ( terraform/02-msoc_vpc/cloud-init/provision_salt_master.sh )
10. Terraform C&C IP whitelisting for salt master and reposerver ( terraform/02-msoc_vpc/security-groups.tf )
11. Terraform customer folder ( terraform/102-saf/ )
12. Terraform common variables ( terraform/common/variables.tf )

Update salt master salt salt* state.sls salt_master

Report the Decommissioned Hosts to the AFCC Team