infrastructure-notes/Customer Decommision Notes.md

Customer decommision Notes.md

Follow these steps to permanently decommision a customer.

Remove the Customer POP/LCP Nodes

5/18/2020

Shutdown Splunk and disable to prevent new data going to the cluster.

salt saf-splunk-syslog-* cmd.run 'systemctl stop splunk'
salt saf-splunk-syslog-* cmd.run 'systemctl disable splunk'

salt -C 'saf-splunk-* not *.local' cmd.run 'systemctl stop splunk'
salt -C 'saf-splunk-* not *.local' cmd.run 'rm -rf /opt/*'

salt -C 'saf-splunk-* not *.local' cmd.run 'rm -rf /var/log/*'
salt -C 'saf-splunk-* not *.local' cmd.run 'rm -rf /etc/salt/minion && shutdown now'

salt saf-splunk-syslog-* cmd.run 'systemctl stop syslog-ng'
salt saf-splunk-syslog-* cmd.run 'systemctl disable syslog-ng'
salt saf-splunk-dcn-* cmd.run 'docker stop mdr-syslog-ng'

Follow these steps to terminate a customer slice

05/3/2021

See Splunk SAF Offboarding Notes.md for notes on pulled data off an indexer to give to the customer.

Terraform, Sensu, SFT Removal

Update TF code and remove whitelisted SG IPs and/or rules to remove access from POP to C&C, Salt master, and splunk indexers. This is stored in globals.hcl or account.hcl

• Destroy the instances with the terraform destroy command in the appropriate folders.
• Create new git branch in XDR-Terraform-Live
• Remove the appropriate folder (e.g. mdr-prod-CUSTOMERPREFIX ) Remove references to LCP nodes in the globals.hcl file.
• Remove Terraform salt provision references ( LEGACY ) ( terraform/02-msoc_vpc/cloud-init/provision_salt_master.sh )

• Terraform C&C IP whitelisting for salt master and reposerver ( terraform/02-msoc_vpc/security-groups.tf )

  1. Terraform customer folder ( terraform/102-saf/ )
  2. Terraform common variables ( terraform/common/variables.tf )

• Remove Customer from Portal Lambda Env Var ( base/customer_portal_lambda/main.tf )

  1. Once sensu starts alerting, delete the sensu entities and resolve the alerts
  2. On the salt master, delete the salt minion keys
  3. On ScaleFT website, delete the servers and project
  4. In the redhat website, remove the entitlements
  5. Ensure the customer vpc is fully deleted and no dependencies remain
  6. Delete the customer folder from the TF and update develop and master branches

Remove the Customer from the Salt Code

Remove references of the customer from these places:

1. Splunk Monitoring Console
2. salt/pillar/mc_variables.sls ( apply the changes here: salt/fileroots/splunk/monitoring_console/init.sls - salt/fileroots/splunk/search_head/init.sls )
3. Salt master configs ( salt/fileroots/salt_master/files/etc/salt/master.d/default_acl.conf )
4. Delete Salt Splunk files ( salt/pillar/${CUSTOMERPREFIX}_variables.sls salt/pillar/${CUSTOMERPREFIX}_pop_settings.sls)
5. Salt top.sls and pillar/top.sls ( salt/fileroots/top.sls - salt/pillar/top.sls )
6. Salt global_variables.sls, os_settings.sls (salt/pillar/global_variables.sls - salt/pillar/os_settings.sls )
7. Salt gitfs pillar ( salt/pillar/salt_master.sls )

Update salt master salt salt* state.sls salt_master

Report the Decommissioned Hosts to the ISSO/AFCC Team

afcc@accenturefederal.com;asha.a.nair@accenturefederal.com

SUBJECT: Decommissioned Devices

Hello,

The below instances have been decommissioned from the environment and should be removed from any reports or inventories. 

<list full splunk UF name of instances>

Thanks,
Brad

The SOC will edit this lookup ( or you can just do it. ) https://moose-splunk.pvt.xdr.accenturefederalcyber.com/en-US/app/SplunkEnterpriseSecuritySuite/ess_lookups_edit?namespace=SA-IdentityManagement&transform=simple_asset_lookup

Salt Master Keys Removal

Deactivate OKTA Apps

Sensu Agent Cleanup

SFT Cleanup

RedHat Licence Cleanup

Qualys Cleanup

Go to Qualys Dashboard -> Cloud Agent -> Activation Keys

Archive Customer Git Repos

Do this after the Salt Master gitfs has been updated to avoid any error messages.

Git > Settings > Options > Archive this repository

Update the AWS Configuration

files/config in infrastructure-notes

Remove the AWS Account if we don't have access anymore. https://github.xdr.accenturefederalcyber.com/mdr-engineering/msoc-infrastructure/wiki/Cloud-Accounts

Clean Up Vault Passwords

Delete engineering/customer_slices/ Disable onboarding-

Remove AMI Access to AWS Account

Refresh the Monitoring Console webpage