# Customer decommision Notes.md

Follow these steps to permently decommision a customer. 

## Remove the Customer POP/LCP Nodes
5/18/2020

Shutdown Splunk and disable to prevent new data going to the cluster. 

```
salt saf-splunk-syslog-* cmd.run 'systemctl stop splunk'
salt saf-splunk-syslog-* cmd.run 'systemctl disable splunk'

salt -C 'saf-splunk-* not *.local' cmd.run 'systemctl stop splunk'
salt -C 'saf-splunk-* not *.local' cmd.run 'rm -rf /opt/*'

salt -C 'saf-splunk-* not *.local' cmd.run 'rm -rf /var/log/*'
salt -C 'saf-splunk-* not *.local' cmd.run 'rm -rf /etc/salt/minion && shutdown now'

salt saf-splunk-syslog-* cmd.run 'systemctl stop syslog-ng'
salt saf-splunk-syslog-* cmd.run 'systemctl disable syslog-ng'
salt saf-splunk-dcn-* cmd.run 'docker stop mdr-syslog-ng'
```


## Follow these steps to terminate a customer slice
05/3/2021

See `Splunk SAF Offboarding Notes.md` for notes on pulled data off an indexer to give to the customer. 

### Terraform, Sensu, SFT Removal

Update TF code and remove whitelisted SG IPs and/or rules to remove access from POP to C&C, Salt master, and splunk indexers. This is stored in globals.hcl or account.hcl

- Destroy the instances with the `terraform destroy` command in the appropriate folders.
- Create new git branch in XDR-Terraform-Live
- Remove the appropriate folder (e.g. mdr-prod-CUSTOMERPREFIX )
Remove references to LCP nodes in the globals.hcl file. 
- Remove Terraform salt provision references ( LEGACY ) ( terraform/02-msoc_vpc/cloud-init/provision_salt_master.sh )
- Terraform C&C IP whitelisting for salt master and reposerver ( terraform/02-msoc_vpc/security-groups.tf )
 11. Terraform customer folder ( terraform/102-saf/ )
 12. Terraform common variables ( terraform/common/variables.tf )

- Remove Customer from Portal Lambda Env Var ( base/customer_portal_lambda/main.tf )
 1. Once sensu starts alerting, delete the sensu entities and resolve the alerts
 2. On the salt master, delete the salt minion keys
 3. On ScaleFT website, delete the servers and project
 4. In the redhat website, remove the entitlements
 5. Ensure the customer vpc is fully deleted and no dependencies remain
 6. Delete the customer folder from the TF and update develop and master branches



### Remove the Customer from the Salt Code

Remove references of the customer from these places:

 2. Splunk Monitoring Console 
 - salt/pillar/mc_variables.sls
  ( apply the changes here: salt/fileroots/splunk/monitoring_console/init.sls  - salt/fileroots/splunk/search_head/init.sls )
 3. Salt master configs ( salt/fileroots/salt_master/files/etc/salt/master.d/default_acl.conf )
 4. Delete Salt Splunk files ( salt/pillar/${CUSTOMERPREFIX}_variables.sls salt/pillar/${CUSTOMERPREFIX}_pop_settings.sls)
 5. Salt top.sls and pillar/top.sls ( salt/fileroots/top.sls - salt/pillar/top.sls )
 6. Salt global_variables.sls, os_settings.sls (salt/pillar/global_variables.sls - salt/pillar/os_settings.sls )
 8. Salt gitfs pillar ( salt/pillar/salt_master.sls )

 
Update salt master
`salt salt* state.sls salt_master`

### Report the Decommissioned Hosts to the ISSO/AFCC Team
```
afcc@accenturefederal.com;asha.a.nair@accenturefederal.com
```

SUBJECT: Decommissioned Devices

```
Hello,

The below instances have been decommissioned from the environment and should be removed from any reports or inventories. 

<list full splunk UF name of instances>

Thanks,
Brad
```

The SOC will edit this lookup ( or you can just do it. )
https://moose-splunk.pvt.xdr.accenturefederalcyber.com/en-US/app/SplunkEnterpriseSecuritySuite/ess_lookups_edit?namespace=SA-IdentityManagement&transform=simple_asset_lookup


### Salt Master Keys Removal

### Deactivate OKTA Apps

### Sensu Agent Cleanup

### SFT Cleanup

### RedHat Licence Cleanup

### Qualys Cleanup
Go to Qualys Dashboard -> Cloud Agent -> Activation Keys

### Archive Customer Git Repos
Do this after the Salt Master gitfs has been updated to avoid any error messages. 

Git > Settings > Options > Archive this repository

### Update the AWS Configuration
files/config in infrastructure-notes 

Remove the AWS Account if we don't have access anymore. 
https://github.xdr.accenturefederalcyber.com/mdr-engineering/msoc-infrastructure/wiki/Cloud-Accounts

### Clean Up Vault Passwords
Delete engineering/customer_slices/<customer-name>
Disable onboarding-<customer-name>

### Remove AMI Access to AWS Account

### Refresh the Monitoring Console webpage
