# C2 decommision Notes.md

Follow these steps to permanently decommision an XDR C2 device; ie, MailRelay, Sensu, etc. 

## Remove the Customer POP/LCP Nodes
9/29/2021 - Double check this

Rough draft for now - 9/29/2021
```
I think the steps are.

Silence entity in sensu

Shutdown the instance via command-line.

Add to https://moose-splunk.pvt.xdr.accenturefederalcyber.com/en-US/app/SplunkEnterpriseSecuritySuite/ess_lookups_edit?namespace=SA-IdentityManagement&transform=simple_asset_lookup&file=assets.csv&owner=nobody

Wait a few days to make sure you didn't make a big mistake.

Update the module(s) and/or xdr-terraform-live repo to disable the server.

Test in test, then do a PR for prod.

Once merged, apply the PR.

Write down notes in your ticket and close it.
```

#Check for Splunk and disable to prevent new data going to the cluster. 

```
salt mailrelay.* cmd.run 'systemctl stop splunk'
salt mailrelay.* cmd.run 'systemctl disable splunk'

salt -C 'mailrelay.* not *.local' cmd.run 'systemctl stop splunk'
salt -C 'mailrelay.* not *.local' cmd.run 'rm -rf /opt/*'

salt -C 'mailrelay.* not *.local' cmd.run 'rm -rf /var/log/*'
salt -C 'mailrelay.* not *.local' cmd.run 'rm -rf /etc/salt/minion && shutdown now'

salt mailrelay.* cmd.run 'systemctl stop syslog-ng'
salt mailrelay.* cmd.run 'systemctl disable syslog-ng'
salt mailrelay.* cmd.run 'docker stop mdr-syslog-ng'
```


## Follow these steps to terminate a customer slice
05/3/2021

See [Splunk SAF Offboarding Notes.md](Splunk%20SAF%20Offboarding%20Notes.md) for notes on pulled data off an indexer to give to the customer. 

### Terraform, Sensu, SFT Removal

Update TF code and remove whitelisted SG IPs and/or rules to remove access from POP to C&C, Salt master, and splunk indexers. This is stored in globals.hcl or account.hcl

- Silence instance(s) in Sensu to avoid notifications
- Disable termination protection in AWS console
- Destroy the AWS objects with the `terragrunt destroy` command in all folders except 005-iam. Ignore error deleting S3 bucket BucketNotEmpty in 006-account-standards. (170-splunk-searchhead, 180-splunk-heavy-forwarder, 150-splunk-cluster-master, 160-splunk-indexer-cluster, 140-splunk-frozen-bucket, 010-vpc-splunk,072-salt-master-inventory-role, 021-qualys-connector-role, 007-backups, 006-account-standards-regional, 006-account-standards) 
- Create new git branch in XDR-Terraform-Live
- Remove the folders that were just destoryed ( NOT 005-iam or account.hcl ) to ensure the instances can not be created again
- Ensure the customer vpc is fully deleted in the AWS console
- Remove AWS Account from the partition.hcl file in the account_map["prod"] variable  ( common/aws-us-gov/partition.hcl )


#### Remove references to LCP nodes in the globals.hcl file. 


- Remove customer IPs from C&C IP whitelisting in xdr-terraform-live/globals.hcl in the c2_services_external_ips variable
- Remove customer IPs from Moose SG whitelisting in xdr-terraform-live/prod/aws-us-gov/mdr-prod-c2/account.hcl in the splunk_data_sources variable 
- Remove customer from Portal Lambda customer_vars variable in xdr-terraform-live/prod/aws-us-gov/mdr-prod-c2/205-customer-portal-lambda/terragrunt.hcl 
- Delete the sensu entities and resolve any alerts
- On the salt master, delete the salt minion keys `sudo salt-key -d <CUSTOMER-PREFIX>*`
- On ScaleFT website, delete the project and servers
- In the redhat website, remove the entitlements. Check for LCP nodes that used an entitlement 

- Commit the changes to the xdr-terraform-live repo and get merged into master 
- After changes have been merged in git, apply the changes to remove the IPs from the security groups and the AWS account from transit gateway
    - prod/aws-us-gov/mdr-prod-c2/275-nessus-security-managers
    - prod/aws-us-gov/mdr-prod-c2/205-customer-portal-lambda
    - prod/aws-us-gov/mdr-prod-c2/160-splunk-indexer-cluster
    - prod/aws-us-gov/mdr-prod-c2/095-instance-sensu
    - prod/aws-us-gov/mdr-prod-c2/080-instance-repo-server
    - prod/aws-us-gov/mdr-prod-c2/071-instance-salt-master
    - prod/aws-us-gov/mdr-prod-c2/008-transit-gateway-hub
    - prod/aws-us-gov/mdr-prod-c2/005-account-standards-c2


### Remove the GovCloud and Commercial AWS account ID from Packer and Salt 

https://github.xdr.accenturefederalcyber.com/mdr-engineering/msoc-infrastructure/wiki/Cloud-Accounts 

- Create new git branch in msoc_infrastructure
- Remove Packer AWS accounts in packer/Makefile
- Remove AWS accounts in salt/fileroots/salt_master/files/xdr_asset_inventory/xdr_asset_inventory.sh

Be sure to check for both Gov and Commerical AWS Accounts

### Remove the Customer from the Salt Code

Remove references of the customer from these places:

- Splunk Monitoring Console - salt/pillar/mc_variables.sls
- Salt master configs in salt/fileroots/salt_master/files/etc/salt/master.d/default_acl.conf )
- Delete Salt Splunk files - salt/pillar/${CUSTOMERPREFIX}_variables.sls salt/pillar/${CUSTOMERPREFIX}_pop_settings.sls
- Salt top.sls and pillar/top.sls - salt/fileroots/top.sls - salt/pillar/top.sls 
- Salt okta auth in salt/pillar/os_settings.sls 
- Salt gitfs pillar in salt/pillar/salt_master.sls 
- Salt FM Shared Search in salt/pillar/fm_shared_search.sls

 
Apply changes in salt to remove references to the old customer. 

Update salt master  
`salt salt* state.sls salt_master --output-diff test=true`

Update the FM search head and monitoring console
`salt splunk-mc-0* state.sls splunk.monitoring_console --output-diff test=true`
`salt fm-shared-search-0* state.sls splunk.fm_shared_search --output-diff test=true`

Disable the instances in the Monitoring Console webpage ( how to delete the instances? )
Verify the search peers have been removed from the distributed search in the FM Shared Searchhead

### Deactivate OKTA Apps
Each customer should have three applications. Deactive the app, then delete it. 
<CUSTOMER-PREFIX> Splunk CM
<CUSTOMER-PREFIX> Splunk HF
<CUSTOMER-PREFIX> Splunk SH


### Qualys Cleanup
Go to Qualys Dashboard -> Cloud Agent -> Activation Keys
Disable the key, not sure how to delete it. Perhaps have to wait a period of time?

### Archive Customer Git Repos
Do this after the Salt Master gitfs has been updated to avoid any error messages. 

Git > Settings > Options > Archive this repository
msoc-<CUSTOMER-PREFIX>-cm
msoc-<CUSTOMER-PREFIX>-pop

### Clean Up Vault Passwords
Delete engineering/customer_slices/<CUSTOMER-PREFIX>
Disable onboarding-<CUSTOMER-PREFIX>

### Report the Decommissioned Hosts to the ISSO/AFCC Team

Look in the splunk inventory for the Splunk names or look for emails indicating the logs are not sending. 

```
afcc@accenturefederal.com;asha.a.nair@accenturefederal.com
Accenture Federal Cyber Center <afcc@accenturefederal.com>; Nair, Asha A. <asha.a.nair@accenturefederal.com>
```

SUBJECT: Decommissioned XDR Devices

```
Hello,

The below instances have been decommissioned from the environment and should be removed from any reports or inventories. 

<list full splunk UF name of instances>


```

This lookup also needs to be edited. 
https://moose-splunk.pvt.xdr.accenturefederalcyber.com/en-US/app/SplunkEnterpriseSecuritySuite/ess_lookups_edit?namespace=SA-IdentityManagement&transform=simple_asset_lookup


### Request AWS account be fully terminated
help desk ticket with camrs. Or have Soofi, Osman <osman.soofi@accenturefederal.com> submit a CAMRS disconnect ticket. not sure which one is the best method yet. IMPORTANT: After the account is closed, AWS allows users to login for 90 days. 

```
AFS.Help <afs.help@accenturefederal.com>; XDR-Engineering <xdr.eng@accenturefederal.com>
```

SUBJECT: Decommission CAMRS AWS Account

```
Hello,

Please inform the CAMRS team that these AWS Accounts for <CUSTOMER-PREFIX> are no longer needed and can be decommissioned.

<AWS-ACCOUNT-ID-GOV>
<AWS-ACCOUNT-ID-COMMERCIAL>
```

### Update the AWS Configuration
files/config in infrastructure-notes 

Mark the AWS Account decommissioned in the WIKI once the email to help desk has been sent. We should keep the AWS Account numbers just in case they are needed in the future. 
https://github.xdr.accenturefederalcyber.com/mdr-engineering/msoc-infrastructure/wiki/Cloud-Accounts


### Remove account from Terraform 

IMPORTANT: After the account is closed, AWS allows users to login for 90 days. After AWS account has been decommissioned by CAMRS team, run `terragrunt destroy` in the 005-iam account to prevent users from assuming role into the account. Then remove the mdr-prod-<CUSTOMER-PREFIX> folder from the xdr-terraform-live git repo.