Home Explore Help
Register Sign In
AFS
/
infrastructure-notes
2
0
Fork 0
Files Issues 0 Pull Requests 0 Wiki
Branch: master
Branches Tags
infrastructure-...
/
Decommission C2 Notes.md

Decommission C2 Notes.md 8.4 KB
Permalink History Raw

C2 decommision Notes.md

Follow these steps to permanently decommision an XDR C2 device; ie, MailRelay, Sensu, etc.

Remove the Customer POP/LCP Nodes

9/29/2021 - Double check this

Rough draft for now - 9/29/2021

I think the steps are.

Silence entity in sensu

Shutdown the instance via command-line.

Add to https://moose-splunk.pvt.xdr.accenturefederalcyber.com/en-US/app/SplunkEnterpriseSecuritySuite/ess_lookups_edit?namespace=SA-IdentityManagement&transform=simple_asset_lookup&file=assets.csv&owner=nobody

Wait a few days to make sure you didn't make a big mistake.

Update the module(s) and/or xdr-terraform-live repo to disable the server.

Test in test, then do a PR for prod.

Once merged, apply the PR.

Write down notes in your ticket and close it.

#Check for Splunk and disable to prevent new data going to the cluster.

salt mailrelay.* cmd.run 'systemctl stop splunk'
salt mailrelay.* cmd.run 'systemctl disable splunk'

salt -C 'mailrelay.* not *.local' cmd.run 'systemctl stop splunk'
salt -C 'mailrelay.* not *.local' cmd.run 'rm -rf /opt/*'

salt -C 'mailrelay.* not *.local' cmd.run 'rm -rf /var/log/*'
salt -C 'mailrelay.* not *.local' cmd.run 'rm -rf /etc/salt/minion && shutdown now'

salt mailrelay.* cmd.run 'systemctl stop syslog-ng'
salt mailrelay.* cmd.run 'systemctl disable syslog-ng'
salt mailrelay.* cmd.run 'docker stop mdr-syslog-ng'

Follow these steps to terminate a customer slice

05/3/2021

See Splunk SAF Offboarding Notes.md for notes on pulled data off an indexer to give to the customer.

Terraform, Sensu, SFT Removal

Update TF code and remove whitelisted SG IPs and/or rules to remove access from POP to C&C, Salt master, and splunk indexers. This is stored in globals.hcl or account.hcl

  • Silence instance(s) in Sensu to avoid notifications
  • Disable termination protection in AWS console
  • Destroy the AWS objects with the terragrunt destroy command in all folders except 005-iam. Ignore error deleting S3 bucket BucketNotEmpty in 006-account-standards. (170-splunk-searchhead, 180-splunk-heavy-forwarder, 150-splunk-cluster-master, 160-splunk-indexer-cluster, 140-splunk-frozen-bucket, 010-vpc-splunk,072-salt-master-inventory-role, 021-qualys-connector-role, 007-backups, 006-account-standards-regional, 006-account-standards)
  • Create new git branch in XDR-Terraform-Live
  • Remove the folders that were just destoryed ( NOT 005-iam or account.hcl ) to ensure the instances can not be created again
  • Ensure the customer vpc is fully deleted in the AWS console
  • Remove AWS Account from the partition.hcl file in the account_map["prod"] variable ( common/aws-us-gov/partition.hcl )

Remove references to LCP nodes in the globals.hcl file.

  • Remove customer IPs from C&C IP whitelisting in xdr-terraform-live/globals.hcl in the c2_services_external_ips variable
  • Remove customer IPs from Moose SG whitelisting in xdr-terraform-live/prod/aws-us-gov/mdr-prod-c2/account.hcl in the splunk_data_sources variable
  • Remove customer from Portal Lambda customer_vars variable in xdr-terraform-live/prod/aws-us-gov/mdr-prod-c2/205-customer-portal-lambda/terragrunt.hcl
  • Delete the sensu entities and resolve any alerts
  • On the salt master, delete the salt minion keys sudo salt-key -d <CUSTOMER-PREFIX>*
  • On ScaleFT website, delete the project and servers

  • In the redhat website, remove the entitlements. Check for LCP nodes that used an entitlement

  • Commit the changes to the xdr-terraform-live repo and get merged into master

  • After changes have been merged in git, apply the changes to remove the IPs from the security groups and the AWS account from transit gateway

    • prod/aws-us-gov/mdr-prod-c2/275-nessus-security-managers
    • prod/aws-us-gov/mdr-prod-c2/205-customer-portal-lambda
    • prod/aws-us-gov/mdr-prod-c2/160-splunk-indexer-cluster
    • prod/aws-us-gov/mdr-prod-c2/095-instance-sensu
    • prod/aws-us-gov/mdr-prod-c2/080-instance-repo-server
    • prod/aws-us-gov/mdr-prod-c2/071-instance-salt-master
    • prod/aws-us-gov/mdr-prod-c2/008-transit-gateway-hub
    • prod/aws-us-gov/mdr-prod-c2/005-account-standards-c2

Remove the GovCloud and Commercial AWS account ID from Packer and Salt

https://github.xdr.accenturefederalcyber.com/mdr-engineering/msoc-infrastructure/wiki/Cloud-Accounts

  • Create new git branch in msoc_infrastructure
  • Remove Packer AWS accounts in packer/Makefile
  • Remove AWS accounts in salt/fileroots/salt_master/files/xdr_asset_inventory/xdr_asset_inventory.sh

Be sure to check for both Gov and Commerical AWS Accounts

Remove the Customer from the Salt Code

Remove references of the customer from these places:

  • Splunk Monitoring Console - salt/pillar/mc_variables.sls
  • Salt master configs in salt/fileroots/salt_master/files/etc/salt/master.d/default_acl.conf )
  • Delete Salt Splunk files - salt/pillar/${CUSTOMERPREFIX}_variables.sls salt/pillar/${CUSTOMERPREFIX}_pop_settings.sls
  • Salt top.sls and pillar/top.sls - salt/fileroots/top.sls - salt/pillar/top.sls
  • Salt okta auth in salt/pillar/os_settings.sls
  • Salt gitfs pillar in salt/pillar/salt_master.sls
  • Salt FM Shared Search in salt/pillar/fm_shared_search.sls

Apply changes in salt to remove references to the old customer.

Update salt master
salt salt* state.sls salt_master --output-diff test=true

Update the FM search head and monitoring console salt splunk-mc-0* state.sls splunk.monitoring_console --output-diff test=true salt fm-shared-search-0* state.sls splunk.fm_shared_search --output-diff test=true

Disable the instances in the Monitoring Console webpage ( how to delete the instances? ) Verify the search peers have been removed from the distributed search in the FM Shared Searchhead

Deactivate OKTA Apps

Each customer should have three applications. Deactive the app, then delete it. Splunk CM Splunk HF Splunk SH

Qualys Cleanup

Go to Qualys Dashboard -> Cloud Agent -> Activation Keys Disable the key, not sure how to delete it. Perhaps have to wait a period of time?

Archive Customer Git Repos

Do this after the Salt Master gitfs has been updated to avoid any error messages.

Git > Settings > Options > Archive this repository msoc--cm msoc--pop

Clean Up Vault Passwords

Delete engineering/customer_slices/ Disable onboarding-

Report the Decommissioned Hosts to the ISSO/AFCC Team

Look in the splunk inventory for the Splunk names or look for emails indicating the logs are not sending.

afcc@accenturefederal.com;asha.a.nair@accenturefederal.com
Accenture Federal Cyber Center <afcc@accenturefederal.com>; Nair, Asha A. <asha.a.nair@accenturefederal.com>

SUBJECT: Decommissioned XDR Devices

Hello,

The below instances have been decommissioned from the environment and should be removed from any reports or inventories. 

<list full splunk UF name of instances>

This lookup also needs to be edited. https://moose-splunk.pvt.xdr.accenturefederalcyber.com/en-US/app/SplunkEnterpriseSecuritySuite/ess_lookups_edit?namespace=SA-IdentityManagement&transform=simple_asset_lookup

Request AWS account be fully terminated

help desk ticket with camrs. Or have Soofi, Osman osman.soofi@accenturefederal.com submit a CAMRS disconnect ticket. not sure which one is the best method yet. IMPORTANT: After the account is closed, AWS allows users to login for 90 days.

AFS.Help <afs.help@accenturefederal.com>; XDR-Engineering <xdr.eng@accenturefederal.com>

SUBJECT: Decommission CAMRS AWS Account

Hello,

Please inform the CAMRS team that these AWS Accounts for <CUSTOMER-PREFIX> are no longer needed and can be decommissioned.

<AWS-ACCOUNT-ID-GOV>
<AWS-ACCOUNT-ID-COMMERCIAL>

Update the AWS Configuration

files/config in infrastructure-notes

Mark the AWS Account decommissioned in the WIKI once the email to help desk has been sent. We should keep the AWS Account numbers just in case they are needed in the future. https://github.xdr.accenturefederalcyber.com/mdr-engineering/msoc-infrastructure/wiki/Cloud-Accounts

Remove account from Terraform

IMPORTANT: After the account is closed, AWS allows users to login for 90 days. After AWS account has been decommissioned by CAMRS team, run terragrunt destroy in the 005-iam account to prevent users from assuming role into the account. Then remove the mdr-prod- folder from the xdr-terraform-live git repo.