infrastructure-notes/New Customer Setup Notes - Legacy.md

New Customer Setup Notes - LEGACY

Legacy Information - For Reference Only

This is in no specific order YET

Git Repos to be made

Shameful but I just add via github UI.

CM repo

Normally named msoc-<customer>-cm. For collaborators/teams:

* infrastructure - Admin
* automation - Read
* onboarding - Write

Then I clone and copy from an existing customer. We need a template-type-thing for this. If you copy another customer remove any HEC tokens

Update the admin password in the CM repo

Duane has some terrible things he uses for this. A script plus a docker image. Outputs something like

[dwaddle@DPS0591 msoc]$ gen-splunk-passwd.sh

PASSWORD:  `v9!T^\6RVU6C^BF%34c
HASH    :  $6$Ly1tn9ZbftJ$CzTH0uQI5bAfTHBnSK8hZsRwmMg/qrSW1tZg3zEcqwJK2cGoRiPkrRvNM1Kj/dUO2bwp87eBdPpwY/aWB2zht/

Once you have the plaintext, put it in vault. When you have the ciphertext put it in the passwd file that is in the repo.

POP repo

Normally named msoc-<customer>-pop. For collaborators/teams:

* infrastructure - Admin
* automation - Read
* onboarding - Write

Then I clone and copy from an existing customer. We need a template-type-thing for this

Okta apps

Make the okta apps for CM, SH, HF. You'll need the App entityId and SignonURL for pillars

• In Okta, Applications, do 'Add' Pick 'Splunk Enterprise'
• Application Name: xxx-Splunk-CM
• Site Url: https://xxx-splunk-cm.msoc.defpoint.local:8000

Under "Sign On" tab click 'Edit'

• Role : matches regex splunk\-role\-.*
• Entity ID: I got it from "View Setup Instructions"

Updated Okta crap

Use Duane's script to make the Okta apps like so:

[dwaddle@DPS0591 okta_app_maker(feature/MSOCI-1266)]$ ./okta_app_maker.py 'LA-C19 Splunk CM' 'https://la-c19-splunk-cm.msoc.defpoint.local:8000'
2020-05-12 17:10:20,215 DEBUG main len=3 args=['./okta_app_maker.py', 'LA-C19 Splunk CM', 'https://la-c19-splunk-cm.msoc.defpoint.local:8000']
2020-05-12 17:10:20,216 INFO main Creating app name="LA-C19 Splunk CM" for login url="https://la-c19-splunk-cm.msoc.defpoint.local:8000/saml/acs"
2020-05-12 17:10:20,216 INFO handle_app Checking for existence of app="LA-C19 Splunk CM"
2020-05-12 17:10:20,235 DEBUG _new_conn Starting new HTTPS connection (1): mdr-multipass.okta.com
2020-05-12 17:10:21,083 DEBUG _make_request https://mdr-multipass.okta.com:443 "GET /api/v1/apps?q=LA-C19+Splunk+CM HTTP/1.1" 200 None
2020-05-12 17:10:21,089 DEBUG handle_app Response code 200
2020-05-12 17:10:21,092 DEBUG _new_conn Starting new HTTPS connection (1): mdr-multipass.okta.com
2020-05-12 17:10:22,780 DEBUG _make_request https://mdr-multipass.okta.com:443 "POST /api/v1/apps HTTP/1.1" 200 None
2020-05-12 17:10:22,787 DEBUG create_saml_app Response code: 200
2020-05-12 17:10:22,787 DEBUG handle_app Getting the IDP metadata for app="LA-C19 Splunk CM"
2020-05-12 17:10:22,791 DEBUG _new_conn Starting new HTTPS connection (1): mdr-multipass.okta.com
2020-05-12 17:10:23,629 DEBUG _make_request https://mdr-multipass.okta.com:443 "GET /api/v1/apps/0oa3o3rmh6Ak4Ramo297/sso/saml/metadata HTTP/1.1" 200 2339
2020-05-12 17:10:23,635 DEBUG get_things_from_metadata Response code 200
2020-05-12 17:10:23,649 DEBUG handle_app entityId=http://www.okta.com/exk3o3rmh5x90JIhg297   ssoUrl=https://mdr-multipass.okta.com/app/mdrmultipass_lac19splunkcm_1/exk3o3rmh5x90JIhg297/sso/saml
2020-05-12 17:10:23,650 INFO update_app Updating app_id="0oa3o3rmh6Ak4Ramo297"
2020-05-12 17:10:23,654 DEBUG _new_conn Starting new HTTPS connection (1): mdr-multipass.okta.com
2020-05-12 17:10:24,815 DEBUG _make_request https://mdr-multipass.okta.com:443 "PUT /api/v1/apps/0oa3o3rmh6Ak4Ramo297 HTTP/1.1" 200 None
2020-05-12 17:10:24,821 DEBUG update_app Response code 200

MDR pillar format:
--------------------------------------------------------
{% if grains['id'].startswith('<REPLACEME>') %}
auth_method: "saml"
okta:
  # This is the entityId / IssuerId
  uid: "http://www.okta.com/exk3o3rmh5x90JIhg297"
  # Login URL / Signon URL
  login: "https://mdr-multipass.okta.com/app/mdrmultipass_lac19splunkcm_1/exk3o3rmh5x90JIhg297/sso/saml"
{% endif %}

Save the results of this into a temp file for stuffing into Pillars. Run it for the CM, SH, and each HF. This does not assign the apps to groups (yet!)

Scaleft

• Add the "project"
• Assign groups to the project
  • mdr-admins: admin / sync group yes
  • mdr-engineers: user / sync group no
• Make an enrollment token
• Put enrollment token in the pillar -- os_settings.sls has an ugly jinja if/else

Salt

• Make a new splunk pillar variables file - copy an existing one
  • create a new pass4symmkey (use like a guid generator or something). it goes in cleartext for now
• You'll need the scaleft token and the okta pillars from above
• Add to the default_acl.conf so that Salt cmds are available to people
• Install the license file in msoc-infrastructure/salt/fileroots/splunk/files/licenses
• Update pillar/salt-master.sls to include the new customer repo(s). On most of these the trailing : matters.
• Add the new pillars to pillar/top.sls
• Add the new minions to fileroots/top.sls This may not be needed?

The road so far

Everything up to this point is pre-setup / staging. Now you need to merge your changes into develop, and kick the tires on it.

1. sudo salt-run fileserver.update
2. salt 'salt-master*' state.apply --state-verbose=False We're updating the master configs so it knows the new hotness
3. sudo systemctl restart salt-master
4. sudo salt-run fileserver.file_list Here we are checking to make sure our new gitfs got picked up. Should see the new files from the new repo(s)

Terraform

• Make the new customer project in terraform, copy an existing one
  • cp ../106-ma-c19/common.tf .
  • cp ../106-ma-c19/customer.tf .
  • ln -s ../common/variables.tf .
  • ln -s ../common/amis.tf .

• update terraform files

  • change terraform block in common.tf for new S3 state bucket and new proj_id variable
  • change customer.tf especially the name in the locals block
  • change variables.tf to add the CIDR. Use the next logical /22

• terraform init

• terraform workspace new test

• terraform apply

Back to salt

• Fiddly bits here
  • salt 'la-c19*' saltutil.sync_all
  • salt 'la-c19*' saltutil.refresh_grains
  • salt 'la-c19*' saltutil.refresh_pillar
  • salt 'la-c19*' state.apply --state-verbose=False
  • Be prepared to repeat these a bunch of times till you get a mostly clean highstate
• Patch, reboot
• Push the master bundle salt 'la-c19-splunk-cm*' state.sls splunk.master.apply_bundle_master

Sanity Checks

1. UI on the cluster master look sane?
2. Can you SAML into everything?
3. Look on the indexers via CLI does everthing look believable

Some helpful sanity check searches

| rest /services/data/indexes splunk_server=*indexer* 
| stats values(homePath_expanded) as home, values(coldPath_expanded) as cold, values(tstatsHomePath_expanded) as tstats by title 
| sort home

Splunk configuration

• Install ES on the search head

Monitoring Console

• Add cluster to monitoring console
• Peer with CM, SH, and HF
• Update MC topology

Create New Vault Engine for Customer for Feed Management

Naming Scheme: onboarding- Example: onboarding-la-covid

Keep George Happy and push out maxmind

salt -C '*splunk-indexer* or *splunk-sh* or *splunk-hf*' state.sls splunk.maxmind.pusher --state-verbose=False --state-output=terse

Ensure the environment grain is working properly

salt <customer-name>* grains.get environment

If the grain is not there, follow troubleshooting steps in Salt Upgrade Notes.md

Got POP nodes? Ensure they are talking to Moose Splunk for Splunk UFs