Additional Tweaks

New Customer Setup Notes - GovCloud.md

@@ -787,11 +787,7 @@ cd 160-splunk-indexer-cluster
 terragrunt-local plan
 ```
 
-LEGACY NOT NEEDED...
-For Legacy, update these files
-terraform/02-msoc_vpc/security-groups.tf
-terraform/common/variables.tf
-and reapply 02-msoc_vpc. This should update salt master and repo. You can use --target, i won't tell on you. 
+
 
 
 ## Is there going to be LCP nodes? 
@@ -838,10 +834,9 @@ salt -C "${CUSTOMERPREFIX}* and G@msoc_pop:True" saltutil.refresh_modules
 #did the customer set the roles correctly?
 salt -C "${CUSTOMERPREFIX}* and G@msoc_pop:True" cmd.run 'cat /etc/salt/minion.d/minion_role_grains.conf'
 salt -C "${CUSTOMERPREFIX}* and G@msoc_pop:True" grains.get customer
+#No customer grain? Look in the LCP troubleshooting section below. 
 #ensure the ec2:billing_products grain is EMPTY unless node is in AWS. ( Do we get the RH subscription from AWS? Not for LCP nodes )
 salt -C "${CUSTOMERPREFIX}* and G@msoc_pop:True" grains.get ec2:billing_products
-#ensure the environment grain is available and set to prod
-salt -C "${CUSTOMERPREFIX}* and G@msoc_pop:True" grains.get environment ( not needed on LCP nodes?)
 #make sure the activation-key pillar is available ( VMware Only )
 salt -C "${CUSTOMERPREFIX}* and G@msoc_pop:True" pillar.get os_settings:rhel:rh_subscription:activation-key
 #VMware LCP nodes need manual RH Subscription enrollment before removing test=true ensure the command is filled out with the pillar, unless they are in AWS.
@@ -854,7 +849,7 @@ Run the highstate, unless you want to configure the syslog-ng drives first.
 Start with ds
 salt ${CUSTOMERPREFIX}-splunk-ds\* state.highstate --output-diff
 
-Finish with syslog servers ( You should see syslog-ng errors )
+Next with syslog servers ( You should see syslog-ng errors due to no syslog-ng partition )
 salt ${CUSTOMERPREFIX}-splunk-syslog-\* state.highstate --output-diff
 
 ## Add 100GB Drive to Syslog Servers
@@ -862,12 +857,13 @@ salt ${CUSTOMERPREFIX}-splunk-syslog-\* state.highstate --output-diff
 Manually setup the 100 GB drives for the two syslog servers. 
 
 ```
+# CUSTOMERLOC is for customers that have multiple LCP clusters and is optional. 
 CUSTOMERLOC=<customer location grain>
 #Alternate: salt -C "${CUSTOMERPREFIX}*syslog* and G@location:${CUSTOMERLOC}"
 #find the drive name
 salt ${CUSTOMERPREFIX}-splunk-syslog-\* cmd.run 'lsblk | grep 100G'
 salt ${CUSTOMERPREFIX}-splunk-syslog-\* cmd.run 'df -hT'
-CUSTOMERDRIVE=<drivename>
+CUSTOMERDRIVE=<drivename> # should look like nvme3n1
 salt ${CUSTOMERPREFIX}-splunk-syslog-\* cmd.run "lsblk /dev/${CUSTOMERDRIVE}"
 salt ${CUSTOMERPREFIX}-splunk-syslog-\* lvm.pvcreate /dev/${CUSTOMERDRIVE}
 salt ${CUSTOMERPREFIX}-splunk-syslog-\* lvm.vgcreate vg_syslog /dev/${CUSTOMERDRIVE}
@@ -875,7 +871,7 @@ salt ${CUSTOMERPREFIX}-splunk-syslog-\* lvm.lvcreate lv_syslog vg_syslog extents
 salt ${CUSTOMERPREFIX}-splunk-syslog-\* cmd.run 'mkfs -t xfs /dev/vg_syslog/lv_syslog'
 salt ${CUSTOMERPREFIX}-splunk-syslog-\* partition.list /dev/${CUSTOMERDRIVE}
 ```
-Add the syslog state to the highstate for the customer and apply highstate. 
+Add the syslog state to the highstate for the customer and apply highstate. syslog state will mount the drive and add to fstab.
 
 ## Configure the Customer LCP Git Repository 
 
@@ -912,7 +908,7 @@ echo $MINIONPASS  | python3 -c "from passlib.hash import sha512_crypt; print(sha
 1. Add the appropriate apps to the Customer DS git repo (msoc-CUSTOMERPREFIX-pop). Double check with Duane/Brandon to ensure correct apps are pushed to the DS! The minimum apps are cust_hf_outputs, xdr_pop_minion_authorize, xdr_pop_ds_summaries.
 
 update the cust_hf_outputs app ( command specific for MAC OS )
-`sed -i '' -e 's/CUSTOMER/'"${CUSTOMERPREFIX}"'/g' deployment-apps/cust_hf_outputs/local/outputs.conf`
+`sed -i '' -e 's/SYSLOG_SERVER/'"${CUSTOMERPREFIX}-splunk-indexers.xdr.accenturefederalcyber.com"'/g' deployment-apps/cust_hf_outputs/local/outputs.conf`
 
 Commit the changes to the git repo. 
 ```
@@ -938,10 +934,13 @@ salt -C "${CUSTOMERPREFIX}* and G@msoc_pop:True" system.reboot
 ## Verify Splunk Connectivity
 Can you see the DS logs in the customer slice splunk? If you don't have a DNS address for the deployement server you will not see the LCP syslog node logs. 
 
+Find out the hostnames for the LCP nodes. 
+`salt -C "${CUSTOMERPREFIX}* and G@msoc_pop:True" cmd.run 'hostname'`
+
 Customer Slice
 `index=_internal NOT host="*.pvt.xdr.accenturefederalcyber.com" source="/opt/splunk/var/log/splunk/splunkd.log" earliest=-1h`
 
-Moose
+Moose, Did you remember to add the LCP IPs to the Moose SG? 
 `index=_internal earliest=-1h host=<host-from-previous-command>`
 
 
@@ -950,21 +949,20 @@ Moose
 TO: "XDR-Feed-Management" <xdr.feed.management@accenturefederal.com>
 CC: "XDR-Engineering" <xdr.eng@accenturefederal.com>
 
-SUBJECT: ${CUSTOMERPREFIX} LCP Servers Ready
+SUBJECT: CUSTOMERPREFIX LCP Servers Ready
 
 ```
 Hello,
 
-This is notification that the ${CUSTOMERPREFIX} LCP servers are ready for Feed Management to configure for customer use. 
+This is notification that the CUSTOMERPREFIX LCP servers are ready for Feed Management to configure for customer use. 
 
 Successfully Completed Tasks
 - Salt highstate completed successfully
 - Splunk and syslog-ng installed successfully
 - Servers fully patched and rebooted successfully
-- Servers sending logs to Splunk customer slice successfully
+- Customer DS sending logs to Splunk customer slice successfully
 - Servers sending logs to Splunk Moose successfully
 - Servers connecting to Sensu successfully
-- Servers setup for vulnerability and compliance scanning
 
 ```
 
@@ -973,13 +971,10 @@ Successfully Completed Tasks
 
 REMEMBER: Our Customers are responsible for setting up the salt minion with grains and allow traffic through the outbound firewall. If they have not done that yet, you will get more errors. 
 
-ISSUE: Help, the environment grain is not showing up! ( is this needed? see minion_role_grains.conf lifecycle)
-SOLUTION: This command will add a static grain in /etc/salt/minion.d/cloud_init_grains.conf. 
-`salt 'target' state.sls salt_minion.salt_grains pillar='{"environment": "prod"}' test=true --output-diff` 
-`cmd.run 'rm -rf /var/cache/salt/minion/extmods/grains/ec2_tags.py'`
+ISSUE: Help, the customer grain is not showing up! This means the customer did not do their job!
+SOLUTION: This command will add a static grain in /etc/salt/minion.d/minion_role_grains.conf. Update variables accordingly. 
+`salt 'target' state.sls salt_minion.salt_grains_lcp pillar='{"customer": "afs", "location": "Ashburn", "lifecycle": "production"}' --output-diff test=true` 
 
-Then restart the minion with `service.restart salt-minion`
-then pillar.refresh
 
 ISSUE: Deployment Server is not running the reload_ds state file correctly and the error, "Client is not authorized to perform requested action" is showing up. 
 SOLUTION: ensure the splunk minion user has the correct splunk role assigned in the passwd file. 

POP-LCP Node Notes.md

@@ -14,10 +14,6 @@ SDC drive
  3017  03/20/20 19:42:51 +0000 sudo lvcreate --help
  3018  03/20/20 19:43:23 +0000 salt 'afs*syslog-[5678]*' cmd.run 'lvcreate -L 500G -n lv_syslog vg_syslog'
  3019  03/20/20 19:43:32 +0000 salt 'afs*syslog-[5678]*' cmd.run 'lvcreate -L 499G -n lv_syslog vg_syslog'
- 3020  03/20/20 19:44:18 +0000 salt 'afs*syslog-[5678]*' cmd.run 'mkfs -t ext4 /dev/vg_syslog/lv_syslog'
-1:46
-sorry needs to be xfs
-1:46
 salt 'afs*syslog-[5678]*' cmd.run 'mkfs -t xfs -f  /dev/vg_syslog/lv_syslog'