Updates

New Customer Setup Notes - GovCloud.md

@@ -596,8 +596,28 @@ find and replace
 
 ## Got POP nodes? Ensure they are talking to Moose Splunk for Splunk UFs
 
-Got customer public IPs after you were done standing up the Splunk cluster? 
+Got customer public IPs after you were done standing up the Splunk cluster? This section is for you!
+
+Not sure on the Public IP? Check the VPC Flow logs. See any Cloudwatch REJECT logs?
+
+Ensure the eni is correct for PROD salt-master. Adjust src_ip for customer.
+ ```
+index=app_aws_flowlogs sourcetype="aws:cloudwatchlogs:vpcflow" vpcflow_action=REJECT eni-017d2e433b9f821d8 4506 src_ip=52.*
+|  timechart span=1d count by src_ip
+```
+
+```
+index=app_aws_flowlogs eni-017d2e433b9f821d8 dest_port IN (4505,4506) |  timechart count by src_ip
+```
+
+### Steps to allow LCP nodes through SG
+
 Add the IPs to account.hcl and reapply 160-splunk-indexer-cluster to add the customer IPs for the splunk environment. 
+`CUSTOMERPREFIX=modelclient`
+`cd xdr-terraform-live/prod/aws-us-gov/mdr-prod-$CUSTOMERPREFIX/`
+`vim account.hcl` # Look for splunk_data_sources
+`cd 160-splunk-indexer-cluster`
+`terragrunt-local plan`
 
 The IPs also need to be allowed for the salt-master, sensu, etc. 
 `vim xdr-terraform-live/globals.hcl`
@@ -610,30 +630,21 @@ terraform/02-msoc_vpc/security-groups.tf
 terraform/common/variables.tf
 and reapply 02-msoc_vpc. This should update salt master and repo. You can use --target, i won't tell on you. 
 
-See any Cloudwatch REJECT logs?
-
-Ensure the eni is correct for PROD salt-master. Adjust src_ip for customer.
- ```
-index=app_aws_flowlogs sourcetype="aws:cloudwatchlogs:vpcflow" vpcflow_action=REJECT eni-017d2e433b9f821d8 4506 src_ip=52.*
-|  timechart span=1d count by src_ip
-```
-
-```
-index=app_aws_flowlogs eni-017d2e433b9f821d8 dest_port IN (4505,4506) |  timechart count by src_ip
-```
 
 
 ## Is there going to be POP/LCP nodes? 
 
 These commands will add the pop settings pillar
 
+Go to Qualys Dashboard -> Cloud Agent -> Activation Keys -> New Key
+Title name scheme: $CUSTOMERPREFIX-lcp-nodes 
+Provision Key for Vuln Management and Policy compliance. 
+Create and add a new tag to the activation key with a title called $CUSTOMERPREFIX with parent tag, CustomerPOP. Don't add any Tag Rules.  ( Use the create link )
 
-
-TODO: Add steps to add Qualys LCP settings in pop_settings file. But Brad, I don't know how to get the qulys customer id and activation key...me either!
+Copy an existing ${CUSTOMERPREFIX}_pop_settings.sls and rename it. Put the activation key in pillar/$CUSTOMERPREFIX_pop_settings.sls. The qualys_customer_id is the same for all customers. 
 
 CUSTOMERPREFIX=modelclient
 
-1. copy an existing ${CUSTOMERPREFIX}_pop_settings.sls and rename it. 
 
 1. add LCP nodes to the pillar top file
 cd salt/pillar
@@ -651,25 +662,27 @@ Commit all the changes to git and open PR. Once the settings are in the master b
 ```
 CUSTOMERPREFIX=modelclient
 salt -C "${CUSTOMERPREFIX}* and G@msoc_pop:True" test.ping
-#are the LCP images up-to-date on the salt minion version? See Salt Upgrade Notes.md. Upgrade salt minions before syncing ec2_tags it needs py3. Make sure the environment grain is set before trying to upgrade salt. 
+#are the LCP images up-to-date on the salt minion version? See Salt Upgrade Notes.md. Make sure the environment grain is set before trying to upgrade salt. 
 salt -C "${CUSTOMERPREFIX}* and G@msoc_pop:True" test.version
 salt -C "${CUSTOMERPREFIX}* and G@msoc_pop:True" saltutil.sync_all
 salt -C "${CUSTOMERPREFIX}* and G@msoc_pop:True" saltutil.refresh_pillar
 salt -C "${CUSTOMERPREFIX}* and G@msoc_pop:True" saltutil.refresh_modules
+#did the customer set the roles correctly?
+salt -C "${CUSTOMERPREFIX}* and G@msoc_pop:True" cmd.run 'cat /etc/salt/minion.d/minion_role_grains.conf'
 #ensure the msoc_pop grain is working properly and set to True
 salt -C "${CUSTOMERPREFIX}* and G@msoc_pop:True" grains.get msoc_pop
 #ensure the ec2:billing_products grain is EMPTY unless node is in AWS. ( Do we get the RH subscription from AWS? Not for LCP nodes )
 salt -C "${CUSTOMERPREFIX}* and G@msoc_pop:True" grains.get ec2:billing_products
 #ensure the environment grain is available and set to prod
-salt -C "${CUSTOMERPREFIX}* and G@msoc_pop:True" grains.get environment
+salt -C "${CUSTOMERPREFIX}* and G@msoc_pop:True" grains.get environment ( not needed if in AWS?)
 #make sure the activation-key pillar is available
 salt -C "${CUSTOMERPREFIX}* and G@msoc_pop:True" pillar.get os_settings:rhel:rh_subscription:activation-key
-#LCP nodes need manual RH Subscription enrollment before removing test=true ensure the command is filled out with the pillar
+#LCP nodes need manual RH Subscription enrollment before removing test=true ensure the command is filled out with the pillar, unless they are in AWS?
 salt -C "${CUSTOMERPREFIX}* and G@msoc_pop:True" state.sls os_modifications.rhel_registration test=true
+# try out the os_modifications then try high state
 salt -C "${CUSTOMERPREFIX}* and G@msoc_pop:True" state.sls os_modifications
 ```
 
-salt ${CUSTOMERPREFIX}-splunk
 Start with ds
 salt ${CUSTOMERPREFIX}-splunk-ds\* state.highstate --output-diff
 
@@ -678,7 +691,7 @@ salt ${CUSTOMERPREFIX}-splunk-syslog-\* state.sls os_modifications
 
 ## Configure the Customer POP Git Repository 
 
-Add DS ServerClass.conf and Apps 
+Add DS ServerClass.conf and Apps
 
 1. Add the passwd to the Customer DS git repo.  
 
@@ -688,12 +701,12 @@ echo $DSADMINPASS
 DSADMINHASH="`echo $DSADMINPASS | python3 -c "from passlib.hash import sha512_crypt; print(sha512_crypt.hash(input(), rounds=5000))"`"
 echo $DSADMINHASH
 ```
-Store the DSADMINPASS in Vault in the engineering/customer_slices/$CUSTOMERPREFIX secret. Create new version with key called frtib-splunk-ds admin
+Store the DSADMINPASS in Vault in the engineering/customer_slices/$CUSTOMERPREFIX secret. Create new version with key called "$CUSTOMERPREFIX-splunk-ds admin".
 
 On laptop
 `cat ../msoc-infrastructure/salt/pillar/${CUSTOMERPREFIX}_variables.sls | grep minion_pass | cut -d "\"" -f 2  | python3 -c "from passlib.hash import sha512_crypt; print(sha512_crypt.hash(input(), rounds=5000))"`
 
-Put these values in the passwd file in the Customer DS git repo (msoc-CUSTOMERREFIX-pop) in the root directory. 
+Put these values in the passwd file in the Customer DS git repo (msoc-$CUSTOMERREFIX-pop) in the root directory. 
 
 1. Add the appropriate apps to the Customer DS git repo (msoc-CUSTOMERPREFIX-pop). Double check with Duane/Brandon to ensure correct apps are pushed to the DS! The minimum apps are $CUSTOMERPREFIX_hf_outputs, xdr_pop_minion_authorize, xdr_pop_ds_summaries.
 

Patching Notes.md

@@ -9,7 +9,7 @@
 
 It might be a good idea to run:
 ```
-salt '*' state.sls os_modifications.rhel_deregistration
+salt '*' state.sls os_modifications.rhel_deregistration --output-diff
 ```
 It's safe to run on `*` and will remove any RHEL registration (or warnings about lack thereof) on systems that have a billing code.
 
@@ -517,6 +517,16 @@ watch "salt -C '*syslog-1* not *.local' test.ping"
 
 If the syslog-ng service doesn't start, check the syslog-ng file for oms agent added configurations. 
 
+##### Possible issue: NGA LCP nodes hostnames change after reboot and Sensu agent name changes. 
+
+```
+salt 'nga-splunk-ds-1' cmd.run 'hostnamectl set-hostname aws-splnks1-tts.nga.gov'
+salt 'nga-splunk-ds-1' cmd.run 'hostnamectl status'
+salt 'nga-splunk-ds-1' cmd.run 'systemctl stop sensu-agent'
+salt 'nga-splunk-ds-1' cmd.run 'systemctl start sensu-agent'
+```
+Repeat for other LCP nodes
+
 #### Verify logs are flowing
 
 https://afs-splunk-sh.msoc.defpoint.local:8000/en-US/app/search/search

Portal Notes.md

@@ -123,13 +123,21 @@ ALLOWED_HOSTS = [
     ]
 ```
 
+python3 manage.py shell
+from user_portal.models import User
+
+User.objects.filter(email="richard.t.page@accenturefederal.com").delete()
+User.objects.filter(email="wesley.a.leonard@accenturefederal.com").delete()
+
 
 ## Troubleshooting the docker image
 
 
 ### Logs
 
-logs are being sent to Splunk
+logs are being sent to Splunk. This logging is configured in Salt. 
+
+`index=os source="http*" host="customer-portal-*"`
 
 Nginx
 /var/log/nginx/portal.access.log

Salt Upgrade 3001.6 -> 3002.6 Notes.md

@@ -0,0 +1,39 @@
+Salt Upgrade 3001.6 -> 3002.6 Notes.md
+
+
+upgrade salt master then minions
+
+Update the pillar in git
+```
+salt salt* cmd.run 'salt-run fileserver.update'
+salt salt* cmd.run 'salt-run git_pillar.update'
+salt salt* saltutil.refresh_pillar
+salt salt* pillar.get yumrepos:salt:version
+```
+
+Update salt master 
+```
+salt salt* cmd.run 'cat /etc/yum.repos.d/salt.repo'
+salt salt* state.sls os_modifications.repo_update test=true --output-diff
+salt salt* cmd.run 'cat /etc/yum.repos.d/salt.repo'
+salt salt* cmd.run 'yum clean all ; yum makecache fast'
+salt salt* cmd.run 'yum check-update | grep salt'
+salt salt* pkg.upgrade name=salt-master
+sudo salt salt* state.sls salt_master.salt_posix_acl --output-diff
+```
+
+Update salt minions
+```
+salt sensu* cmd.run 'cat /etc/yum.repos.d/salt.repo'
+salt sensu* state.sls os_modifications.repo_update test=true --output-diff
+salt sensu* cmd.run 'cat /etc/yum.repos.d/salt.repo'
+salt sensu* cmd.run 'yum clean all ; yum makecache fast'
+salt sensu* cmd.run 'yum check-update | grep salt'
+salt sensu* cmd.run_bg 'systemd-run --scope yum update salt-minion -y && sleep 20 && systemctl daemon-reload && sleep 20 && systemctl start salt-minion'
+salt sensu* test.version
+```
+
+Did you miss any?
+`salt -G saltversion:3001.6 test.ping`
+
+repeat for PROD.