Various Artists

Bastion.md

@@ -2,6 +2,6 @@
 
 ## SSH Through the Bastion with Keys
 
-The -A will forward the SSH key to the bastion 
+The -A will forward the SSH key to the bastion ( might not work with Zscaler )
 `ssh-add msoc_build_fips`
-`ssh -A centos@18.252.61.79`
+`ssh -A centos@18.253.126.199`

Proxy Notes.md

@@ -1,5 +1,8 @@
 # Proxy Notes
 
+## Restart the service
+`systemctl restart squid`
+
 ## Where are the logs?
 `cat /var/log/squid/access.log | grep 10.20.0.81`
 

Reposerver Notes.md

@@ -53,11 +53,14 @@ restorecon -R /var/www/html/redhat/
 #Oneliner
 chown -R apache:apache /var/www/html/redhat/msoc/Packages/ && cd /var/www/html/redhat/ && sudo -u apache createrepo msoc && restorecon -R /var/www/html/redhat/
 
+#Splunk Repo uses the version
+chown -R apache:apache /var/www/html/splunk/8.2/ && cd /var/www/html/splunk/ && sudo -u apache createrepo 8.2 && restorecon -R /var/www/html/splunk/8.2
 ```
 
 ```
 #From target server; clean out the cache
 yum clean all
+yum makecache fast
 
 #From target server; view the available packages
 yum --disablerepo="*" --enablerepo="msoc" list available
@@ -81,10 +84,9 @@ cd /var/www/html/splunk
 mkdir 7.2
 chown -R apache: .
 cd 7.2
-createrepo `pwd`
 wget -O splunk-7.2.5.1-962d9a8e1586-linux-2.6-x86_64.rpm 'https://www.splunk.com/page/download_track?file=7.2.5.1/linux/splunk-7.2.5.1-962d9a8e1586-linux-2.6-x86_64.rpm&ac=&wget=true&name=wget&platform=Linux&architecture=x86_64&version=7.2.5.1&product=splunk&typed=release'
 chown -R apache: .
 cd /var/www/html/splunk/7.2
-createrepo `pwd`
+sudo -u apache createrepo `pwd`
 restorecon -R /var/www/html/splunk
 ```

Splunk Notes.md

@@ -78,6 +78,9 @@ index=app_vault
 #Splunk
 | rest /services/data/indexes/
 | search title=app_mscas OR title = app_o365 OR title=dns OR title=forescout OR title=network OR title=security OR title=Te
+
+#are we freezing data off?
+index=_internal ayncfreezer freeze succeeded NOT ( _internaldb OR _introspection OR lastchance ) source=*splunkd.log
 ```
 
 ## coldToFrozenScript

Splunk Upgrade Notes.md

@@ -3,13 +3,18 @@
 User Calendar Apt to notify when you are upgrading Splunk. 
 
 ```
-Naughton, Brandon <brandon.naughton@accenturefederal.com>; Williams, Colby <colby.williams@accenturefederal.com>; Waddle, Duane E. <duane.e.waddle@accenturefederal.com>; Damstra, Frederick T. <frederick.t.damstra@accenturefederal.com>; Reuther, John M. <john.m.reuther@accenturefederal.com>; Leonard, Wesley A. <wesley.a.leonard@accenturefederal.com>; Starcher, George <george.a.starcher@accenturefederal.com>; Rivas, Gregory A. <gregory.a.rivas@accenturefederal.com>; Jarrett, James M. <james.m.jarrett@accenturefederal.com>
+Naughton, Brandon <brandon.naughton@accenturefederal.com>; Williams, Colby <colby.williams@accenturefederal.com>; Waddle, Duane E. <duane.e.waddle@accenturefederal.com>; Damstra, Frederick T. <frederick.t.damstra@accenturefederal.com>; Reuther, John M. <john.m.reuther@accenturefederal.com>; Leonard, Wesley A. <wesley.a.leonard@accenturefederal.com>; Starcher, George <george.a.starcher@accenturefederal.com>; Rivas, Gregory A. <gregory.a.rivas@accenturefederal.com>; Jarrett, James M. <james.m.jarrett@accenturefederal.com>; Kerr, James <j.kerr@accenturefederal.com>
 ```
-`This is an FYI only. I plan on upgrading PROD Splunk during this time. `
+`This is an FYI only. I plan on upgrading PROD Splunk during this time.`
 
 
 No need to notify the customer since this is a "behind the scences" change. No customer facing downtime.  
 
+Post to slack channels before you begin. xdr-patching, xdr-engineering, xdr-soc
+
+```
+Starting ma-c19 Splunk upgrade. please plan on outages. 
+```
 
 ## Splunk Upgrade 2021 "The Big One Part 2"
 09/27/2021
@@ -18,6 +23,7 @@ No need to notify the customer since this is a "behind the scences" change. No c
 - Download Splunk RPMs to reposerver and check hash ( follow Reposerver Notes to setup repo based on the version of Splunk )
     - wget -O splunk-8.2.2.1-ae6821b7c64b-linux-2.6-x86_64.rpm 'https://d7wz6hmoaavd0.cloudfront.net/products/splunk/releases/8.2.2.1/linux/splunk-8.2.2.1-ae6821b7c64b-linux-2.6-x86_64.rpm'
     - wget -O splunkforwarder-8.2.2.1-ae6821b7c64b-linux-2.6-x86_64.rpm 'https://d7wz6hmoaavd0.cloudfront.net/products/universalforwarder/releases/8.2.2.1/linux/splunkforwarder-8.2.2.1-ae6821b7c64b-linux-2.6-x86_64.rpm'
+    - wget -O splunk-8.2.3-cd0848707637-linux-2.6-x86_64.rpm 'https://download.splunk.com/products/splunk/releases/8.2.3/linux/splunk-8.2.3-cd0848707637-linux-2.6-x86_64.rpm?_ga=2.213141332.1340323660.1635200179-268321405.1634569782'
 - Modify pillar in modelclient_variables.sls to set modelclient to new version
     - Upgrade Test Modelclient
 - Review Apps in PROD Splunk and identify apps that are not compatable with new version 
@@ -34,6 +40,15 @@ No need to notify the customer since this is a "behind the scences" change. No c
 
 
 ### Upgrade Steps
+- Ensure recent and persistent snapshot of SH, HF, CM, etc. EBS Volumes
+    - Backup /opt/splunk `tar -cvzf /opt/splunk/opt-splunk-backup.tar.gz /opt/splunk`
+    - Stop Splunk service on CM,SH,HF,Cust-SH and take EBS snapshot of ALL the drives so that the snapshots will not be deleted in two days! 
+    - snapshot name: <hostname>-pre-upgrade-backup-<current-version>
+    - snapshot name: modelclient-splunk-hf-pre-upgrade-backup-8.0.5
+    - Update the profile, InstanceId, and tag to create snapshots of all volumes
+    ```
+    aws --profile mdr-test-c2-gov ec2 create-snapshots --instance-specification 'InstanceId=i-02a546c0de3d20030,ExcludeBootVolume=false' --tag-specifications 'ResourceType=snapshot,Tags=[{Key=Name,Value=modelclient-splunk-hf-pre-upgrade-backup-8.0.5}]'
+    ```
 - Before Splunk Upgrades
     - Upgrade ES 6.1.1/6.2.0 -> 6.6.2
         - version 6.6.2 is minimum version supported by 8.2.2.1
@@ -47,65 +62,37 @@ No need to notify the customer since this is a "behind the scences" change. No c
 - Update salt pillar data to new Splunk repo to reflect new splunk repo.
     - Dump all passwords from the password store PRIOR to upgrade. 
         - Run on the HF: `| rest /services/storage/passwords`
-- Stop all Indexers at the same time
-    - apply the updated pillar data `salt afs* saltutil.refresh_pillar`
-    - verify the pillar is updated `salt afs* pillar.item yumrepos:splunk`
-    - verify there is enough disk space
-    - stop Splunk on the indexers `cmd.run 'systemctl stop splunk'`
-- Ensure recent and persistent snapshot of SH, HF, CM, etc. EBS Volumes
-    - Backup /opt/splunk `tar -cvzf /opt/splunk/opt-splunk-backup.tar.gz /opt/splunk`
-    - Stop Splunk service on CM,SH,HF,Cust-SH and take EBS snapshot of ALL the drives so that the snapshots will not be deleted in two days! 
-    - snapshot name: <hostname>-pre-upgrade-backup-<current-version>
-    - snapshot name: modelclient-splunk-hf-pre-upgrade-backup-8.0.5
-    - Update the profile, InstanceId, and tag to create snapshots of all volumes
-    ```
-    aws --profile mdr-test-c2-gov ec2 create-snapshots --instance-specification 'InstanceId=i-02a546c0de3d20030,ExcludeBootVolume=false' --tag-specifications 'ResourceType=snapshot,Tags=[{Key=Name,Value=modelclient-splunk-hf-pre-upgrade-backup-8.0.5}]'
-    ```
-- Upgrade CM
+- Sensu and update the repo at the same time 
     - Setup silence on Sensu for ALL servers
-        - Run: `state.sls splunk.update_repo` to update repo
-        - Stop splunk `cmd.run 'systemctl stop splunk'`
-        - Upgrade splunk `pkg.upgrade name=splunk`
-            - Splunk is now waiting for accept license. Do Not Start Splunk Until after indexers are upgraded.
-- Upgrade All SH
-    - Setup silnce on Sensu
-        - Run: `state.sls splunk.update_repo` to update repo
-        - Stop splunk `cmd.run 'systemctl stop splunk'`
-        - Did you take an AWS EBS snapshot ???
+    - verify there is enough disk space `cmd.run 'df -h'`
+    - apply the updated pillar data `salt -C 'moose* not moose-alsi*' saltutil.refresh_pillar`
+    - verify the pillar is updated `salt -C 'moose* not moose-alsi*' pillar.item yumrepos:splunk`
+    - Run: `state.sls splunk.update_repo` to update repo
+- Stop all servers at the same time
+    - Stop Splunk on all servers `cmd.run 'systemctl stop splunk'`
+- Upgrade CM, SH, HF, customer SH ( if applicable )
+    - Did you take an AWS EBS snapshot ???
+    - Upgrade splunk `salt -C '( *cm* or *sh* or *hf* ) and moose*' pkg.upgrade name=splunk`
+    - Splunk is now waiting for accept license. Do Not Start Splunk Until after indexers are upgraded.
     - Swap George's app SA-AFS-XDR-Threat62 for SA-AFS-XDR-Threat64 on Search Heads with ES installed
+        - Install via the UI after extracting the app builder
         - rm –rf /opt/splunk/etc/apps/SA-AFS-XDR-Threat62
         - scp via teleport the new app 
-    - Upgrade splunk `pkg.upgrade name=splunk`
-        - Splunk is now waiting for accept license.
-- Upgrade Customer SH ( if applicable )
-    - Setup silence on Sensu
-        - Run: `state.sls splunk.update_repo` to update repo
-        - Stop splunk `cmd.run 'systemctl stop splunk'`
-        - Did you take an AWS EBS snapshot ???
-    - Upgrade splunk `pkg.upgrade name=splunk`
-        - Splunk is now waiting for accept license.
-- Upgrade Indexers
-    - Setup silence on Sensu
-        - Run: `state.sls splunk.update_repo` to update repo
-        - Stop splunk `cmd.run 'systemctl stop splunk'`
-        - Upgrade splunk `pkg.upgrade name=splunk`
-        - Start indexers and accept license `cmd.run 'systemctl start splunk'`
-            - `cmd.run '/opt/splunk/bin/splunk version'`
-            - `cmd.run '/opt/splunk/bin/splunk status'`
+        - besure to untar it twice!
+- Upgrade and Start Indexers 
+    - Upgrade splunk `salt -C '*idx* and moose*' pkg.upgrade name=splunk`
+    - Start indexers and accept license `cmd.run 'systemctl start splunk'`
+        - `cmd.run '/opt/splunk/bin/splunk version'`
+        - `cmd.run '/opt/splunk/bin/splunk status'`
 - Start CM and SH and Cust-SH
-    - Start CM/SH and accept license `cmd.run 'systemctl start splunk'`
+    - Start CM/SH/HF and accept license 
+        - `salt -C '( *cm* or *sh* or *hf* ) and moose*' cmd.run 'systemctl start splunk'`
 - Verify Splunk Web is up and searches of _internal index are working and three green checkmarks
-- Upgrade HF (slice only, not LCPs!)
-    - Run: `state.sls splunk.update_repo` to update repo
-    - Stop splunk `cmd.run 'systemctl stop splunk'`
-        - Did you take an AWS EBS snapshot ???
-    - Upgrade splunk `pkg.upgrade name=splunk`
-    - Start Splunk and accept license `cmd.run 'systemctl start splunk'`
+
 - Upgrade fm-shared-search-0/splunk-mc-0/qcompliance-splunk-sh
     - update pillar in...
         - salt/pillar/mc_variables.sls
         - salt/pillar/fm_shared_search.sls
-        - 
     - see above steps for upgrading a SH
 
 
@@ -113,10 +100,11 @@ No need to notify the customer since this is a "behind the scences" change. No c
     - Migrate KV store storage engine to WiredTiger on the SHs ( where the KV store is used. )
         - backup kvstore first! 
         - https://docs.splunk.com/Documentation/Splunk/8.2.2/Admin/BackupKVstore#Back_up_and_restore_the_KV_store_with_point_in_time_consistency 
+            - Verify backup is there in /opt/splunk/var/lib/splunk/kvstorebackup
         - https://docs.splunk.com/Documentation/Splunk/8.2.2/Admin/MigrateKVstore#Migrate_the_KV_store_after_an_upgrade_to_Splunk_Enterprise_8.1_or_higher_in_a_single-instance_deployment
     - upgrade apps slowly so Brandon can troubleshoot errors!!!!)
     - Ensure 3 green checkmarks (Prevents 3 green checkmarks on CM) Update the CM bundle to include `_cluster` see here: [Fixes for not replicating indexes?](https://github.xdr.accenturefederalcyber.com/mdr-engineering/msoc-afs-cm/pull/9)  (index _metrics and _introspection not in _cluster)
-- Delete Sensu Silences
+- Remove Sensu Silences
 - On AFS/FRTIB Cluster ensure that OKTA logs are coming in still 
 - Check lastchance index for unusual data. If the upgrade of ES introducing new indexes, and the new indexes are not on the Splunk indexers, then the data will be put into the lastchance index. 
 - Wait a week or two and delete the snapshots?
@@ -131,7 +119,29 @@ No need to notify the customer since this is a "behind the scences" change. No c
     - See which UFs have been upgraded. Splunk server UFs may get upgraded with a yum update. 
     `salt 'minion*' cmd.run '/opt/splunkforwarder/bin/splunk version'`
     `salt 'minion*' cmd.run '/opt/splunkforwarder/bin/splunk status'`
-- Upgrade Splunk/SplunkForwarder on LCPs
+
+
+### Upgrade Splunk on LCPs
+- Upgrade pillar in {customer}variables.sls
+    - `state.sls splunk.update_repo`
+    - `yum clean all ; yum makecache fast`
+- Create a backup
+    - Ensure you have room to take a backup
+        - `df -h /opt`
+    - Stop Splunk and take a backup
+        - `systemctl stop splunk`
+        - `cmd.run 'tar -czf /opt/opt-splunk-backup-8.0.5.tar.gz /opt/splunk'`
+        - `cmd.run 'ls -larth /opt'`
+    - Upgrade Splunk
+        - `pkg.upgrade name=splunk`
+        - `yum update splunk`
+        - `systemctl start splunk`
+        - `cmd.run 'systemctl start splunk'`
+        - `cmd.run 'tail /opt/splunk/var/log/splunk/splunkd.log'`
+
+
+
+
 
 
 

Splunk ma-c19 Offboarding Notes.md

@@ -0,0 +1,28 @@
+# Splunk MA-c19 Offboarding Notes
+
+Customer asked for test data prior to roll off. 
+
+Steps
+- tar up a bucket
+- test the bucket on a standalone splunk install
+    Standalone splunk
+    centos7
+    m5a.xlarge
+    vpc-038e00d0478425598
+    subnet-035fc7b980229db6a
+    xdr-indexer-instance-role
+    ma-c19-splunk-test-standalone
+    msoc-build
+- install a modern version of the aws cli. 
+    - `/usr/local/bin/aws --version`
+        - aws-cli/1.21.4 is too old. 
+    
+- set it to the customer via aws s3 presign url
+    - use the xdr-ma-c19-prod-splunk-frozen bucket
+        `/usr/local/bin/aws s3 cp azure_bucket.tar.gz s3://xdr-ma-c19-prod-splunk-frozen`
+    - Create presigned URL
+        `/usr/local/bin/aws s3 presign s3://xdr-ma-c19-prod-splunk-frozen/azure_bucket.tar.gz --expires-in 604800`
+
+
+
+

Tenable Notes.md

@@ -167,8 +167,6 @@ Use admin user to login ( shared cred in Vault )
 
 The agent key is generated and viewable in the Nessus Manager.
 
-Scans are run and then sent to SC. The Agent Synchronization Job on SC syncs the plugins and other data. The Agent scans syncs the scan data. Is this correct?
+Scans are run and then sent to SC. The Agent Synchronization Job on SC pulls the scans from the Nessus mananger. 
 
-In Nessus manager, the agent scans are scheduled. Agents are linked to the Nessus Manager through the Linking Key in the Nessus Manager. 
-
-Nessus Manager does not have a scan policy. All applicable plugins are used on the agents and the data is reported back to SC. 
+In Nessus manager, the agent scans are scheduled. Agents are linked to the Nessus Manager through the Linking Key in the Nessus Manager.  

Zscaler Notes.md

@@ -0,0 +1,5 @@
+# Zscaler Notes
+
+## XDR Exception Policy
+
+XDR has a custom Zscaler policy that allows access to *.accenturefederalcyber.com. To add new users to the policy, add them to the "XDR-ZScaler" Security Group here: https://directory.accenturefederal.com/.