Threatq

ThreatQ Notes.md

@@ -1,5 +1,38 @@
 # ThreatQ Notes
 
+## TQ HelpCenter access
+
+https://helpcenter.threatq.com/welcome.htm
+
+## TQ Passwords:
+
+TQ HelpCenter: https://vault.pvt.xdr.accenturefederalcyber.com/ui/vault/secrets/engineering/show/threatq/helpcenter
+Admin Account: https://vault.pvt.xdr.accenturefederalcyber.com/ui/vault/secrets/engineering/show/threatq/admin
+MySQL root account: https://vault.pvt.xdr.accenturefederalcyber.com/ui/vault/secrets/engineering/show/threatq/mysql_admin
+
+
+## Operational things
+
+### Maintenance mode
+
+
+TQ has "maintenance mode" where app won't let anyone log in.
+
+To enable maintenance mode:
+
+```
+php /var/www/api/artisan down
+```
+
+Hardcore maintenance is to also stop apache httpd.  Use systemctl.
+
+To go out of maint:
+
+```
+php /var/www/api/artisan up
+```
+
+
 ## Compilation Notes
 
 ### Packer things
@@ -10,6 +43,12 @@ We have a specific packer build for the TQ AMI.  It's (mostly) unlike any other
 
 ### Firstboot
 
+First thing - make sure all the filesystems are grown.  Somehow we miss this?
+
+```
+mount | awk '$5 == "xfs" { print $3 }' | xargs -n 1 xfs_growfs
+```
+
 When doing firstboot on our TQ AMI, there are some un-hardenings in place.
 TQ expects during firstboot that the root password is a known value, that
 apache has the rights to `su` to root using that password, and that
@@ -17,23 +56,27 @@ selinux is permissive.
 
 After the GUI panel during firstboot is filled out (setting a new root
 password, a mysql root password, and an initial admin account), then we can
-undo the un-hardenings.
+undo the un-hardenings.  The firstboot GUI handles selinux.
 
-If you're doing a firstboot as part of restoring a backup from another TQ
-system, then please skip this part and come back!
+IF YOU'RE DOING A FIRSTBOOT AS PART OF RESTORING A BACKUP FROM ANOTHER TQ
+SYSTEM, THEN PLEASE SKIP THIS PART AND COME BACK!
 
 ```
 gpasswd -d apache sugroup
 echo "root:*" | chpasswd -e
 systemctl mask cockpit.service
 systemctl mask cockpit.socket
-shutdown -r now
+systemctl stop cockpit
 ```
 
-Now hopefully you can enable a salt minion without breaking anything
+You might have to reboot after this, depending on what you're doing.  If you're
+restoring TQ from backup, probably not.
 
 ### Restoring a backup
 
+GO THROUGH THE FIRSTBOOT STEPS VIA THE WEBUI, SET ADMIN AND MYSQL PASSWORDS
+AND INSTALL THE LICENSE
+
 You have to go through firstboot first, but DO NOT DO the unhardening
 above yet!
 
@@ -48,6 +91,15 @@ APP_INSTANCE_ID=18112b11-a13a-414b-93e8-b6b87adf27a5
 Now to do the restore ... docs are at https://helpcenter.threatq.com/index.htm#t=ThreatQ_Platform%2FBackup_and_Restore%2FBackup_and_Restore.htm
 but there are couple of things not there.
 
+Check to see if your mysql root password is right
+
+```
+mysql -u root -p
+```
+
+If you can connect to the DB good.  If not reset the pw.  There's some tomfoolery here I don't get yet.
+See bottom of these notes for how to reset the mysql PW.
+
 ```
 $ sudo -i
 # umask 0022
@@ -62,6 +114,71 @@ Now go back up and look at the un-hardening above and do that.  You probably don
 have to reboot twice.
 
 
+## Making a backup
+
+We have a script /usr/local/bin/tqbackup that does the needful-ish.  But if it's not
+there or if you need to specify some weird options (like --exclude-solr),
+you can run a backup directly.  Why would you want to exclude solr?  We've had problems
+with the backup working.  If you restore a backup that lacks solr, then you'll need to
+reindex everything.
+
+```
+cd /var/www/api
+php artisan threatq:backup --exclude-solr
+```
+
+## Reindexing
+
+TQ uses Solr like elasticsearch.  Most of your UI interactions use Solr, but the
+real data is persisted in the DB.  Sometimes you'll need to reindex.  What you'd
+think of as an index in elasticsearch is a "core" in Solr.
+
+From TQ support to do a full reindex of everything:
+
+```
+php /var/www/api/artisan threatq:solr-import --full --all
+```
+
+This may take a few hours.  The biggest cores by size are "indicators" and
+"objectlinks", and they will take the longest to reindex.  Last reindex took
+4ish hours.  Folks can still use TQ while it's going, but a lot of things
+"won't show up"
+
+### Checking current reindex status
+
+```
+for i in indicators events objectlinks signatures attachments adversaries tasks campaign course_of_action exploit_target incident ttp attack_pattern identity intrusion_set malware report tool vulnerability; do
+	echo "============ $i ============"
+	php /var/www/api/artisan threatq:solr-import-status $i
+done
+```
+
+Each "core" (table/index?) will dump something like:
+
+```
+Solr Dataimport
+
+Status: busy
+  Type: delta
+
+Started At: 2022-08-19 22:28:02
+
+Stats:
++-----------------------------------+---------------------+
+| Time Elapsed                      | 0:23:23.485         |
+| Total Requests made to DataSource | 2165017             |
+| Total Rows Fetched                | 4806984             |
+| Total Documents Processed         | 203189              |
+| Total Documents Skipped           | 0                   |
+| Delta Dump started                | 2022-08-19 22:28:02 |
+| Identifying Delta                 | 2022-08-19 22:28:02 |
+| Deltas Obtained                   | 2022-08-19 22:28:08 |
+| Building documents                | 2022-08-19 22:28:08 |
+| Total Changed Documents           | 1574988             |
++-----------------------------------+---------------------+
+```
+
+
 ## Patching Notes
 
 TQ patching is a little different (of course).  You have to be very careful about
@@ -80,3 +197,45 @@ package
 
 ALWAYS do a `yum check-update` and make sure everything looks reasonable and that
 Centos packages aren't replacing their RHEL equivalents.
+
+
+## Some Common Problems
+
+### Mysql root password
+
+During firstboot, our password we set doesn't "take".  Connecting to the database becomes hard.
+Or any other situation where you need to reset the mysql password.  Adapted from a Digital Ocean
+help article https://www.digitalocean.com/community/tutorials/how-to-reset-your-mysql-or-mariadb-root-password
+
+```
+cat <<EOF > /etc/systemd/system/mariadb.service.d/damnit.conf
+[Service]
+Environment="MYSQLD_OPTS=--skip-grant-tables --skip-networking"
+EOF
+
+systemctl daemon-reload
+systemctl restart mariadb
+
+ps -ef | grep mysql  # looking for the skip settings to exist
+
+mysql -u root
+# These are mysql commands
+FLUSH PRIVILEGES;
+ALTER USER 'root'@'localhost' IDENTIFIED BY '<SOMEPASSWORD>';
+exit
+
+# Back to shell
+rm -f /etc/systemd/system/mariadb.service.d/damnit.conf
+systemctl daemon-reload
+systemctl restart mariadb
+ps -ef | grep mysql  # looking for the skip settings to NOT exist
+
+```
+
+### Weird things in sensu
+
+I had to comment out the IPv6 "::1" /etc/hosts entry to shut up sensu and its https check.
+
+I also had to `chmod 755 /var/log/audit` to shut up the disk space check for it.
+
+I thought both of these were in salt but maybe not