infrastructure-notes/ThreatQ Notes.md

ThreatQ Notes

TQ HelpCenter access

https://helpcenter.threatq.com/welcome.htm

Support / Marketplace Access

To get a support account, email support@threatq.com from your accenture federal address and they will open an account.

TQ Passwords:

TQ HelpCenter: https://vault.pvt.xdr.accenturefederalcyber.com/ui/vault/secrets/engineering/show/threatq/helpcenter Admin Account: https://vault.pvt.xdr.accenturefederalcyber.com/ui/vault/secrets/engineering/show/threatq/admin MySQL root account: https://vault.pvt.xdr.accenturefederalcyber.com/ui/vault/secrets/engineering/show/threatq/mysql_admin

Operational things

Maintenance mode

TQ has "maintenance mode" where app won't let anyone log in.

To enable maintenance mode:

php /var/www/api/artisan down

Hardcore maintenance is to also stop apache httpd. Use systemctl.

To go out of maint:

php /var/www/api/artisan up

Compilation Notes

Packer things

We have a specific packer build for the TQ AMI. It's (mostly) unlike any other we have.

Setup Steps

Firstboot

First thing - make sure all the filesystems are grown. Somehow we miss this?

mount | awk '$5 == "xfs" { print $3 }' | xargs -n 1 xfs_growfs

When doing firstboot on our TQ AMI, there are some un-hardenings in place. TQ expects during firstboot that the root password is a known value, that apache has the rights to su to root using that password, and that selinux is permissive.

After the GUI panel during firstboot is filled out (setting a new root password, a mysql root password, and an initial admin account), then we can undo the un-hardenings. The firstboot GUI handles selinux.

IF YOU'RE DOING A FIRSTBOOT AS PART OF RESTORING A BACKUP FROM ANOTHER TQ SYSTEM, THEN PLEASE SKIP THIS PART AND COME BACK!

gpasswd -d apache sugroup
echo "root:*" | chpasswd -e
systemctl mask cockpit.service
systemctl mask cockpit.socket
systemctl stop cockpit

You might have to reboot after this, depending on what you're doing. If you're restoring TQ from backup, probably not.

Restoring a backup

GO THROUGH THE FIRSTBOOT STEPS VIA THE WEBUI, SET ADMIN AND MYSQL PASSWORDS AND INSTALL THE LICENSE

You have to go through firstboot first, but DO NOT DO the unhardening above yet!

Take the backup of the old TQ, make it available on the new one. Also save the contents of /etc/threatq/env/app-id.conf from the old TQ. It will look like

APP_INSTANCE_ID=18112b11-a13a-414b-93e8-b6b87adf27a5

Now to do the restore ... docs are at https://helpcenter.threatq.com/index.htm#t=ThreatQ_Platform%2FBackup_and_Restore%2FBackup_and_Restore.htm but there are couple of things not there.

Check to see if your mysql root password is right

mysql -u root -p

If you can connect to the DB good. If not reset the pw. There's some tomfoolery here I don't get yet. See bottom of these notes for how to reset the mysql PW.

$ sudo -i
# umask 0022
# cd /var/www/api
# php artisan threatq:restore </PATH/TO/BACKUP/FILE>
# php artisan threatq:update-events
# sed -i "s/^APP_INSTANCE_ID=.*/APP_INSTANCE_ID=18112b11-a13a-414b-93e8-b6b87adf27a5/" /etc/threatq/env/app-id.conf
# shutdown -r now

Now go back up and look at the un-hardening above and do that. You probably don't have to reboot twice.

Making a backup

We have a script /usr/local/bin/tqbackup that does the needful-ish. But if it's not there or if you need to specify some weird options (like --exclude-solr), you can run a backup directly. Why would you want to exclude solr? We've had problems with the backup working. If you restore a backup that lacks solr, then you'll need to reindex everything.

cd /var/www/api
php artisan threatq:backup --exclude-solr

The backup of solr goes to /tmp first so you'll need sufficient space there. If the backup seems to be hanging / taking forever, it might be because of insufficient space.

Reindexing

TQ uses Solr like elasticsearch. Most of your UI interactions use Solr, but the real data is persisted in the DB. Sometimes you'll need to reindex. What you'd think of as an index in elasticsearch is a "core" in Solr.

From TQ support to do a full reindex of everything:

php /var/www/api/artisan threatq:solr-import --full --all

This may take a few hours. The biggest cores by size are "indicators" and "objectlinks", and they will take the longest to reindex. Last reindex took 4ish hours. Folks can still use TQ while it's going, but a lot of things "won't show up"

Checking current reindex status

for i in indicators events objectlinks signatures attachments adversaries tasks campaign course_of_action exploit_target incident ttp attack_pattern identity intrusion_set malware report tool vulnerability; do
	echo "============ $i ============"
	php /var/www/api/artisan threatq:solr-import-status $i
done

Each "core" (table/index?) will dump something like:

Solr Dataimport

Status: busy
  Type: delta

Started At: 2022-08-19 22:28:02

Stats:
+-----------------------------------+---------------------+
| Time Elapsed                      | 0:23:23.485         |
| Total Requests made to DataSource | 2165017             |
| Total Rows Fetched                | 4806984             |
| Total Documents Processed         | 203189              |
| Total Documents Skipped           | 0                   |
| Delta Dump started                | 2022-08-19 22:28:02 |
| Identifying Delta                 | 2022-08-19 22:28:02 |
| Deltas Obtained                   | 2022-08-19 22:28:08 |
| Building documents                | 2022-08-19 22:28:08 |
| Total Changed Documents           | 1574988             |
+-----------------------------------+---------------------+

Patching Notes

TQ patching is a little different (of course). You have to be very careful about how you patch it because TQ provides a whole set of centos RPMs, and centos is trying very hard to infect our RHEL build with their RPMs.

Always read the TQ upgrade notes at https://helpcenter.threatq.com when you're upgrading TQ or when you're patching the base OS. They may change from time to time things like RPM excludes during updates.

Where TQ may do an explicit exclude of a package during an upgrade, I (Duane) will versionlock it instead. And, sometimes, other versionlocks are needed as well. As of now I am versionlocking the Java runtime (because TQ packages expect a SPECIFIC patch level of Java) and the redhat-rpm-config package so that it's not replaced by a centos package

ALWAYS do a yum check-update and make sure everything looks reasonable and that Centos packages aren't replacing their RHEL equivalents.

Some Common Problems

Mysql root password

During firstboot, our password we set doesn't "take". Connecting to the database becomes hard. Or any other situation where you need to reset the mysql password. Adapted from a Digital Ocean help article https://www.digitalocean.com/community/tutorials/how-to-reset-your-mysql-or-mariadb-root-password

cat <<EOF > /etc/systemd/system/mariadb.service.d/damnit.conf
[Service]
Environment="MYSQLD_OPTS=--skip-grant-tables --skip-networking"
EOF

systemctl daemon-reload
systemctl restart mariadb

ps -ef | grep mysql  # looking for the skip settings to exist

mysql -u root
# These are mysql commands
FLUSH PRIVILEGES;
ALTER USER 'root'@'localhost' IDENTIFIED BY '<SOMEPASSWORD>';
exit

# Back to shell
rm -f /etc/systemd/system/mariadb.service.d/damnit.conf
systemctl daemon-reload
systemctl restart mariadb
ps -ef | grep mysql  # looking for the skip settings to NOT exist

Weird things in sensu

I had to comment out the IPv6 "::1" /etc/hosts entry to shut up sensu and its https check.

I also had to chmod 755 /var/log/audit to shut up the disk space check for it.

I thought both of these were in salt but maybe not

I get a 502 error when accessing Server Administration

When you go to the gear -> server administration, you may get an error:

Service Unavailable
The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

This is intentional, and is caused by the disabling of the 'cockpit' service.

See the notes from '### Firstboot' above. The cockpit.service and socket have been masked.

saml authentication

• Users should be added to the appropriate okta threatq-* group. Their accounts will be automatically provisioned.
• New users will have to manually set their email address.
• The chicklet from okta is only a bookmark. Users must click the 'log in via sso' button to sign in.

The Azure Sentinel Integration

NOTE: The connector work in prod only. The connector will not function in test.

Documentation for the integration is available here

The salt state threatquotient.azure installs the two connectors (commercial and government) and initializes them. They then need to be configured via the web interface at Integrations->My Integrations, and search for 'Sentinel' (there should be two).

The salt state also creates teh cron job.

Testing the Sentinel Integration

You can run either connector manually:

For commercial:

/opt/tqvenv/com/bin/tq-conn-ms-sentinel -v3 -c /etc/tq_labs/com/ -ll /var/log/tq_labs/com/ --name "Microsoft Sentinel"

For Government:

/opt/tqvenv/us/bin/tq-conn-ms-sentinel -v3 -c /etc/tq_labs/us/ -ll /var/log/tq_labs/us/ --name "Microsoft Sentinel - Government"

(tip: if it asks for the log file location, you forgot sudo!)