# VMRay Notes

## Quick Reference

Prod: https://vmray.pvt.xdr.accenturefederalcyber.com
Test: https://vmray.pvt.xdrtest.accenturefederalcyber.com

iso's are stored in `/opt/vmray/iso/`. I downloaded 3:
  * `Win10_21H1_English_x32.iso`
  * `Win10_21H1_English_x64.iso`
  * `Win11_English_x64.iso`

VPN Required. SAML signon enabled.

NOTICE: New users must be manually created in VMRay before the SAML account will work properly! Email address must match between the VMRay account and the SAML account. 

- Disable password
- Role Administrator 
- Uncheck create API key


# Summary
VMRay Analyzer is a tool to detonate malware in a controlled environment.

## Generalized Architecture

VMRay Analyzer consists of a VMRay Server, which coordinates the use of other systems, and one or more _bare metal_ worker machines on which malware detonates. The systems run Ubuntu 20.04 LTS.

The system is deployed in its own account in GovCloud (one for prod, one for test)

Test does not have a license, so will remain shutdown/disabled/unconfigured most of the time, and with 0 workers.

# Source

documentation and downloads are at: https://portal.vmray.com/customer/login?

### Install Log

Stand up via terraform.
Apply the highstate, probably 2x to get 0 errors.
Run 'pkg.upgrade'
Run 'system.reboot'

Login to web page with username and password: `admin@example.com` (same un and password)
Add the license (license can be grabbed from https://portal.vmray.com/customer/login?) and restart.
Navigate to user settings, update admin email address to xdr.eng@accenturefederal.com and update the password.
Configure the system

Under worker, click 'create worker'
You'll have to provide the IP address, unfortunately.

On the worker:
```
cd /opt/vmray
sudo aws --region us-gov-east-1 s3 sync s3://afsxdr-binaries/iso iso
sudo chown -R vmray:vmray iso
sudo chmod 755 iso
sudo find iso -type f -exec chmod 644 {} \;
sudo find iso -type d -exec chmod 755 {} \;
```

Set up SSO

Install a VM

### Installing a VM

** Recommendation: Run this in a screen or tmux session so that you can reconnect if the vpn disconnects you **

1. log into the _worker_
2. `cd /opt/vmray/bin/`
3. `sudo -u vmray ./vm_setup.py`
4. Option 2 - VM Operations
5. Option 1 - Create a new VM
NOTE: No spaces in the name
... follow instructions. Most defaults are fine. See page 48 of the OnPrem installation guide for an example
6. Option 3 - Guest OS Operations
7. Option 1 - Install OS
8. Wait for the message `VM started. You can now connect via VNC (port :0)`. You can monitor the installation by using a VNC viewer to connect. VNC Ports start at 5900, so :0 is port 5900, :1 would be 5901, and so forth.
9. After it returns to the main menu, select option 4, prepare VM for analysis, then option 2. 
10. Defaults are probalby fine. You must specify the proxy as `proxy.pvt.xdr.accenturefederalcyber.com:80`
11. After initialization completes, select option 4, prepare vm for analysis, and then option 3, create VM Snapshot. (NOTE: The documents specify that the snapshot is _not_ what we'd think of as a snapshot (i.e. a point in time image), but is something else... and you probably only need one named 'def' for each VM.)

Take 2:
Steps 1-5 above
Then:
6. Select option 3 "Maintain Windows VM", select 'y' multiple times, run periodic maintenance, * for all of the above.





# Troubleshooting

## 2022-03-01: Had error in detection-update.log:
```
requests.exceptions.SSLError: HTTPSConnectionPool(host='download.vmray.com', port=443): Max retries exceeded with url: /repository/platform-updates/yara/4.4/index.json (Caused by SSLError('Fingerprints did not match. Expected "fcb64419c025ddf06042e2461d30171c17627edc9bfefed277789f501ffb3d52", got "b\'ea8f4b0b6a3519f10343195473d6cf0a63f652a7242fc768c502e091cd57e198\'".'))
```

Fix: 
Support sent an updated replacement file “communication_lib.so”.

# Upgrading

To upgrade VMRay: You should be able to safely run this at any time (still, follow best practices: make backups first, run Salt with `test=true`).
0) Make sure the system is fully patched. Run `salt vmray\* pkg.upgrade` if in doubt.
1) Download the latest from https://portal.vmray.com/customer/login? . Download both the platform and the extended-kvm packages.
2) Upload into the S3 afs-xdr-binaries bucket. `aws --profile mdr-common-services-gov s3 cp vmray-4.6.1.zip s3://afsxdr-binaries/VMRay/`
3) Update the version and the hashes in `msoc-infrastructure/salt/fileroots/vmray/init.sls`
4) Run and Test (initially) `salt vmray\* state.sls vmray test=true --output-diff` from the salt master.
5) On both server and worker:
  - Run `find /opt/vmray -type f -exec chmod g+r,o+r {} \;`
  - Run `find /opt/vmray -type d -exec chmod g+rx,o+rx {} \;`
  - Run `chown -R vmray:vmray /opt/vmray`
  - Run `find /opt/vmray -perm 0777 -type f -exec chmod 755 {} \;`
6) log into the _worker_
7) `cd /opt/vmray/bin/`
8) `sudo -u vmray ./vm_setup.py`
9) Choose option 6: "Update VMRay Tools"


## Handling Failed Upgrades
If the upgrade fails, check `/usr/local/src/vmray-x.y.z/install.log` to find the reason. After fixing, delete the tarballs, zips, and directories for vmray and extended-kvm from `/usr/local/src/` on the vmray servers, and then rerun the state.

## Permissions after upgrade

Sometimes after upgrades we get an error. A triangle with an exclamation point will display in the new analysis. If you hover over this symbol, an error message will appear, such as "Post processing error Error during post processing: report_engine_result.json not found (possible crash).". This has usually been due to permissions issues.

To fix, run from the salt master:
```
salt vmray\* cmd.run 'find /opt/vmray -type f -exec chmod g+r,o+r {} \;; find /opt/vmray -type d -exec chmod g+rx,o+rx {} \;; chown -R vmray:vmray /opt/vmray; find /opt/vmray -perm 0777 -type f -exec chmod 755 {} \;'
```

## Backups

A script `/usr/local/bin/xdr-vmray-backups.sh` backs the system up to a bucket named `xdr-<ENV>-vmray-backups`. The bucket has folders for the server name. For server nodes, the script uses the VMRay distribution's backup script. For workers and combined servers, ISO's are backed up to /iso and vm images are backed up into the vms subfolder.

To restore:

For the server, run the `/opt/vmray/bin/restore_server.sh` script. See the admin documentation for more details.

For the worker: 
1. copy the ISOs into /opt/vmray/iso
2. Extract the vms into /opt/vmray/img/vms/


## Restoring from Backup

Build from terraform, reboot, hightstate 2x, reboot again

On the worker:
```
sudo systemctl stop vmray-worker
sudo aws --region us-gov-east-1 s3 sync s3://xdr-prod-vmray-backups/vmray-worker-0/iso /opt/vmray/iso
sudo chown -R vmray:vmray /opt/vmray/iso
sudo chmod 755 /opt/vmray/iso
aws --region us-gov-east-1 s3 ls s3://xdr-prod-vmray-backups/vmray-worker-0/
sudo aws --region us-gov-east-1 s3 sync s3://xdr-prod-vmray-backups/vmray-worker-0/<PASTE LATEST> /opt/vmray/restore
cd /opt/vmray/img/vms
for i in $( sudo find /opt/vmray/restore/vms/ ); do
  sudo tar xvzf $i
done
sudo chown -R vmray:vmray /opt/vmray/img
```

On the server:
```
sudo systemctl stop vmray-worker
sudo systemctl stop vmray-server
aws --region us-gov-east-1 s3 ls s3://xdr-prod-vmray-backups/vmray-server/
sudo aws --region us-gov-east-1 s3 sync s3://xdr-prod-vmray-backups/vmray-server/<PASTE LATEST "vmray_server*"> /opt/vmray/restore/server/
sudo aws --region us-gov-east-1 s3 sync s3://xdr-prod-vmray-backups/vmray-server/<PASTE LATEST "vmray_worker*"> /opt/vmray/restore/worker/
sudo chown -R vmray:vmray /opt/vmray/restore
cd /opt/vmray
sudo /opt/vmray/bin/restore_server.sh /opt/vmray/restore/server
sudo /opt/vmray/bin/restore_server.sh /opt/vmray/restore/worker
sudo find . -type f -exec chmod g+r,o+r {} \;
sudo find . -type d -exec chmod g+rx,o+rx {} \;
sudo reboot
```

On the worker
```
sudo reboot
```

## Log of activities after restore

After restoring, things still weren't working. Just keeping a log of thigns I tried.

On server:
```
sudo systemctl stop vmray-worker
sudo systemctl stop vmray-server
cd /opt/vmray/log
sudo rm -f *
sudo systemctl start vmray-server
ls -l
# Reviewed all logs, nothing too notable. uwsgi-ui.log has a lot of writing to a closed pipe, but I think that's the alb.
sudo systemctl start vmray-worker
# Reviewed logs again

# Ran analysis of pageant.exe as static only.
"Post processing error Error during post processing: report_engine_result.json not found (possible crash)."
```

# Additional Troubleshooting

In addition to `/opt/vmray/log` on both the worker and the server, check:
* /opt/vmray/var/analyses - Will contain a zip file with a log of the analysis
* Generate a support package and download it through the server.

Verify permissions. The umask caused issues once, and had to be fixed via:
```
sudo find /opt/vmray -type f -exec chmod g+r,o+r {} \;
sudo find /opt/vmray -type d -exec chmod g+rx,o+rx {} \;
```