# ScaleFT Notes.md

OKTA owns ScaleFT (now Okta "Advanced Server Access") and we use it for managed SSH. [See Advanced Server Access on Okta Website](https://help.okta.com/asa/en-us/Content/Topics/Adv_Server_Access/docs/asa-overview.htm)


## Adding users to groups

To add a user to a scaleFT group, just add them to the matching group in OKTA, ScaleFT will automagically query OKTA to pull the new user in via a service account. 

> :warning: This may take some time to propogate. 

If the groups never update, try updating the sftd agent `yum update scaleft-server-tools`

Don't forget to add the user to the sft owner group manually if approppriate. This is for super admins. 

## Client Setup (Mac)

Install ScaleFT on your local machine.  

> :warning: Do NOT run these commands as root user. 

Choose your OS platform - [Install the Advanced Server Access client](https://help.okta.com/asa/en-us/Content/Topics/Adv_Server_Access/docs/sft.htm)

Enroll the system from the cmd line as a new client using the `--team` switch and value "mdr" `sft enroll --team mdr` . A web page opens joining client to the Advanced Server Access platform. Ensure you are authenticated in [MDR Portal](https://mdr-multipass.okta.com) via Okta. 

SSH Setup - To configure the SSH client, run `sft ssh-config`. This command outputs an SSH configuration block. Append this block to your SSH configuration file (usually `~/.ssh/config`). 

> :warning: You can append the configuration to your file in one step by using this cmd `sft ssh-config >> $HOME/.ssh/config`

Client customization - Any paths provided are from a MacOS perspective and use `/Users/Admin/` as an example folder path. Paths on your machine may read differently.

Add `!User centos` as shown below in the `Match exec` line and edit your Username in `UserKnownHostsFile` line. `~/.ssh/config` for MacOS. Depending on your Advanced Server Access client and your SSH configuration, you should see something like the following within your config file:
```
Match exec "/usr/local/bin/sft resolve -q  %h" !User centos
    ProxyCommand "/usr/local/bin/sft" proxycommand  %h
    UserKnownHostsFile "/Users/INSERT_YOUR_USERNAME_HERE/Library/Application Support/ScaleFT/proxycommand_known_hosts"
```

List available servers `sft list-servers`

Output from list of available servers:
```
HOSTNAME                      OS_TYPE    PROJECT_NAME            ID                                      ACCESS_ADDRESS
dev-afs-splunk-cm             linux      AFS                     6b637c27-d885-44ea-9074-18cde8bfaa51    10.x.x.x
```

> :warning: VPN required - Ensure you are connected to the correct VPN (in this case, `XDR`) when attempting to SSH into a server. SSH into server from output using the `Id:` field in the cmd `ssh 6b637c27-d885-44ea-9074-18cde8bfaa51` or by hostname `ssh dev-afs-splunk-cm`

If using a proxy, resolve proxy server (retrieve ID) `sft resolve proxy`

Output from list of available proxy servers
```
Name: 	gc-dev-proxy
		Id: 		e1c10ac7-f152-45f4-9c42-ba6f30ffd2db
		Alt Names:      proxy, proxy.pvt.xdrtest.accenturefederalcyber.com
		Project: 	Other_Infrastructure
		LastSeen: 	13h38m0s ago
```

> :warning: VPN required - Ensure you are connected to the correct VPN (in this case, `XDRTest`) when attempting to SSH into a server. SSH into proxy server from output using the `Id:` field in the cmd `ssh e1c10ac7-f152-45f4-9c42-ba6f30ffd2db` or by hostname `ssh gc-dev-proxy`

With the bastion            
`sft ssh gc-dev-salt-master --via gc-dev-bastion`           




###  SSH without sft Using the msoc_build Key
The ssh key used when Packer builds the instance is called `msoc_build`. Because the servers are setup for FIPS mode, the `msoc_build` SSH key needs to be in "FIPS mode" before you use it. 

To bypass sft and use the `msoc_build` key use this command.

`ssh -i msoc_build_fips centos@10.80.101.126`

To use the key to ssh into hosts without the VPN use these commands. ( Agent Authentication forwarding )

First, add `msoc_build` key to your ssh agent `ssh-add msoc_build_fips`
Then, SSH into bastion with `ssh -A centos@18.253.126.199`
Finally, SSH into target server with `ssh centos@10.96.101.249`
The key authentication will get passed through the proxy server and sent to the target host.  


### Troubleshooting SFT Client

Review the cache file: /var/lib/sftd/osync
Make sure the agent is up-to-date. 

ERROR: x509: certificate signed by unknown authority" when trying to do things like "sft login" from the CLI

This has to do with AFS Forcepoint proxy doing TLS interception.  ScaleFT by default uses a compiled-in certificate list.  Naturally, our man-in-the-middle isn't in their list.

Solution: sft config network.tls_use_bundled_cas false

This will make it use your local OS-specific cert stores.  How you get the Forcepoint CA cert into your local OS-specific cert stores is another question for another day.

## How Do I SCP?

SCP push a file works with scaleFT
scp deleteme.txt dev-bastion:~/deleteme
scp junk_index_new.tar.gz dev-bastion:~/junk_index_new.tar.gz

SCP pull a file

Duane's script.  One major issue with this script is the sudo logging will
scoop file contents into the sudoreplay logs.

```
#!/usr/bin/env bash
#sftp_as2 afs-splunk-sh splunk
REMOTE_HOST=$1
REMOTE_USER=$2
SFTP_SUBSYSTEM="/usr/libexec/openssh/sftp-server"
function usage {
echo "sftp_as afs-splunk-sh splunk"
}
if [[ $# -ne 2 ]]; then
        usage
        exit 1
fi
sftp -s "sudo -i -u $REMOTE_USER $SFTP_SUBSYSTEM" $REMOTE_HOST
```

usage

./sftp_as2 dev-saf-splunk-indexer-1 brad_poulton

## Agent/Server Setup

Salt pushes out token and agent then starts the agent. The agent connects to ScaleFT and updates the webpage. 

Reenroll the agent if they are not showing up in the scaleft.com website. 
systemctl restart sftd

Install dir
/etc/sft

enrollment token (gets deleted after server is enrolled successfully)
/etc/sft/enrollment.token

remove the server's auth token to force them to reauth with scaleft.com (use this if you have deleted the server in the webpage)
rm -rf /var/lib/sftd/device.token

Configuration file
cat /etc/sft/sftd.yaml

Salt grain/pillar is used to determine if dev or prod

salt '' state.sls os_modifications.scaleft

Troubleshooting 
tail -200 /var/log/messages | grep sftd
level=error msg="task init failed" err="Server is deleted" task=refreshServerToken
remove device.token, place the enrollment.token and restart

Temporarily change the name (salt state currently is not working on the name for dev-salt-master)
Change the name and cannical name in sftd.yaml
vim /etc/sft/sftd.yaml

restart the service
systemctl restart sftd

ERROR: error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
ssh_exchange_identification: Connection closed by remote host

SOLUTION: reenroll with ScaleFT, apply all updates, restart server. 

## ScaleFT Projects 

servers belong to projects
people / groups can be granted access to projects, which gives access to the related servers
