# ScaleFT Notes.md OKTA owns ScaleFT (now Okta "Advanced Server Access") and we use it for managed SSH. [See Advanced Server Access on Okta Website](https://help.okta.com/asa/en-us/Content/Topics/Adv_Server_Access/docs/asa-overview.htm) ## Adding users to groups To add a user to a scaleFT group, just add them to the matching group in OKTA, ScaleFT will automagically query OKTA to pull the new user in via a service account. > :warning: This may take some time to propogate. If the groups never update, try updating the sftd agent `yum update scaleft-server-tools` Don't forget to add the user to the sft owner group manually if approppriate. This is for super admins. ## Client Setup (Mac) Install ScaleFT on your local machine. > :warning: Do NOT run these commands as root user. Choose your OS platform - [Install the Advanced Server Access client](https://help.okta.com/asa/en-us/Content/Topics/Adv_Server_Access/docs/sft.htm) Enroll the system from the cmd line as a new client using the `--team` switch and value "mdr" `sft enroll --team mdr` . A web page opens joining client to the Advanced Server Access platform. Ensure you are authenticated in [MDR Portal](https://mdr-multipass.okta.com) via Okta. SSH Setup - To configure the SSH client, run `sft ssh-config`. This command outputs an SSH configuration block. Append this block to your SSH configuration file (usually `~/.ssh/config`). > :warning: You can append the configuration to your file in one step by using this cmd `sft ssh-config >> $HOME/.ssh/config` Client customization - Any paths provided are from a MacOS perspective and use `/Users/Admin/` as an example folder path. Paths on your machine may read differently. Add `!User centos` as shown below in the `Match exec` line and edit your Username in `UserKnownHostsFile` line. `~/.ssh/config` for MacOS. Depending on your Advanced Server Access client and your SSH configuration, you should see something like the following within your config file: ``` Match exec "/usr/local/bin/sft resolve -q %h" !User centos ProxyCommand "/usr/local/bin/sft" proxycommand %h UserKnownHostsFile "/Users/INSERT_YOUR_USERNAME_HERE/Library/Application Support/ScaleFT/proxycommand_known_hosts" ``` List available servers `sft list-servers` Output from list of available servers: ``` HOSTNAME OS_TYPE PROJECT_NAME ID ACCESS_ADDRESS dev-afs-splunk-cm linux AFS 6b637c27-d885-44ea-9074-18cde8bfaa51 10.x.x.x ``` > :warning: VPN required - Ensure you are connected to the correct VPN (in this case, `XDR`) when attempting to SSH into a server. SSH into server from output using the `Id:` field in the cmd `ssh 6b637c27-d885-44ea-9074-18cde8bfaa51` or by hostname `ssh dev-afs-splunk-cm` If using a proxy, resolve proxy server (retrieve ID) `sft resolve proxy` Output from list of available proxy servers ``` Name: gc-dev-proxy Id: e1c10ac7-f152-45f4-9c42-ba6f30ffd2db Alt Names: proxy, proxy.pvt.xdrtest.accenturefederalcyber.com Project: Other_Infrastructure LastSeen: 13h38m0s ago ``` > :warning: VPN required - Ensure you are connected to the correct VPN (in this case, `XDRTest`) when attempting to SSH into a server. SSH into proxy server from output using the `Id:` field in the cmd `ssh e1c10ac7-f152-45f4-9c42-ba6f30ffd2db` or by hostname `ssh gc-dev-proxy` With the bastion `sft ssh gc-dev-salt-master --via gc-dev-bastion` ### SSH without sft Using the msoc_build Key The ssh key used when Packer builds the instance is called `msoc_build`. Because the servers are setup for FIPS mode, the `msoc_build` SSH key needs to be in "FIPS mode" before you use it. To bypass sft and use the `msoc_build` key use this command. `ssh -i msoc_build_fips centos@10.80.101.126` To use the key to ssh into hosts without the VPN use these commands. ( Agent Authentication forwarding ) First, add `msoc_build` key to your ssh agent `ssh-add msoc_build_fips` Then, SSH into bastion with `ssh -A centos@18.253.126.199` Finally, SSH into target server with `ssh centos@10.96.101.249` The key authentication will get passed through the proxy server and sent to the target host. ### Troubleshooting SFT Client Review the cache file: /var/lib/sftd/osync Make sure the agent is up-to-date. ERROR: x509: certificate signed by unknown authority" when trying to do things like "sft login" from the CLI This has to do with AFS Forcepoint proxy doing TLS interception. ScaleFT by default uses a compiled-in certificate list. Naturally, our man-in-the-middle isn't in their list. Solution: sft config network.tls_use_bundled_cas false This will make it use your local OS-specific cert stores. How you get the Forcepoint CA cert into your local OS-specific cert stores is another question for another day. ## How Do I SCP? SCP push a file works with scaleFT scp deleteme.txt dev-bastion:~/deleteme scp junk_index_new.tar.gz dev-bastion:~/junk_index_new.tar.gz SCP pull a file Duane's script. One major issue with this script is the sudo logging will scoop file contents into the sudoreplay logs. ``` #!/usr/bin/env bash #sftp_as2 afs-splunk-sh splunk REMOTE_HOST=$1 REMOTE_USER=$2 SFTP_SUBSYSTEM="/usr/libexec/openssh/sftp-server" function usage { echo "sftp_as afs-splunk-sh splunk" } if [[ $# -ne 2 ]]; then usage exit 1 fi sftp -s "sudo -i -u $REMOTE_USER $SFTP_SUBSYSTEM" $REMOTE_HOST ``` usage ./sftp_as2 dev-saf-splunk-indexer-1 brad_poulton ## Agent/Server Setup Salt pushes out token and agent then starts the agent. The agent connects to ScaleFT and updates the webpage. Reenroll the agent if they are not showing up in the scaleft.com website. systemctl restart sftd Install dir /etc/sft enrollment token (gets deleted after server is enrolled successfully) /etc/sft/enrollment.token remove the server's auth token to force them to reauth with scaleft.com (use this if you have deleted the server in the webpage) rm -rf /var/lib/sftd/device.token Configuration file cat /etc/sft/sftd.yaml Salt grain/pillar is used to determine if dev or prod salt '' state.sls os_modifications.scaleft Troubleshooting tail -200 /var/log/messages | grep sftd level=error msg="task init failed" err="Server is deleted" task=refreshServerToken remove device.token, place the enrollment.token and restart Temporarily change the name (salt state currently is not working on the name for dev-salt-master) Change the name and cannical name in sftd.yaml vim /etc/sft/sftd.yaml restart the service systemctl restart sftd ERROR: error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain ssh_exchange_identification: Connection closed by remote host SOLUTION: reenroll with ScaleFT, apply all updates, restart server. ## ScaleFT Projects servers belong to projects people / groups can be granted access to projects, which gives access to the related servers