infrastructure-notes/ScaleFT Notes.md

ScaleFT Notes.md

OKTA owns ScaleFT (now Okta "Advanced Server Access") and we use it for managed SSH. See Advanced Server Access on Okta Website

Adding users to groups

To add a user to a scaleFT group, just add them to the matching group in OKTA, ScaleFT will automagically query OKTA to pull the new user in via a service account.

:warning: This may take some time to propogate.

If the groups never update, try updating the sftd agent yum update scaleft-server-tools

Don't forget to add the user to the sft owner group manually if approppriate. This is for super admins.

Client Setup (Mac)

Install ScaleFT on your local machine.

:warning: Do NOT run these commands as root user.

Choose your OS platform - Install the Advanced Server Access client

Enroll the system from the cmd line as a new client using the --team switch and value "mdr" sft enroll --team mdr . A web page opens joining client to the Advanced Server Access platform. Ensure you are authenticated in MDR Portal via Okta.

SSH Setup - To configure the SSH client, run sft ssh-config. This command outputs an SSH configuration block. Append this block to your SSH configuration file (usually ~/.ssh/config).

:warning: You can append the configuration to your file in one step by using this cmd sft ssh-config >> $HOME/.ssh/config

Client customization - Any paths provided are from a MacOS perspective and use /Users/Admin/ as an example folder path. Paths on your machine may read differently.

Add !User centos as shown below in the Match exec line and edit your Username in UserKnownHostsFile line. ~/.ssh/config for MacOS. Depending on your Advanced Server Access client and your SSH configuration, you should see something like the following within your config file:

Match exec "/usr/local/bin/sft resolve -q  %h" !User centos
    ProxyCommand "/usr/local/bin/sft" proxycommand  %h
    UserKnownHostsFile "/Users/INSERT_YOUR_USERNAME_HERE/Library/Application Support/ScaleFT/proxycommand_known_hosts"

List available servers sft list-servers

Output from list of available servers:

HOSTNAME                      OS_TYPE    PROJECT_NAME            ID                                      ACCESS_ADDRESS
dev-afs-splunk-cm             linux      AFS                     6b637c27-d885-44ea-9074-18cde8bfaa51    10.x.x.x

:warning: VPN required - Ensure you are connected to the correct VPN (in this case, XDR) when attempting to SSH into a server. SSH into server from output using the Id: field in the cmd ssh 6b637c27-d885-44ea-9074-18cde8bfaa51 or by hostname ssh dev-afs-splunk-cm

If using a proxy, resolve proxy server (retrieve ID) sft resolve proxy

Output from list of available proxy servers

Name: 	gc-dev-proxy
		Id: 		e1c10ac7-f152-45f4-9c42-ba6f30ffd2db
		Alt Names:      proxy, proxy.pvt.xdrtest.accenturefederalcyber.com
		Project: 	Other_Infrastructure
		LastSeen: 	13h38m0s ago

:warning: VPN required - Ensure you are connected to the correct VPN (in this case, XDRTest) when attempting to SSH into a server. SSH into proxy server from output using the Id: field in the cmd ssh e1c10ac7-f152-45f4-9c42-ba6f30ffd2db or by hostname ssh gc-dev-proxy

With the bastion
sft ssh gc-dev-salt-master --via gc-dev-bastion

SSH without sft Using the msoc_build Key

The ssh key used when Packer builds the instance is called msoc_build. Because the servers are setup for FIPS mode, the msoc_build SSH key needs to be in "FIPS mode" before you use it.

To bypass sft and use the msoc_build key use this command.

ssh -i msoc_build_fips centos@10.80.101.126

To use the key to ssh into hosts without the VPN use these commands. ( Agent Authentication forwarding )

First, add msoc_build key to your ssh agent ssh-add msoc_build_fips Then, SSH into bastion with ssh -A centos@18.253.126.199 Finally, SSH into target server with ssh centos@10.96.101.249 The key authentication will get passed through the proxy server and sent to the target host.

Troubleshooting SFT Client

Review the cache file: /var/lib/sftd/osync Make sure the agent is up-to-date.

ERROR: x509: certificate signed by unknown authority" when trying to do things like "sft login" from the CLI

This has to do with AFS Forcepoint proxy doing TLS interception. ScaleFT by default uses a compiled-in certificate list. Naturally, our man-in-the-middle isn't in their list.

Solution: sft config network.tls_use_bundled_cas false

This will make it use your local OS-specific cert stores. How you get the Forcepoint CA cert into your local OS-specific cert stores is another question for another day.

How Do I SCP?

SCP push a file works with scaleFT scp deleteme.txt dev-bastion:~/deleteme scp junk_index_new.tar.gz dev-bastion:~/junk_index_new.tar.gz

SCP pull a file

Duane's script. One major issue with this script is the sudo logging will scoop file contents into the sudoreplay logs.

#!/usr/bin/env bash
#sftp_as2 afs-splunk-sh splunk
REMOTE_HOST=$1
REMOTE_USER=$2
SFTP_SUBSYSTEM="/usr/libexec/openssh/sftp-server"
function usage {
echo "sftp_as afs-splunk-sh splunk"
}
if [[ $# -ne 2 ]]; then
        usage
        exit 1
fi
sftp -s "sudo -i -u $REMOTE_USER $SFTP_SUBSYSTEM" $REMOTE_HOST

usage

./sftp_as2 dev-saf-splunk-indexer-1 brad_poulton

Agent/Server Setup

Salt pushes out token and agent then starts the agent. The agent connects to ScaleFT and updates the webpage.

Reenroll the agent if they are not showing up in the scaleft.com website. systemctl restart sftd

Install dir /etc/sft

enrollment token (gets deleted after server is enrolled successfully) /etc/sft/enrollment.token

remove the server's auth token to force them to reauth with scaleft.com (use this if you have deleted the server in the webpage) rm -rf /var/lib/sftd/device.token

Configuration file cat /etc/sft/sftd.yaml

Salt grain/pillar is used to determine if dev or prod

salt '' state.sls os_modifications.scaleft

Troubleshooting tail -200 /var/log/messages | grep sftd level=error msg="task init failed" err="Server is deleted" task=refreshServerToken remove device.token, place the enrollment.token and restart

Temporarily change the name (salt state currently is not working on the name for dev-salt-master) Change the name and cannical name in sftd.yaml vim /etc/sft/sftd.yaml

restart the service systemctl restart sftd

ERROR: error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain ssh_exchange_identification: Connection closed by remote host

SOLUTION: reenroll with ScaleFT, apply all updates, restart server.

ScaleFT Projects

servers belong to projects people / groups can be granted access to projects, which gives access to the related servers