infrastructure-notes/Portal Notes.md

Portal Notes

XDR Portal Notes link in our Github

Portal is a custom application running on Django app in docker.

Help! Portal is down!

Start with a docker container restart

salt 'customer-portal*' cmd.run 'docker container ls'
salt 'customer-portal*' cmd.run 'docker restart portal nginx'

Verify Vault service is running. Logs are sent to Splunk. See Splunk search below in this document.

Command line access

docker exec -ti nginx bash
docker exec -ti portal bash

# root user careful!
docker exec -u root -ti portal bash

salt 'customer-portal*' cmd.run 'docker restart portal'
salt 'customer-portal*' cmd.run 'docker rm -f portal'
salt 'customer-portal*' cmd.run 'docker rm -f nginx'
salt 'customer-portal*' cmd.run 'docker pull 350838957895.dkr.ecr.us-east-1.amazonaws.com/portal_server'
salt 'customer-portal*' cmd.run 'docker pull 350838957895.dkr.ecr.us-east-1.amazonaws.com/django_nginx'
salt 'customer-portal*' cmd.run 'docker image ls'

salt 'customer-portal*' state.sls docker.portal

Docker entrypoint.sh & Varibles

If the app.env is empty or has null values in it, then the docker did not pull the vault variables correctly.

cat /opt/portal/docker-entrypoint.sh
cat /opt/portal/app.env

Vault Connectivity

The setting of the ENVIRONMENT variable is in Salt code in the docker.portal.sls. The vault URL changes based on the ENVIRONMENT URL in the docker-entrypoint.sh file:

if [[ $ENVIRONMENT == "production" ]]; then
  export vault_addr="https://vault.mdr.defpoint.com"
  export token=$(/usr/local/src/vault_auth.sh production | jq .auth.client_token | sed -e 's/"//g')
elif [[ $ENVIRONMENT == "test" ]]; then
  export vault_addr="https://vault.mdr-test.defpoint.com"
  export token=$(/usr/local/src/vault_auth.sh test | jq .auth.client_token | sed -e 's/"//g')
fi

Did the container pull the variables correctly?
salt customer-portal* cmd.run 'docker exec portal cat /opt/portal/app.env | grep POSTGRES'

#get the token from AWS Auth
/usr/local/src/vault_auth.sh test
export token=xxx

#use the token to access the portal secrets
curl --header "X-Vault-Token: $token" https://vault.pvt.xdrtest.accenturefederalcyber.com/v1/portal/data/env

Admin access

After attempting to login to the site, run this command to get admin access.

python3 manage.py promote richard.t.page@accenturefederal.com python3 manage.py promote brad.poulton@accenturefederal.com

Your IP must be whitelisted to access the Admin Site

TEST LB ACCESS

6b6cdd72cd ERROR 2021-04-19 23:10:46,130 - Invalid HTTP_HOST header: 'portal-alb-test-617285333.us-gov-east-1.elb.amazonaws.com'. You may need to add 'portal-alb-test-617285333.us-gov-east-1.elb.amazonaws.com' to ALLOWED_HOSTS. [django.security.DisallowedHost:88] 54

ALLOWED_HOSTS is located.... /opt/portal/dps_customer_portal/settings.py

You can allow all subdomains like this, but OKTA doesn't support it:

ALLOWED_HOSTS = [
    PORTAL_DOMAIN,
    'localhost',
    '127.0.0.1',
    'portal.xdr.accenturefederalcyber.com',  # This will make prod transition smoother
    'portal.xdrtest.accenturefederalcyber.com',  # This will make testing easier
    '.us-gov-east-1.elb.amazonaws.com', # Allow LB 
    ]

python3 manage.py shell from user_portal.models import User

User.objects.filter(email="richard.t.page@accenturefederal.com").delete() User.objects.filter(email="wesley.a.leonard@accenturefederal.com").delete()

Troubleshooting the docker image

Logs

logs are being sent to Splunk. This logging is configured in Salt.

index=os source="journald://docker" host="customer-portal-*"

Nginx

/var/log/nginx/portal.access.log
docker logs -f nginx

Portal

startup.log I assume the logs going to splunk are the stdout.

salt 'customer-portal*' cmd.run 'docker container ls'
salt 'customer-portal*' cmd.run 'docker exec portal ls'
salt 'customer-portal*' cmd.run 'docker exec portal cat /opt/portal/saml/idps.json'
salt 'customer-portal*' cmd.run 'docker exec portal cat /opt/portal/saml/sp.json'

#This will init the portal variables by pulling them from vault. SHOULD NOT NEED TO RUN IT
salt 'customer-portal*' cmd.run 'docker exec portal sh /opt/portal/init.sh'
salt 'customer-portal*' cmd.run 'docker exec portal cat /opt/portal/init.sh'

#Portal auths to Vault then pulls the creds
salt 'customer-portal*' cmd.run 'docker exec portal cat /usr/local/src/vault_auth.sh'

docker exec -ti portal /usr/local/src/vault_auth.sh test

ALL THE ERRORS: nginx: [emerg] host not found in upstream "portal:8000" in /etc/nginx/nginx.conf:27

{"errors":["error making upstream request: error making request: Post https://sts.amazonaws.com//: dial tcp 52.94.241.129:443: i/o timeout"]}

[WARNING ] The following arguments were ignored because they are not recognized by docker-py: [u'dns-search', u'network-alias'] [WARNING ] The following arguments were ignored because they are not recognized by docker-py: [u'dns-search']

SOULTION: NOT SURE! try stopping docker containers and service and starting back up with salt state. seems to be proxy issue working server... [dev]root@ip-10-97-10-248:~:# docker exec portal wget portal --2020-04-30 17:44:37-- http://portal/ Resolving proxy.msoc.defpoint.local (proxy.msoc.defpoint.local)... 10.96.101.188 Connecting to proxy.msoc.defpoint.local (proxy.msoc.defpoint.local)|10.96.101.188|:80... connected. Proxy request sent, awaiting response... 503 Service Unavailable 2020-04-30 17:44:38 ERROR 503: Service Unavailable. broken server... [dev]root@ip-10-97-9-59:~:# docker exec portal wget portal --2020-04-30 17:27:45-- http://portal/ Resolving proxy.msoc.defpoint.local (proxy.msoc.defpoint.local)... failed: Name or service not known. wget: unable to resolve host address 'proxy.msoc.defpoint.local'

docker exec portal wget portal

sha256:598168ec922e79106fa3f8af35dd33313aa32ae859e77673b65d52ce93852810

Updating Email Settings

A bunch of settings (email, postgres, etc) are pulled dynamically from Vault as if Vault is a parameter store.

Vault Prod Vault Dev

Changes to the parameters require restarting the container.

Migrating to GovCloud

2. Database Dump

on legacy portal server ip-10-81-8-205

curl https://s3.amazonaws.com/rds-downloads/rds-ca-2019-root.pem -o root.crt
mkdir .postgresql
mv root.crt .postgresql
# Have to install pg10 from the official postgres repositories, but we'll remove it when we're done
sudo yum install https://download.postgresql.org/pub/repos/yum/10/redhat/rhel-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm
sudo yum install postgresql11
# yes, yes, yes
pg_dump postgresql://root-e26bbf10-ee24-2d05-97b7-47c54d9224a1@customerportal.csskgjc1suov.us-east-1.rds.amazonaws.com:5432/customerportal?sslmode=verify-full | gzip > backup.psql.gz
# Enter password which you got from Vault - portal/env/POSTGRES_PASSWORD

Restore the database

NOTE: New DB URL: customerportal.csqclvntmsrg.us-gov-east-1.rds.amazonaws.com

scp prod-ip-10-81-8-205:backup.psql.gz ./restore.psql.gz
scp restore.psql.gz gc-prod-customer-portal-26a:
ssh gc-prod-customer-portal-26a
# NOTE: This url is different from above, as there's a differnet cert needed
curl https://s3.us-gov-west-1.amazonaws.com/rds-downloads/rds-ca-us-gov-east-1-2017-root.pem -o root.crt
mkdir .postgresql
mv root.crt .postgresql
# Have to install pg10 from the official postgres repositories, but we'll remove it when we're done
sudo yum install https://download.postgresql.org/pub/repos/yum/10/redhat/rhel-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm
sudo yum install postgresql10
# create user
CREATE USER root-e26bbf10-ee24-2d05-97b7-47c54d9224a1 WITH PASSWORD '<see-vault>';
GRANT ALL PRIVILEGES ON DATABASE customerportal TO root-e26bbf10-ee24-2d05-97b7-47c54d9224a1;
zcat restore.psql.gz | psql --set ON_ERROR_STOP=on --single-transaction postgresql://portal@customerportal.csqclvntmsrg.us-gov-east-1.rds.amazonaws.com:5432/customerportal?sslmode=verify-full
# No errors the first time, wtf?

To restore a second time:

echo 'DROP DATABASE customerportal;' | psql postgresql://portal@customerportal.csqclvntmsrg.us-gov-east-1.rds.amazonaws.com:5432/postgres?sslmode=verify-full
echo 'CREATE DATABASE customerportal;' | psql postgresql://portal@customerportal.csqclvntmsrg.us-gov-east-1.rds.amazonaws.com:5432/postgres?sslmode=verify-full
psql postgresql://portal@customerportal.csqclvntmsrg.us-gov-east-1.rds.amazonaws.com:5432/postgres?sslmode=verify-full
CREATE ROLE "root-e26bbf10-ee24-2d05-97b7-47c54d9224a1";
DROP ROLE "root-e26bbf10-ee24-2d05-97b7-47c54d9224a1";
\du
\q

# the root-e26bbf10-ee24-2d05-97b7-47c54d9224a1 user is not really needed lets switch it to the portal user. 
find and replace root-e26bbf10-ee24-2d05-97b7-47c54d9224a1 to portal




echo 'DROP DATABASE jira' | psql postgresql://jira@jira2.csqclvntmsrg.us-gov-east-1.rds.amazonaws.com:5432/postgres?sslmode=verify-full
echo 'CREATE DATABASE jira' | psql postgresql://jira@jira2.csqclvntmsrg.us-gov-east-1.rds.amazonaws.com:5432/postgres?sslmode=verify-full
zcat restore.psql.gz | psql --set ON_ERROR_STOP=on --single-transaction postgresql://jira@jira2.csqclvntmsrg.us-gov-east-1.rds.amazonaws.com:5432/jira?sslmode=verify-full

Vuln Scan

08/16/2021

Create user

[gc-dev]root@customer-portal-982:~:# docker exec portal python3 manage.py shell --command="from user_portal.models import User,Company; u=User.objects.create_user(username='svc-scan', password='Uj4eY45cu59]eVL'); u.userprofileinfo.company=Company.objects.first(); u.save();"
INFO 2021-08-16 16:12:26,022 - TokenAdmin already registered (TokenProxy) [user_portal.admin:87] 6787
INFO 2021-08-16 16:12:26,022 - TokenAdmin already registered (TokenProxy) [user_portal.admin:87] 6787

Verify user

login into portal and go to /admin side. then users. Ensure svc-scan shows up.

View login form

go to https://portal.xdrtest.accenturefederalcyber.com/login/ and open developer tools in browser. Find the below code with the style="display: none"

<div class="container" style="display: none">
				<div class="row">
				<div class="col-md-4 col-xs-2">
...

Change "display: none" to just "display:". This will reveal the login form.