infrastructure-notes/GitHub Server Notes.md

GitHub Server Notes

GitHub Enterprise Server is an APPLIANCE. No Salt minion, no Teleport. To SSH in you must have your public key manually added.

Host github Port 122 User admin HostName 10.80.101.78

Adding New Users to GitHub Teams

OKTA does NOT manage the permissions on the GitHub server. To give a user access to a new team, like mdr-engineering, log into the Github server and access this URL: Login . Find the new user by clicking on the "Add a member" button.

Updating

ghe-update-check
ghe-upgrade /var/lib/ghe-updates/github-enterprise-2.17.22.hpkg

Upgrading minor version

ghe-upgrade
fdisk -l

NOTE: The output of ghe-upgrade-check will provide you with the command to use to upgrade GitHub Enterprise.

Two partitions are installed. When you run an upgrade the VM will install the upgrade to the other partition. After the upgrade it will switch the primary boot partitions. This leaves the previous version available for roll back.

Hit ghe- (TAB) to view all ghe commands. GitHub Command-line utilities

Installing new license

Use the admin web UI to install a new license. Once uploaded and saved to the appliance, all GHE services will restart similar to minor version upgrades. The restarts may affect users.

ℹ️ The following instructions may no longer be relevant.

Should be able to do just via the Web UI But there's a gotcha with disabling the DSA key (for a FEDRAMP POAM). Your services may not restart after updating the license.

cp /data/user/common/ssh_host_rsa_key /data/user/common/ssh_host_rsa_key.pub /data/user/common/ssh_host_dsa_key /data/user/common/ssh_host_dsa_key.pub /data/user/common/ssh_host_ecdsa_key /data/user/common/ssh_host_ecdsa_key.pub /etc/ssh/
cp: cannot stat '/data/user/common/ssh_host_dsa_key': No such file or directory
cp: cannot stat '/data/user/common/ssh_host_dsa_key.pub': No such file or directory
Jun 30 16:09:54 ERROR: Preparing storage device

License install procedure:

# Put the DSA key back in place
sudo bash -c "mv /data/user/user-tmp/ssh_host_dsa* /data/user/common/"

# Now install the new license using the UI

# And re-remove the DSA key ...
sudo mv /data/user/common/ssh_host_dsa_key* /data/user/user-tmp/
sudo systemctl restart babeld

GitHub-Backup

The ghe-backup servers are instances running Docker.

Docker is installed via the docker Salt state.

Most backup configuration is managed by the Salt github.backup state:

• /usr/local/github-backup-utils contains a copy of the github repository. Be sure to run git pull origin master prior to upgrading/rebuilding the docker image and use the release version in the image tag.
• Build of the docker image, replace 'vX.y.z' with the backup-utils release version. Manual command is: docker build --build-arg=http_proxy=$HTTP_PROXY --build-arg=https_proxy=$HTTPS_PROXY -t github/backup-utils:vX.y.z .. You can run this if you get an error when applying the state.
• A script is run via a cronjob in /etc/cron.d/ghe-backup, which calls the script /root/github-backup.sh. This script calls docker to run the backup.
• You must update the /root/ghe-backup.sh script whenever the Docker image is upgraded or tagged differently.

First time accepting the key

If there is a new GitHub or a new ghe-backup server, you will need to accept the SSH key. To do so, from the ghe-backup server, run:

sudo ssh -p 122 -i /etc/github-backup-utils/.ssh/id_rsa -o UserKnownHostsFile=/etc/github-backup-utils/.ssh/known_hosts github-enterprise-0.pvt.xdrtest.accenturefederalcyber.com -l admin

And accept the key.

Restoring

Restoring should be similar to the command called by /root/github-backup.sh, except with a 'ghe-restore' command. You may need to update the script to use the latest Docker image build/tag.

Troubleshooting Backup Failures

SSH to the ghe-backup server, sudo -iu root to become root and cd /efs/github-prod/log (or /efs/github-test/log if on XDR Test), then ls -lrth | tail -3. Grab the newest (last listed) backup log file and use tail to see where it is at.

Log entries to look for: No leaked keys found -- The job completed successfully

Error: A backup of github-enterprise-0.pvt.xdr.accenturefederalcyber.com may still be running on PID 1. If PID 1 is not a process related to the backup utilities, please remove the /data/in-progress file and try again. -- Something prevented the job from completing such as a reboot whilst the Docker container was creating the backup. Delete the /efs/github-prod/data/in-progress file.

Some failure alerts from Splunk may be due to the backup job taking longer than one hour to complete. This is likely due to some other process taking up CPU/memory on the ghe-backup host, preventing the Docker process from working efficiently (such as clamd).

Migration Steps to Govcloud:

1) Create Okta App Manually 1) Stand everything up. 2) Run highstate 2x (This can t

• May have to pkg.upgrade and/or reboot 3) Copy /root/ghe-backup.sh to /root/ghe-backup-old.sh, and update hostname to legacy hostname 4) Run ssh command (above) to get key into known hosts file 5) Run the ghe-backup-old.sh script 6) Copy ghe-backup.sh to ghe-restore.sh 7) Edit ghe-restore.sh, change log file name and ghe-backup to ghe-restore 8) Run ghe-restore. 9) Log onto instance on port 8443 10) Let it do its thing, then go to settings:
• Update hostname to github.xdr.accenturefederalcyber.com
• Fix authentication with info from Okta and step 0
  • both the url and the http:// address need to be updated from the metadata
• Enable Allow X-Forwarded-For
• Keep Enable Support for Proxy enabled
• Fix proxy configuration
• Fix mailserver 11) Restore crontab to original 12) Disable old app in okta 13) Highstate Salt