# GitHub Server Notes

`GitHub Enterprise Server` is an APPLIANCE. No Salt minion, no Teleport.
To SSH in you must have your public key manually added.

Host github
  Port 122
  User admin
  HostName 10.80.101.78

# Adding New Users to GitHub Teams

OKTA does NOT manage the permissions on the GitHub server. To give a user access to a new team, like `mdr-engineering`, log into the Github server and access this URL: [Login](https://github.xdr.accenturefederalcyber.com/orgs/mdr-engineering/teams/onboarding/members) . Find the new user by clicking on the "Add a member" button.

# Updating

```
ghe-update-check
ghe-upgrade /var/lib/ghe-updates/github-enterprise-2.17.22.hpkg
```

Upgrading major version

```
ghe-upgrade
fdisk -l
```

NOTE: The output of `ghe-upgrade-check` will provide you with the command to use to upgrade GitHub Enterprise.

Two partitions are installed. When you run an `upgrade` the VM will install the upgrade to the other partition. After the upgrade it will switch the primary boot partitions. This leaves the previous version available for roll back.


Hit ghe- (TAB) to view all ghe commands. GitHub [Command-line utilities](https://docs.github.com/en/enterprise/2.17/admin/installation/command-line-utilities)


# Installing new license

Should be able to do just via the [Web UI](https://github.xdr.accenturefederalcyber.com:8443/setup/upgrade)
But there's a gotcha with disabling the DSA key (for a FEDRAMP POAM). Your services may not restart after updating the license.

```
cp /data/user/common/ssh_host_rsa_key /data/user/common/ssh_host_rsa_key.pub /data/user/common/ssh_host_dsa_key /data/user/common/ssh_host_dsa_key.pub /data/user/common/ssh_host_ecdsa_key /data/user/common/ssh_host_ecdsa_key.pub /etc/ssh/
cp: cannot stat '/data/user/common/ssh_host_dsa_key': No such file or directory
cp: cannot stat '/data/user/common/ssh_host_dsa_key.pub': No such file or directory
Jun 30 16:09:54 ERROR: Preparing storage device
```

License install procedure:

```
# Put the DSA key back in place
sudo bash -c "mv /data/user/user-tmp/ssh_host_dsa* /data/user/common/"

# Now install the new license using the UI

# And re-remove the DSA key ...
sudo mv /data/user/common/ssh_host_dsa_key* /data/user/user-tmp/
sudo systemctl restart babeld
```

I'll open a case with Github too.

# GitHub-Backup

The `ghe-backup` servers are instances running `Docker`.

Docker is installed via the `docker` Salt state.

Most backup configuration is managed by the Salt `github.backup` state:
* `/usr/local/github-backup-utils` contains a copy of the [github repository](https://github.com/github/backup-utils). Be sure to run `git pull origin master` prior to upgrading/rebuilding the docker image and use the release version in the image tag.
* Build of the docker image, replace 'vX.y.z' with the backup-utils release version. Manual command is: `docker build --build-arg=http_proxy=$HTTP_PROXY --build-arg=https_proxy=$HTTPS_PROXY -t github/backup-utils:vX.y.z .`. You can run this if you get an error when applying the state.
* A script is run via a cronjob in `/etc/cron.d/ghe-backup`, which calls the script `/root/github-backup.sh`. This script calls docker to run the backup.
* You must update the `/root/ghe-backup.sh` script whenever the Docker image is upgraded or tagged differently.

# First time accepting the key

If there is a new GitHub or a new ghe-backup server, you will need to accept the SSH key. To do so, from the ghe-backup server, run:

```
sudo ssh -p 122 -i /etc/github-backup-utils/.ssh/id_rsa -o UserKnownHostsFile=/etc/github-backup-utils/.ssh/known_hosts github-enterprise-0.pvt.xdrtest.accenturefederalcyber.com -l admin
```

And accept the key.

# Restoring

Restoring should be similar to the command called by /root/github-backup.sh, except with a 'ghe-restore' command. You may need to update the script to use the latest Docker image build/tag.

# Troubleshooting Backup Failures

SSH to the ghe-backup server, `sudo -iu root` to become root and `cd /efs/github-prod/log` (or `/efs/github-test/log` if on XDR Test), then `ls -lrth | tail -3`. Grab the newest (last listed) backup log file and use `tail` to see where it is at.

Log entries to look for:
`No leaked keys found` -- The job completed successfully

`Error: A backup of github-enterprise-0.pvt.xdr.accenturefederalcyber.com may still be running on PID 1.
If PID 1 is not a process related to the backup utilities, please remove the /data/in-progress file and try again.` -- Something prevented the job from completing such as a reboot whilst the Docker container was creating the backup. Delete the `/efs/github-prod/data/in-progress` file.

Some failure alerts from Splunk may be due to the backup job taking longer than one hour to complete. This is likely due to some other process taking up CPU/memory on the ghe-backup host, preventing the Docker process from working efficiently (such as clamd).


# Migration Steps to Govcloud:

1) Create Okta App Manually
1) Stand everything up.
2) Run highstate 2x (This can t
   * May have to `pkg.upgrade` and/or reboot
3) Copy `/root/ghe-backup.sh` to `/root/ghe-backup-old.sh`, and update `hostname` to legacy hostname
4) Run `ssh` command (above) to get key into known hosts file
5) Run the `ghe-backup-old.sh` script
6) Copy `ghe-backup.sh` to `ghe-restore.sh`
7) Edit `ghe-restore.sh`, change log file name and `ghe-backup` to `ghe-restore`
8) Run `ghe-restore`.
9) Log onto instance on port `8443`
10) Let it do its thing, then go to settings:
   * Update hostname to `github.xdr.accenturefederalcyber.com`
   * Fix authentication with info from Okta and step 0
     * both the url and the `http:// address` need to be updated from the metadata
   * Enable `Allow X-Forwarded-For`
   * Keep `Enable Support for Proxy` enabled
   * Fix proxy configuration
   * Fix mailserver
11) Restore crontab to original
12) Disable old app in okta
13) Highstate Salt