# Jira Notes

# Quick Reference:

* RDS instance is `jira2`, but database is just `jira`
* Started by systemd / `/etc/rc.d/init.d/jira`
* Code/binary in `/opt/atlassian/jira`
    - `JIRA_HOME` set in `/opt/atlassian/jira/atlassian-jira/WEB-INF/classes/jira-application.properties`
* Data in `/opt/jira-data/jira` [ JIRA_HOME ]
    - DB Config in `/opt/jira-data/jira/dbconfig.xml`
    - Attachments in `/opt/jira/data/jira/attachments`


# TLS Setup for RDS

```
# for commercial
curl https://s3.amazonaws.com/rds-downloads/rds-ca-2019-root.pem
# For govcloud
curl https://s3.us-gov-west-1.amazonaws.com/rds-downloads/rds-ca-us-gov-east-1-2017-root.pem -o root.crt
```

Place output in /home/jira/.postgresql/root.crt

```
vim /opt/jira-data/jira/dbconfig.xml
# Add ?sslmode=verify-full to the <url>
```


~# TLS Setup for RDS - OUTDATED!!!
~
~First need to update `dbconfig.xml` to tell it to use TLS and what root certs to use:
~
~```
~    <url><![CDATA[jdbc:postgresql://jira.cm5pc4cb8hlj.us-east-1.rds.amazonaws.com:5432/jira?sslmode=verify-full&sslrootcert=/opt/atlassian/jira/rds-root-chain.pem]]></url>
~
~```
~
~Then in `/opt/atlassian/jira/rds-root-chain.pem` you need the root cert(s) for RDS.  Use something like this:
~
~```
~#!/bin/bash
~
~URLS="https://s3.amazonaws.com/rds-downloads/rds-ca-2019-root.pem"
~URLS="${URLS} https://s3.amazonaws.com/rds-downloads/rds-ca-2015-root.pem"
~URLS="${URLS} https://s3-us-gov-west-1.amazonaws.com/rds-downloads/rds-ca-us-gov-east-1-2017-root.pem"
~URLS="${URLS} https://s3-us-gov-west-1.amazonaws.com/rds-downloads/rds-ca-us-gov-west-1-2017-root.pem"
~
~rm rds-root-chain.pem
~
~for i in $URLS; do
~        echo "# `basename $i`"
~        curl -s $i
~done >> rds-root-chain.pem
~
~
~```
~see [Using SSL/TLS with RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html)
~
~There is mention of ways with newer versions of the PostgreSQL JDBC driver to use the 
~standard Java keystore for root certs.  This does not work with the version of the JDBC 
~driver skipping with Jira version 7.13, as the class needed is missing.  (There's no
~DefaultJavaSSLFactory in `postresql-9.4.1212.jar`)

One handy trick:

```
openssl s_client -starttls postgres -connect my.postgres.host:5432 # etc...
```


# Proxy setup

In ~`JIRA_HOME/bin/setenv.sh`~ `<jira binaries directory>/bin/setenv.sh`

```
JVM_SUPPORT_RECOMMENDED_ARGS=" -Dhttp.proxyHost=proxy.msoc.defpoint.local -Dhttp.proxyPort=80 -Dhttps.proxyHost=proxy.msoc.defpoint.local -Dhttps.proxyPort=80 -Dhttp.nonProxyHosts='*.defpoint.local|localhost|127.0.0.1|169.254.169.254|*.amazonaws.com'"
```

Without this, JIRA cannot download new plugins and things from the Atlassian repositories.

# Okta stuff

Okta appears to have provided their own SAML implementation for JIRA.  Which is weird, I
expected JIRA to have their own.

[Okta Jira Authenticator 3.x Configuration Guide](https://saml-doc.okta.com/Provisioning_Docs/Okta_Jira_Authenticator_Configuration_Guide.html)

There's a config file in `/opt/atlassian/jira/atlassian-jira/WEB-INF/classes/seraph-config.xml`
that refers to another config file `/opt/docker/okta-config-jira.xml`.  That is where the actual
SAML magic is stored.

# Load Balancer Stuff

There's stuff in `web.xml` that tells it that it's in front of a load balancer.  The 
proxyName and proxyPort settings matter, because they will cause redirects when
you connect to the wrong name.  Note that in the current config, the load balancer 
terminates TLS and sends plain HTTP back to JIRA itself.


```
        <Connector
        port="8080"
        relaxedPathChars="[]|"
        relaxedQueryChars="[]|{}^\`&quot;&lt;&gt;"
        maxThreads="150"
        minSpareThreads="25"
        connectionTimeout="20000"
        enableLookups="false"
        maxHttpHeaderSize="8192"
        protocol="HTTP/1.1"
        useBodyEncodingForURI="true"
        redirectPort="443"
        acceptCount="100"
        disableUploadTimeout="true"
        bindOnInit="false"
        proxyName="jira.mdr-test.defpoint.com"
        proxyPort="443"
        scheme="https"
        secure="true"
    />
```


# Useful links

[Setting properties and options on startup](https://confluence.atlassian.com/adminjiraserver085/setting-properties-and-options-on-startup-981155694.html)

[Change the Base URL of Jira server in the database](https://confluence.atlassian.com/jirakb/change-the-base-url-of-jira-server-in-the-database-733940375.html)


# Undockerizing

* Fix the split attachments dir
* Move attachments out to something like EFS
* Load balancer expects to connect to port 80, which is being forwarded by docker to 8080 inside the container.

# Migrating to GovCloud

## 0. Stop Jira

```
ssh prod-jira-server
sudo systemctl stop jira
sudo systemctl disable jira
```

## 1. Sync original:
```
cd
# I believe /opt/jira-data/jira/data/attachments is the correct directory, but it could be
#   /opt/jira/data/attachments
#   /opt/jira-data/jira/import/attachments
time rsync --rsync-path="sudo rsync" -avz --delete --progress prod-jira-server:/opt/atlassian tmp/atlassian
time rsync --rsync-path="sudo rsync" -avz --delete --progress prod-jira-server:/opt/jira-data tmp/jira-data

# then restore
rsync --rsync-path="sudo rsync" -avz --delete --progress tmp/atlassian/atlassian/ gc-prod-jira-server:/opt/atlassian/
rsync --rsync-path="sudo rsync" -avz --delete --progress tmp/jira-data/jira-data/ gc-prod-jira-server:/opt/jira-data/
```

## 2. Database Dump
on legacy server:
```
curl https://s3.amazonaws.com/rds-downloads/rds-ca-2019-root.pem -o root.crt
mkdir .postgresql
mv root.crt .postgresql
# Have to install pg11 from the official postgres repositories, but we'll remove it when we're done
sudo yum install https://download.postgresql.org/pub/repos/yum/11/redhat/rhel-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm
sudo yum install postgresql11
# yes, yes, yes
pg_dump postgresql://jira@jira2.csskgjc1suov.us-east-1.rds.amazonaws.com:5432/jira?sslmode=verify-full | gzip > backup.psql.gz
# Enter password which you got from /opt/jira-data/jira/dbconfig.xml
```

## Restore the database

NOTE: New DB URL: jira2.csqclvntmsrg.us-gov-east-1.rds.amazonaws.com

```
scp prod-jira-server:backup.psql.gz ./restore.psql.gz
scp restore.psql.gz gc-prod-jira-server:
ssh gc-prod-jira-server
# NOTE: This url is different from above, as there's a differnet cert needed
curl https://s3.us-gov-west-1.amazonaws.com/rds-downloads/rds-ca-us-gov-east-1-2017-root.pem -o root.crt
mkdir .postgresql
mv root.crt .postgresql
# Have to install pg11 from the official postgres repositories, but we'll remove it when we're done
sudo yum install https://download.postgresql.org/pub/repos/yum/11/redhat/rhel-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm
sudo yum install postgresql11
zcat restore.psql.gz | psql --set ON_ERROR_STOP=on --single-transaction postgresql://jira@jira2.csqclvntmsrg.us-gov-east-1.rds.amazonaws.com:5432/jira?sslmode=verify-full
# No errors the first time, wtf?
```

To restore a second time:
```
echo 'DROP DATABASE jira' | psql postgresql://jira@jira2.csqclvntmsrg.us-gov-east-1.rds.amazonaws.com:5432/postgres?sslmode=verify-full
echo 'CREATE DATABASE jira' | psql postgresql://jira@jira2.csqclvntmsrg.us-gov-east-1.rds.amazonaws.com:5432/postgres?sslmode=verify-full
zcat restore.psql.gz | psql --set ON_ERROR_STOP=on --single-transaction postgresql://jira@jira2.csqclvntmsrg.us-gov-east-1.rds.amazonaws.com:5432/jira?sslmode=verify-full
```

## 3. Install software

on the new server:
```
wget https://www.atlassian.com/software/jira/downloads/binary/atlassian-jira-software-8.12.1-x64.bin
sudo chmod 755 atlassian-jira-software-8.12.1-x64.bin
sudo ./atlassian-jira-software-8.12.1-x64.bin
# Yes to install fonts
# 1 for express install
# /opt/atlassian/jira (default) for install path
# /opt/jira-data/jira (NOT THE DEFAULT) for data path
# 1 for Default ports
# y to service
# Summary:
#   Installation Directory: /opt/atlassian/jira
#   Home Directory: /opt/jira-data/jira
#   HTTP Port: 8080
#   RMI Port: 8005
#   Install as service: Yes
# no to not start jira now
```

## 4. Update Configuration

```
sudo chown -R jira:jira /opt/atlassian /opt/jira-data
vim /opt/jira-data/jira/dbconfig.xml
# update the db connection string
vim /opt/atlassian/jira/conf/server.xml
# Update jira.mdr.defpoint.com to jira.xdr.accenturefederalcyber.com
vim /opt/atlassian/jira/bin/setenv.sh
# Set:
#  JVM_SUPPORT_RECOMMENDED_ARGS="-Dhttp.proxyHost=proxy.pvt.xdr.accenturefederalcyber.com -Dhttp.proxyPort=80 -Dhttps.proxyHost=proxy.pvt.xdr.accenturefederalcyber.com -Dhttps.proxyPort=80"
```

```
vim /opt/atlassian/jira/conf/okta-config-jira.xml
# copy from okta, remove the <xml> header
```

Login and resolve any issues, probably with basename
Update mail server.

NOTES:
new jira db:
jira2.csqclvntmsrg.us-gov-east-1.rds.amazonaws.com

# Upgrade Notes - 5/2021, 6/29/2021

Manual upgrade, recommended for Linux:
- [Upgrading Jira (manual)](https://confluence.atlassian.com/adminjiraserver0813/upgrading-jira-manual-1027137578.html)

Tickets: 
- [MSOCI-1606 - FedRAMP High Upgrade Jira to Resolve Qualys Findings](https://jira.xdr.accenturefederalcyber.com/browse/MSOCI-1606)
- [JRASERVER-71652 - CVE-2020-14184](https://jira.atlassian.com/browse/JRASERVER-71652)

Current Version: v8.12.1#812002-sha1:7c28d59
Target Version:  5/2021: v8.13.6 (LTS) [Link](https://www.atlassian.com/software/jira/download)
                 6/2021: v8.13.8 (LTS) 

Download 'tar.gz' from [Link](https://www.atlassian.com/software/jira/download)
Copy to server.

## Prep: Plenty of backups
```
sudo cat /opt/jira-data/jira/dbconfig.xml
pg_dump postgresql://jira@jira2.csqclvntmsrg.us-gov-east-1.rds.amazonaws.com:5432/jira?sslmode=verify-full | gzip > backup.202105.psql.gz
cd /opt
sudo rsync -rav atlassian atlassian.bak.202105
sudo rsync -rav jira-data jira-data.202106
```

Then in the GUI:
1) Take a snapshot of the instance (be sure to check the box to not restart it)
2) Take a snapshot of the database

## Prep: Extract and Configure

```
cd /opt/atlassian
sudo tar xvzf ~/atlassian-jira-software-8.13.6.tar.gz
sudo mv atlassian-jira-software-8.13.6-standalone jira-8.13.6
cd jira-8.13.6/
sudo chown -R jira:jira .
sudo cp ../jira/atlassian-jira/WEB-INF/lib/okta-jira-3.1.3.jar ./atlassian-jira/WEB-INF/lib/okta-jira-3.1.3.jar
sudo cp ../jira/atlassian-jira/okta_login.jsp ./atlassian-jira/okta_login.jsp
sudo cp ../jira/conf/okta-config-jira.xml ./conf/okta-config-jira.xml

sudo vim atlassian-jira/WEB-INF/classes/jira-application.properties
# set jira.home to /opt/jira-data/jira
sudo vim bin/setenv.sh
# set JIRA_HOME (maybe? It wasn't set in the old one, but i'm setting it)
# Copy JVM_SUPPORT_RECOMMENDED_ARGS from old copy
# Note: previous setting JVM_MAXIMUM_MEMORY="768m" is now JVM_MAXIMUM_MEMORY="2048m"
```

In `atlassian-jira/WEB-INF/web.xml`, just before `THIS MUST BE THE LAST FILTER IN THE DEFINED CHAIN`, add the following:
```
    <filter>
        <filter-name>OktaLoginFilter</filter-name>
        <filter-class>com.atlassian.jira.authenticator.okta.OktaLoginFilter</filter-class>
    </filter>

    <filter-mapping>
        <filter-name>OktaLoginFilter</filter-name>
        <url-pattern>/*</url-pattern>
        <dispatcher>REQUEST</dispatcher>
        <dispatcher>FORWARD</dispatcher>
    </filter-mapping>
```

In `conf/server.xml` in the first 'connector' clause, update the last two lines (only chagne on the first line is the `/>`):: 
```
acceptCount="100" disableUploadTimeout="true" bindOnInit="false"
proxyName="jira.xdr.accenturefederalcyber.com" proxyPort="443" scheme="https" secure="true"/>
```

In `atlassian-jira/WEB-INF/classes/seraph-config.xml`:

1. For `<param-name>logout.url</param-name>`, set:
```
<param-value>https://mdr-multipass.okta.com</param-value>
```

2. Comment out the following:
```
    <!-- <authenticator class="com.atlassian.jira.security.login.JiraSeraphAuthenticator"/> -->
```

3. After the last `<!-- CROWD:END -->`, add the following section (note the fixed path):
```
    <!-- MDR OKTA:START -->

        <authenticator class="com.atlassian.jira.authenticator.okta.OktaJiraAuthenticator30">
            <init-param>
                <param-name>okta.config.file</param-name>
                <param-value>/opt/atlassian/jira-8.13.6/conf/okta-config-jira.xml</param-value>
            </init-param>
        </authenticator>

    <!-- MDR OKTA:END -->
```

Update okta to latest (optional):
1. Log into okta admin
2. go to settings->downloads
3. Download the latest okta plugin
4. Copy to the server

```
ssh gc-prod-okta-server
cd /opt/atlassian/jira-8.13.6
sudo rm atlassian-jira/WEB-INF/lib/okta-jira-3.1.3.jar
sudo mv ~frederick_t_damstra/okta-jira-3.1.5.jar atlassian-jira/WEB-INF/lib/
sudo chown jira:jira atlassian-jira/WEB-INF/lib/okta-jira-3.1.5.jar
sudo chmod 644 atlassian-jira/WEB-INF/lib/okta-jira-3.1.5.jar
```

## Cutover:

```
sudo /etc/init.d/jira stop
# In GUI, take db snapshot
cd /opt
sudo rsync -rav atlassian atlassian.bak.202105
sudo rsync -rav jira-data jira-data.202106
sudo mv /opt/atlassian/jira /opt/atlassian/jira-8.12.1
vim /etc/rc.d/init.d/jira
# Update path to /opt/atlassian/jira/bin
sudo /etc/init.d/jira start
```

Note, the 'check upgrade' step, suggested that these might have been modified, too:
```
jira-application.properties
WEB-INF/web.xml
seraph-config.xml
``` 

Also, if okta doesn't work, you can try placing okta back:
```
sudo cp /opt/atlassian/jira/atlassian-jira/WEB-INF/lib/okta-jira-3.1.3.jar /opt/atlassian/jira-8.13.6/atlassian-jira/WEB-INF/lib/okta-jira-3.1.3.jar
```

## Problems during 4/27/2021 upgrade:

### Jira wouldn't start because JAVA_HOME was not yet. 
1. edit bin/setenv.sh, search for JAVA_HOME and replace with:
```
JAVA_HOME="/opt/atlassian/jre/"; export JAVA_HOME
```
2. Copy the old JRE (and I bet this needs to be updated)

### Jira started, but database won't connect. This was because of SSL.
1. temporarily removed the verify-ssl command from /opt/jira-data/jira/dbconfig.xml
2. It started.
3. Stopped it after it initialized

1. copied /home/jira/.posgres to /opt/jira-data/jira/.postgres (this contains the root cert)
no change
Tried /opt/jira-data/.postgres, too.

Left it with `sslmode=require`.

### Noisy catalina error: `The encoding [binary] is not recognised by the JRE`, with full stack trace:

see [JRASERVER - 71265 - The encoding [binary] is not recognised by the JRE](https://jira.atlassian.com/browse/JRASERVER-71265)
added 3 lines to `/opt/atlassian/jira-8.13.6/conf/logging.properties`:
```
# per https://jira.atlassian.com/browse/JRASERVER-71265
# Surpress 'The encoding [binary] is not recognised by the JRE'
org.apache.catalina.connector.Response.level = ERROR
```

## Problems during 6/29/2021 Upgrade

The startup says `Unsupported collation error` after update. This page https://confluence.atlassian.com/jirakb/unsupported-collation-error-thrown-in-jira-server-using-postgresql-776657961.html says it's right. We have en_US.UTF-8 instead of one of the supported ones.

However, I was able to resolve by editing /opt/jira-data/jira/dbconfig.xml and temporarily removing `?sslmode=require` from teh connection string. After stopping and starting, I re-added this option and stopped nad started again, and the error was gone.
 
## If all is well

do a little dance

# Updating JRE

If jira was installed via the tarball (as in the govcloud move above), java is not updated as part of the package and may need to be updated separately.

To update:

~1. Download from AdoptOpenJDK from https://adoptopenjdk.net/releases.html?variant=openjdk8&jvmVariant=hotspot~
1. Download from AdoptOpenJDK from https://adoptium.net/releases.html?variant=openjdk11
  a. Do *NOT* download from oracle.com, as that requires a commercial license.
  b. Select "Temurin 11 (LTS)", "Linux", "x64", and download the JRE, not the JDK
2. Copy the file to the server. E.g.:
```
tshp scp OpenJDK11U-jre_x64_linux_hotspot_11.0.11_9.tar.gz jira-server.pvt.xdr.accenturefederalcyber.com:
```
3. Extract and copy the JRE into a temporary location
```
tshp jira-server
tar xvzf OpenJDK11U-jre_x64_linux_hotspot_11.0.11_9.tar.gz
sudo mv jdk-11.0.11+9-jre/ /opt/atlassian/jre-11.0.1+9
sudo chown -R jira:jira /opt/atlassian/jre-11.0.1+9
sudo find /opt/atlassian/jre-11.0.1+9 -type d -exec chmod o+rx {} \;
sudo find /opt/atlassian/jre-11.0.1+9 -type f -exec chmod o+r {} \;
sudo find /opt/atlassian/jre-11.0.1+9/bin -type f -exec chmod o+rx {} \;
/opt/atlassian/jre-11.0.1+9/bin/java -version
```
4. Back up the old jre:
```
cd /opt/atlassian
sudo cp -a jre jre.20210629
```
5. Install the updated JRE
```
cd /opt/atlassian
sudo /etc/rc.d/init.d/jira stop
sudo mv jre jre.old
sudo mv jre-11.0.1+9 jre
sudo /etc/rc.d/init.d/jira start
```

# Upgrading

A salt state can be used for upgrading. See `msoc-infrastructure/salt/fileroots/jira/README.md`.

## Prepare

1. Go to jira, click on teh Options->Applications option.
1. Click on "Plan your upgrade"
1. Review anything that looks dangerous.

## Staging the Update

The state will install the update into a new directory (/opt/atlassian/jira-{version}), but will not stop/start it. You should be able to safely run this at any time (still, follow best practices: make backups first, run salt with `test=true`).

1. Download the latest LTS from [Jira Software](https://www.atlassian.com/software/jira/download-journey). Download in .tar.gz
  a. upgrade to latest release
  b. Server
  c. Latest Release
  d. tar.gz archive
1. Copy to afsxdr-binaries:
```
aws --profile mdr-common-services-gov s3 cp atlassian-jira-software-8.13.10.tar.gz s3://afsxdr-binaries/jira/
```
1. update `msoc-infrastructure/salt/fileroots/jira/init.sls` with the latest version
1. Optionally, update the `okta_jar` using the same steps as above (download from okta admin, downloads).
1. Optionally, update the `xmlsec` jar using the same steps as above (download from url in `init.sls`).
1. Do a PR to develop.
1. Run the state in test, follow cutover below.
  `salt jira\* state.sls jira --output-diff test=true` # not useful; will show lots of errors, since it doesn't download the update
  `salt jira\* state.sls jira --output-diff test=false`
1. Test in test ( https://jira.xdrtest.accenturefederalcyber.com/ ; Configuration is incomplete -- can't even log in )
1. PR to master
1. Apply, cutover, and test in prod

## Cutover

The salt state does not change to the new version.

To cut over:

1. Judiciously remove older files in `/opt/jira-data/jira/export`
1. Create backups of jira and jira-data
```
cd /opt/
sudo tar cvzf jira-backup.20220818.tar.gz atlassian jira-data
```
1. Run `sudo systemctl stop jira`
1. Edit `/etc/init.d/jira`
1. Change the path to the updated jira version
1. `sudo systemctl daemon-reload`
1. Run `sudo systemctl start jira && sudo tail -F /opt/atlassian/jira-VERSION/logs/catalina.out`
1. If it warns "Upgrade: Custom changes have not been carried over", click 'ignore and continue'. If it returns to teh same screen, stop and start jira. The saltstate carried over these changes.
```
sudo systemctl restart jira && sudo tail -F /opt/atlassian/jira-9.1.0/logs/catalina.out
```
1. Test jira may redirect you to prod jira. Watch your URL!

## Cleanup

Once everything is runnign, there are a few things left:
1. Clean out hte old directory to remove any warnings.
```
cd /opt/atlassian
rm -rf jira-9.0.0 # or whatever previous version was
```
1. Go to the web interface, to the gear (settings)->manage apps
1. click on 'manage applications' from the new menu
1. click on 'update' on each application that has an update


...

## Java Updates

No support in this state yet for updating/installing the JRE. See `infrastructure-notes/Jira\ Notes.md` for detailed steps, or quick reference:

## Error after Upgrading to 8.22.2

There wasn't a lot of useful information in the logs. In `/opt/atlassian/jira-8.22.2/logs/catalina.out`:

```
12-May-2022 13:54:49.205 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal One or more Filters failed to start. Full details will be found in the appropriate container log file
12-May-2022 13:54:49.210 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal Context [] startup failed due to previous errors
```

```
2022-05-12 13:54:59,133+0000 JIRA-Bootstrap ERROR      [c.a.jira.upgrade.PluginSystemLauncher] A fatal error occured during initialisation. JIRA has been locked.
com.atlassian.jira.InfrastructureException: Error occurred while starting Plugin Manager. null
```

```
2022-05-12 13:54:59,420+0000 JIRA-Bootstrap ERROR      [c.a.jira.startup.DefaultJiraLauncher] JIRA has failed to start because of the following errors: [(Event: Level = (EventLevel: fatal) , Key = (EventType: startup-unexpected) , Desc = We couldn't start JIRA , Exception = An error occurred while trying to start JIRA. We can't give you any more detail right now, we suggest checking the logs for more detail and contacting our support team.<br/>See our documentation for more information on contacting our support team and creating a support zip.), (Event: Level = (EventLevel: fatal) , Key = (EventType: database) , Desc = Could not execute health check, DatabaseConfigurationManager not available. , Exception = ]
```

Solution was here: 
https://confluence.atlassian.com/jirakb/jira-integrated-with-okta-fails-to-start-after-upgraging-to-8-22-2-1127254525.html

YOU DO NOT NEED TO DO THIS AS IT IS NOW HANDLED BY SALT. But Basically:
1. Downloaded xmlsec-2.3.0.jar from https://javalibs.com/artifact/org.apache.santuario/xmlsec
2. Place it in `/opt/atlassian/jira-VERSION/atlassian-jira/WEB-INFO/lib/`
