Jira Notes

Quick Reference:

RDS instance is jira2, but database is just jira
Started by systemd / /etc/rc.d/init.d/jira
Code/binary in /opt/atlassian/jira
- JIRA_HOME set in /opt/atlassian/jira/atlassian-jira/WEB-INF/classes/jira-application.properties
Data in /opt/jira-data/jira [ JIRA_HOME ]
- DB Config in /opt/jira-data/jira/dbconfig.xml
- Attachments in /opt/jira/data/jira/attachments

TLS Setup for RDS

# for commercial
curl https://s3.amazonaws.com/rds-downloads/rds-ca-2019-root.pem

# For govcloud
curl https://s3.us-gov-west-1.amazonaws.com/rds-downloads/rds-ca-rsa4096-g1.pem -o root.crt

Place output in /home/jira/.postgresql/root.crt

vim /opt/jira-data/jira/dbconfig.xml
# Add ?sslmode=verify-full to the <url>

~# TLS Setup for RDS - OUTDATED!!! ~ ~First need to update dbconfig.xml to tell it to use TLS and what root certs to use: ~ ~~ <url><![CDATA[jdbc:postgresql://jira.cm5pc4cb8hlj.us-east-1.rds.amazonaws.com:5432/jira?sslmode=verify-full&sslrootcert=/opt/atlassian/jira/rds-root-chain.pem]]></url> ~ ~ ~ ~Then in /opt/atlassian/jira/rds-root-chain.pem you need the root cert(s) for RDS. Use something like this: ~ ~~#!/bin/bash ~ ~URLS="https://s3.amazonaws.com/rds-downloads/rds-ca-2019-root.pem" ~URLS="${URLS} https://s3.amazonaws.com/rds-downloads/rds-ca-2015-root.pem" ~URLS="${URLS} https://s3-us-gov-west-1.amazonaws.com/rds-downloads/rds-ca-us-gov-east-1-2017-root.pem" ~URLS="${URLS} https://s3-us-gov-west-1.amazonaws.com/rds-downloads/rds-ca-us-gov-west-1-2017-root.pem" ~ ~rm rds-root-chain.pem ~ ~for i in $URLS; do ~ echo "# `basename $i`" ~ curl -s $i ~done >> rds-root-chain.pem ~ ~ ~ ~see Using SSL/TLS with RDS ~ ~There is mention of ways with newer versions of the PostgreSQL JDBC driver to use the ~standard Java keystore for root certs. This does not work with the version of the JDBC ~driver skipping with Jira version 7.13, as the class needed is missing. (There's no ~DefaultJavaSSLFactory in postresql-9.4.1212.jar)

One handy trick:

openssl s_client -starttls postgres -connect my.postgres.host:5432 # etc...

Proxy setup

In ~JIRA_HOME/bin/setenv.sh~ <jira binaries directory>/bin/setenv.sh

JVM_SUPPORT_RECOMMENDED_ARGS=" -Dhttp.proxyHost=proxy.msoc.defpoint.local -Dhttp.proxyPort=80 -Dhttps.proxyHost=proxy.msoc.defpoint.local -Dhttps.proxyPort=80 -Dhttp.nonProxyHosts='*.defpoint.local|localhost|127.0.0.1|169.254.169.254|*.amazonaws.com'"

Without this, JIRA cannot download new plugins and things from the Atlassian repositories.

Okta stuff

Okta appears to have provided their own SAML implementation for JIRA. Which is weird, I expected JIRA to have their own.

Okta Jira Authenticator 3.x Configuration Guide

There's a config file in /opt/atlassian/jira/atlassian-jira/WEB-INF/classes/seraph-config.xml that refers to another config file /opt/docker/okta-config-jira.xml. That is where the actual SAML magic is stored.

Load Balancer Stuff

There's stuff in web.xml that tells it that it's in front of a load balancer. The proxyName and proxyPort settings matter, because they will cause redirects when you connect to the wrong name. Note that in the current config, the load balancer terminates TLS and sends plain HTTP back to JIRA itself.

        <Connector
        port="8080"
        relaxedPathChars="[]|"
        relaxedQueryChars="[]|{}^\`&quot;&lt;&gt;"
        maxThreads="150"
        minSpareThreads="25"
        connectionTimeout="20000"
        enableLookups="false"
        maxHttpHeaderSize="8192"
        protocol="HTTP/1.1"
        useBodyEncodingForURI="true"
        redirectPort="443"
        acceptCount="100"
        disableUploadTimeout="true"
        bindOnInit="false"
        proxyName="jira.mdr-test.defpoint.com"
        proxyPort="443"
        scheme="https"
        secure="true"
    />

Useful links

Setting properties and options on startup

Change the Base URL of Jira server in the database

Undockerizing

Fix the split attachments dir
Move attachments out to something like EFS
Load balancer expects to connect to port 80, which is being forwarded by docker to 8080 inside the container.

Migrating to GovCloud

0. Stop Jira

ssh prod-jira-server
sudo systemctl stop jira
sudo systemctl disable jira

1. Sync original:

cd
# I believe /opt/jira-data/jira/data/attachments is the correct directory, but it could be
#   /opt/jira/data/attachments
#   /opt/jira-data/jira/import/attachments
time rsync --rsync-path="sudo rsync" -avz --delete --progress prod-jira-server:/opt/atlassian tmp/atlassian
time rsync --rsync-path="sudo rsync" -avz --delete --progress prod-jira-server:/opt/jira-data tmp/jira-data

# then restore
rsync --rsync-path="sudo rsync" -avz --delete --progress tmp/atlassian/atlassian/ gc-prod-jira-server:/opt/atlassian/
rsync --rsync-path="sudo rsync" -avz --delete --progress tmp/jira-data/jira-data/ gc-prod-jira-server:/opt/jira-data/

2. Database Dump

on legacy server:

curl https://s3.amazonaws.com/rds-downloads/rds-ca-2019-root.pem -o root.crt
mkdir .postgresql
mv root.crt .postgresql
# Have to install pg11 from the official postgres repositories, but we'll remove it when we're done
sudo yum install https://download.postgresql.org/pub/repos/yum/11/redhat/rhel-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm
sudo yum install postgresql11
# yes, yes, yes
pg_dump postgresql://jira@jira2.csskgjc1suov.us-east-1.rds.amazonaws.com:5432/jira?sslmode=verify-full | gzip > backup.psql.gz
# Enter password which you got from /opt/jira-data/jira/dbconfig.xml

Restore the database

NOTE: New DB URL: jira2.csqclvntmsrg.us-gov-east-1.rds.amazonaws.com

scp prod-jira-server:backup.psql.gz ./restore.psql.gz
scp restore.psql.gz gc-prod-jira-server:
ssh gc-prod-jira-server
# NOTE: This url is different from above, as there's a differnet cert needed
curl https://s3.us-gov-west-1.amazonaws.com/rds-downloads/rds-ca-us-gov-east-1-2017-root.pem -o root.crt
mkdir .postgresql
mv root.crt .postgresql
# Have to install pg11 from the official postgres repositories, but we'll remove it when we're done
sudo yum install https://download.postgresql.org/pub/repos/yum/11/redhat/rhel-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm
sudo yum install postgresql11
zcat restore.psql.gz | psql --set ON_ERROR_STOP=on --single-transaction postgresql://jira@jira2.csqclvntmsrg.us-gov-east-1.rds.amazonaws.com:5432/jira?sslmode=verify-full
# No errors the first time, wtf?

To restore a second time:

echo 'DROP DATABASE jira' | psql postgresql://jira@jira2.csqclvntmsrg.us-gov-east-1.rds.amazonaws.com:5432/postgres?sslmode=verify-full
echo 'CREATE DATABASE jira' | psql postgresql://jira@jira2.csqclvntmsrg.us-gov-east-1.rds.amazonaws.com:5432/postgres?sslmode=verify-full
zcat restore.psql.gz | psql --set ON_ERROR_STOP=on --single-transaction postgresql://jira@jira2.csqclvntmsrg.us-gov-east-1.rds.amazonaws.com:5432/jira?sslmode=verify-full

3. Install software

on the new server:

wget https://www.atlassian.com/software/jira/downloads/binary/atlassian-jira-software-9.2.0.tar.gz
sudo chmod 755 atlassian-jira-software-9.2.0-x64.bin
sudo ./atlassian-jira-software-9.2.0-x64.bin
# Yes to install fonts
# 1 for express install
# /opt/atlassian/jira (default) for install path
# /opt/jira-data/jira (NOT THE DEFAULT) for data path
# 1 for Default ports
# y to service
# Summary:
#   Installation Directory: /opt/atlassian/jira
#   Home Directory: /opt/jira-data/jira
#   HTTP Port: 8080
#   RMI Port: 8005
#   Install as service: Yes
# no to not start jira now

4. Update Configuration

sudo chown -R jira:jira /opt/atlassian /opt/jira-data
vim /opt/jira-data/jira/dbconfig.xml
# update the db connection string
vim /opt/atlassian/jira/conf/server.xml
# Update jira.mdr.defpoint.com to jira.xdr.accenturefederalcyber.com
vim /opt/atlassian/jira/bin/setenv.sh
# Set:
#  JVM_SUPPORT_RECOMMENDED_ARGS="-Dhttp.proxyHost=proxy.pvt.xdr.accenturefederalcyber.com -Dhttp.proxyPort=80 -Dhttps.proxyHost=proxy.pvt.xdr.accenturefederalcyber.com -Dhttps.proxyPort=80"

vim /opt/atlassian/jira/conf/okta-config-jira.xml
# copy from okta, remove the <xml> header

NOTES: new jira db: jira2.csqclvntmsrg.us-gov-east-1.rds.amazonaws.com

Upgrade Notes - 5/2021, 6/29/2021

Manual upgrade, recommended for Linux:

Upgrading Jira (manual)

Tickets:

Current Version: v8.12.1#812002-sha1:7c28d59 Target Version: 5/2021: v8.13.6 (LTS) Link

             6/2021: v8.13.8 (LTS)
             8/2022: v9.2.0 (LTS)

Download 'tar.gz' from Link Copy to server.

Prep: Plenty of backups

sudo cat /opt/jira-data/jira/dbconfig.xml
pg_dump postgresql://jira@jira2.csqclvntmsrg.us-gov-east-1.rds.amazonaws.com:5432/jira?sslmode=verify-full | gzip > backup.202105.psql.gz
cd /opt
sudo rsync -rav atlassian atlassian.bak.202105
sudo rsync -rav jira-data jira-data.202106

Then in the GUI: 1) Take a snapshot of the instance (be sure to check the box to not restart it) 2) Take a snapshot of the database

Prep: Extract and Configure

cd /opt/atlassian
sudo tar xvzf ~/atlassian-jira-software-8.13.6.tar.gz
sudo mv atlassian-jira-software-8.13.6-standalone jira-8.13.6
cd jira-8.13.6/
sudo chown -R jira:jira .
sudo cp ../jira/atlassian-jira/WEB-INF/lib/okta-jira-3.1.3.jar ./atlassian-jira/WEB-INF/lib/okta-jira-3.1.3.jar
sudo cp ../jira/atlassian-jira/okta_login.jsp ./atlassian-jira/okta_login.jsp
sudo cp ../jira/conf/okta-config-jira.xml ./conf/okta-config-jira.xml

sudo vim atlassian-jira/WEB-INF/classes/jira-application.properties
# set jira.home to /opt/jira-data/jira
sudo vim bin/setenv.sh
# set JIRA_HOME (maybe? It wasn't set in the old one, but i'm setting it)
# Copy JVM_SUPPORT_RECOMMENDED_ARGS from old copy
# Note: previous setting JVM_MAXIMUM_MEMORY="768m" is now JVM_MAXIMUM_MEMORY="2048m"

In atlassian-jira/WEB-INF/web.xml, just before THIS MUST BE THE LAST FILTER IN THE DEFINED CHAIN, add the following:

    <filter>
        <filter-name>OktaLoginFilter</filter-name>
        <filter-class>com.atlassian.jira.authenticator.okta.OktaLoginFilter</filter-class>
    </filter>

    <filter-mapping>
        <filter-name>OktaLoginFilter</filter-name>
        <url-pattern>/*</url-pattern>
        <dispatcher>REQUEST</dispatcher>
        <dispatcher>FORWARD</dispatcher>
    </filter-mapping>

In conf/server.xml in the first 'connector' clause, update the last two lines (only chagne on the first line is the />)::

acceptCount="100" disableUploadTimeout="true" bindOnInit="false"
proxyName="jira.xdr.accenturefederalcyber.com" proxyPort="443" scheme="https" secure="true"/>

In atlassian-jira/WEB-INF/classes/seraph-config.xml:

For <param-name>logout.url</param-name>, set:

<param-value>https://mdr-multipass.okta.com</param-value>

Comment out the following:

<!-- <authenticator class="com.atlassian.jira.security.login.JiraSeraphAuthenticator"/> -->

After the last , add the following section (note the fixed path):

<!-- MDR OKTA:START -->

    <authenticator class="com.atlassian.jira.authenticator.okta.OktaJiraAuthenticator30">
        <init-param>
            <param-name>okta.config.file</param-name>
            <param-value>/opt/atlassian/jira-8.13.6/conf/okta-config-jira.xml</param-value>
        </init-param>
    </authenticator>

<!-- MDR OKTA:END -->

Update okta to latest (optional):

Log into okta admin
go to settings->downloads
Download the latest okta plugin

Copy to the server

ssh gc-prod-okta-server
cd /opt/atlassian/jira-8.13.6
sudo rm atlassian-jira/WEB-INF/lib/okta-jira-3.1.3.jar
sudo mv ~frederick_t_damstra/okta-jira-3.1.5.jar atlassian-jira/WEB-INF/lib/
sudo chown jira:jira atlassian-jira/WEB-INF/lib/okta-jira-3.1.5.jar
sudo chmod 644 atlassian-jira/WEB-INF/lib/okta-jira-3.1.5.jar

Cutover:

sudo /etc/init.d/jira stop
# In GUI, take db snapshot
cd /opt
sudo rsync -rav atlassian atlassian.bak.202105
sudo rsync -rav jira-data jira-data.202106
sudo mv /opt/atlassian/jira /opt/atlassian/jira-8.12.1
vim /etc/rc.d/init.d/jira
# Update path to /opt/atlassian/jira/bin
sudo /etc/init.d/jira start

Note, the 'check upgrade' step, suggested that these might have been modified, too:

jira-application.properties
WEB-INF/web.xml
seraph-config.xml

Also, if okta doesn't work, you can try placing okta back:

sudo cp /opt/atlassian/jira/atlassian-jira/WEB-INF/lib/okta-jira-3.1.3.jar /opt/atlassian/jira-8.13.6/atlassian-jira/WEB-INF/lib/okta-jira-3.1.3.jar

Problems during 4/27/2021 upgrade:

Jira wouldn't start because JAVA_HOME was not yet.

edit bin/setenv.sh, search for JAVA_HOME and replace with:
```
JAVA_HOME="/opt/atlassian/jre/"; export JAVA_HOME
```
Copy the old JRE (and I bet this needs to be updated)

Jira started, but database won't connect. This was because of SSL.

temporarily removed the verify-ssl command from /opt/jira-data/jira/dbconfig.xml
It started.
Stopped it after it initialized
copied /home/jira/.posgres to /opt/jira-data/jira/.postgres (this contains the root cert) no change Tried /opt/jira-data/.postgres, too.

Left it with sslmode=require.

Noisy Catalina error: `The encoding [binary] is not recognised by the JRE`, with full stack trace:

see JRASERVER - 71265 - The encoding [binary] is not recognised by the JRE added 3 lines to /opt/atlassian/jira-8.13.6/conf/logging.properties:

# per https://jira.atlassian.com/browse/JRASERVER-71265
# Surpress 'The encoding [binary] is not recognised by the JRE'
org.apache.catalina.connector.Response.level = ERROR

Problems during 6/29/2021 Upgrade

The startup says Unsupported collation error after update. This page Unsupported collation error thrown in Jira server using PostgreSQL says it's right. We have en_US.UTF-8 instead of one of the supported ones.

However, I was able to resolve by editing /opt/jira-data/jira/dbconfig.xml and temporarily removing ?sslmode=require from teh connection string. After stopping and starting, I re-added this option and stopped nad started again, and the error was gone.

If all is well

do a little dance

Updating JRE

If Jira was installed via the tarball (as in the Govcloud move above), Java is not updated as part of the package and may need to be updated separately.

To update:

Download from AdoptOpenJDK from Eclipse Temurin version 11
a. Do NOT download from oracle.com, as that requires a commercial license.
b. Select "Temurin 11 (LTS)", "Linux", "x64", and download the JRE, not the JDK

Copy the file to the server. E.g.:

tshp scp OpenJDK11U-jre_x64_linux_hotspot_11.0.16.1_1.tar.gz jira-server.pvt.xdr.accenturefederalcyber.com:

Extract and copy the JRE into a temporary location

tshp jira-server
tar xvzf OpenJDK11U-jre_x64_linux_hotspot_11.0.16.1_1.tar.gz
sudo mv jdk-11.0.16.1+1-jre/ /opt/atlassian/jre-11.0.16.1+1
sudo chown -R jira:jira /opt/atlassian/jre-11.0.16.1+1
sudo find /opt/atlassian/jre-11.0.16.1+1 -type d -exec chmod o+rx {} \;
sudo find /opt/atlassian/jre-11.0.16.1+1 -type f -exec chmod o+r {} \;
sudo find /opt/atlassian/jre-11.0.16.1+1/bin -type f -exec chmod o+rx {} \;
/opt/atlassian/jre-11.0.16.1+1/bin/java -version

Back up the old jre:

cd /opt/atlassian
sudo cp -a jre jre.20220830

Install the updated JRE

cd /opt/atlassian
sudo /etc/rc.d/init.d/jira stop
sudo mv jre jre.old
sudo mv jre-11.0.16.1+1 jre
sudo /etc/rc.d/init.d/jira start

Upgrading

A salt state can be used for upgrading. See msoc-infrastructure/salt/fileroots/jira/README.md.

Prepare

Go to Jira, click on teh Options > Applications option.
Click on "Plan your upgrade"
Review anything that looks dangerous.

Staging the Update

The Salt State will install the update into a new directory /opt/atlassian/jira-{version}, but will not stop/start it. You should be able to safely run this at any time (still, follow best practices: make backups first, run salt with test=true).

Download the latest Latest Release from Jira Software. Download in .tar.gz
a. upgrade to latest release
b. Server
c. Latest Release
d. tar.gz archive
Copy to afsxdr-binaries S3 bucket:
aws --profile mdr-common-services-gov s3 cp atlassian-jira-software-9.2.0.tar.gz s3://afsxdr-binaries/jira/
Update msoc-infrastructure/salt/fileroots/jira/init.sls with the latest version and sha-256 Hash.
Optionally, update the okta_jar using the same steps as above (download from Okta Admin > Settings > Downloads > Okta Atlassian Authenticators > Okta Jira Authenticator).
Optionally, update the xmlsec jar using the same steps as above (download from url in init.sls).
Do a PR to develop.
Run the state in test, follow cutover below. salt jira\* state.sls jira --output-diff test=true # not useful; will show lots of errors, since it doesn't download the update salt jira\* state.sls jira --output-diff test=false
Test in test ( https://jira.xdrtest.accenturefederalcyber.com/ ; Configuration is incomplete -- can't even log in )
PR to master
Apply, cutover, and test in prod

Cutover

The salt state does not change to the new version.

To cut over:

Judiciously remove older files in /opt/jira-data/jira/export

Create backups of jira and jira-data

cd /opt/
sudo tar cvzf jira-backup.20220829.tar.gz atlassian jira-data

Run sudo systemctl stop jira
Edit the Jira Linux service controller script - /etc/init.d/jira
Change the path to the updated jira version
sudo systemctl daemon-reload
Run sudo systemctl start jira && sudo tail -F /opt/atlassian/jira-VERSION/logs/catalina.out
Check the Jira Home page and if it warns "Upgrade: Custom changes have not been carried over", click 'ignore and continue'. If it returns to teh same screen, stop and start jira. The saltstate carried over these changes.
sudo systemctl restart jira && sudo tail -F /opt/atlassian/jira-VERSION/logs/catalina.out
Test Jira may redirect you to PROD Jira. Watch your URL!

Cleanup

Once everything is running, there are a few things left:

Clean out the old directory to remove any warnings.

cd /opt/atlassian
rm -rf jira-VERSION # or whatever previous version was

Go to the web interface, to the gear > Jira Administration > Manage Apps
Click on 'Manage Apps' from the new menu
Click on 'Update' on each application that has an update

...

Java Updates

No support in this state yet for updating/installing the JRE. See infrastructure-notes/Jira\ Notes.md for detailed steps, or quick reference:

Error after Upgrading to 8.22.2

There wasn't a lot of useful information in the logs. In /opt/atlassian/jira-8.22.2/logs/catalina.out:

12-May-2022 13:54:49.205 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal One or more Filters failed to start. Full details will be found in the appropriate container log file
12-May-2022 13:54:49.210 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal Context [] startup failed due to previous errors

2022-05-12 13:54:59,133+0000 JIRA-Bootstrap ERROR      [c.a.jira.upgrade.PluginSystemLauncher] A fatal error occured during initialisation. JIRA has been locked.
com.atlassian.jira.InfrastructureException: Error occurred while starting Plugin Manager. null

2022-05-12 13:54:59,420+0000 JIRA-Bootstrap ERROR      [c.a.jira.startup.DefaultJiraLauncher] JIRA has failed to start because of the following errors: [(Event: Level = (EventLevel: fatal) , Key = (EventType: startup-unexpected) , Desc = We couldn't start JIRA , Exception = An error occurred while trying to start JIRA. We can't give you any more detail right now, we suggest checking the logs for more detail and contacting our support team.<br/>See our documentation for more information on contacting our support team and creating a support zip.), (Event: Level = (EventLevel: fatal) , Key = (EventType: database) , Desc = Could not execute health check, DatabaseConfigurationManager not available. , Exception = ]

Solution was here: Jira integrated with OKTA fails to start after upgrading to 8.22.2

NOTE YOU DO NOT NEED TO DO THIS AS IT IS NOW HANDLED BY SALT:

# Updated Repo information for xmlsec as of 29 AUG 2022
"Due to rising energy prices and inflation this website will shut down at the end of this year. Indexing of new Maven dependencies and their versions has been stopped already. It's been fun for the last 7 years, but now it's time to go back to mvnrepository.com or search.maven.org."

Jira Notes.md 22 KB Permalink History Raw