# Jira Notes # Quick Reference: * RDS instance is `jira2`, but database is just `jira` * Started by systemd / `/etc/rc.d/init.d/jira` * Code/binary in `/opt/atlassian/jira` - `JIRA_HOME` set in `/opt/atlassian/jira/atlassian-jira/WEB-INF/classes/jira-application.properties` * Data in `/opt/jira-data/jira` [ JIRA_HOME ] - DB Config in `/opt/jira-data/jira/dbconfig.xml` - Attachments in `/opt/jira/data/jira/attachments` # TLS Setup for RDS ``` # for commercial curl https://s3.amazonaws.com/rds-downloads/rds-ca-2019-root.pem # For govcloud curl https://s3.us-gov-west-1.amazonaws.com/rds-downloads/rds-ca-rsa4096-g1.pem -o root.crt ``` Place output in /home/jira/.postgresql/root.crt ``` vim /opt/jira-data/jira/dbconfig.xml # Add ?sslmode=verify-full to the ``` ~# TLS Setup for RDS - OUTDATED!!! ~ ~First need to update `dbconfig.xml` to tell it to use TLS and what root certs to use: ~ ~``` ~ ~ ~``` ~ ~Then in `/opt/atlassian/jira/rds-root-chain.pem` you need the root cert(s) for RDS. Use something like this: ~ ~``` ~#!/bin/bash ~ ~URLS="https://s3.amazonaws.com/rds-downloads/rds-ca-2019-root.pem" ~URLS="${URLS} https://s3.amazonaws.com/rds-downloads/rds-ca-2015-root.pem" ~URLS="${URLS} https://s3-us-gov-west-1.amazonaws.com/rds-downloads/rds-ca-us-gov-east-1-2017-root.pem" ~URLS="${URLS} https://s3-us-gov-west-1.amazonaws.com/rds-downloads/rds-ca-us-gov-west-1-2017-root.pem" ~ ~rm rds-root-chain.pem ~ ~for i in $URLS; do ~ echo "# `basename $i`" ~ curl -s $i ~done >> rds-root-chain.pem ~ ~ ~``` ~see [Using SSL/TLS with RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html) ~ ~There is mention of ways with newer versions of the PostgreSQL JDBC driver to use the ~standard Java keystore for root certs. This does not work with the version of the JDBC ~driver skipping with Jira version 7.13, as the class needed is missing. (There's no ~DefaultJavaSSLFactory in `postresql-9.4.1212.jar`) One handy trick: ``` openssl s_client -starttls postgres -connect my.postgres.host:5432 # etc... ``` # Proxy setup In ~`JIRA_HOME/bin/setenv.sh`~ `/bin/setenv.sh` ``` JVM_SUPPORT_RECOMMENDED_ARGS=" -Dhttp.proxyHost=proxy.msoc.defpoint.local -Dhttp.proxyPort=80 -Dhttps.proxyHost=proxy.msoc.defpoint.local -Dhttps.proxyPort=80 -Dhttp.nonProxyHosts='*.defpoint.local|localhost|127.0.0.1|169.254.169.254|*.amazonaws.com'" ``` Without this, JIRA cannot download new plugins and things from the Atlassian repositories. # Okta stuff Okta appears to have provided their own SAML implementation for JIRA. Which is weird, I expected JIRA to have their own. [Okta Jira Authenticator 3.x Configuration Guide](https://saml-doc.okta.com/Provisioning_Docs/Okta_Jira_Authenticator_Configuration_Guide.html) There's a config file in `/opt/atlassian/jira/atlassian-jira/WEB-INF/classes/seraph-config.xml` that refers to another config file `/opt/docker/okta-config-jira.xml`. That is where the actual SAML magic is stored. # Load Balancer Stuff There's stuff in `web.xml` that tells it that it's in front of a load balancer. The proxyName and proxyPort settings matter, because they will cause redirects when you connect to the wrong name. Note that in the current config, the load balancer terminates TLS and sends plain HTTP back to JIRA itself. ``` ``` # Useful links [Setting properties and options on startup](https://confluence.atlassian.com/adminjiraserver085/setting-properties-and-options-on-startup-981155694.html) [Change the Base URL of Jira server in the database](https://confluence.atlassian.com/jirakb/change-the-base-url-of-jira-server-in-the-database-733940375.html) # Undockerizing * Fix the split attachments dir * Move attachments out to something like EFS * Load balancer expects to connect to port 80, which is being forwarded by docker to 8080 inside the container. # Migrating to GovCloud ## 0. Stop Jira ``` ssh prod-jira-server sudo systemctl stop jira sudo systemctl disable jira ``` ## 1. Sync original: ``` cd # I believe /opt/jira-data/jira/data/attachments is the correct directory, but it could be # /opt/jira/data/attachments # /opt/jira-data/jira/import/attachments time rsync --rsync-path="sudo rsync" -avz --delete --progress prod-jira-server:/opt/atlassian tmp/atlassian time rsync --rsync-path="sudo rsync" -avz --delete --progress prod-jira-server:/opt/jira-data tmp/jira-data # then restore rsync --rsync-path="sudo rsync" -avz --delete --progress tmp/atlassian/atlassian/ gc-prod-jira-server:/opt/atlassian/ rsync --rsync-path="sudo rsync" -avz --delete --progress tmp/jira-data/jira-data/ gc-prod-jira-server:/opt/jira-data/ ``` ## 2. Database Dump on legacy server: ``` curl https://s3.amazonaws.com/rds-downloads/rds-ca-2019-root.pem -o root.crt mkdir .postgresql mv root.crt .postgresql # Have to install pg11 from the official postgres repositories, but we'll remove it when we're done sudo yum install https://download.postgresql.org/pub/repos/yum/11/redhat/rhel-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm sudo yum install postgresql11 # yes, yes, yes pg_dump postgresql://jira@jira2.csskgjc1suov.us-east-1.rds.amazonaws.com:5432/jira?sslmode=verify-full | gzip > backup.psql.gz # Enter password which you got from /opt/jira-data/jira/dbconfig.xml ``` ## Restore the database NOTE: New DB URL: jira2.csqclvntmsrg.us-gov-east-1.rds.amazonaws.com ``` scp prod-jira-server:backup.psql.gz ./restore.psql.gz scp restore.psql.gz gc-prod-jira-server: ssh gc-prod-jira-server # NOTE: This url is different from above, as there's a differnet cert needed curl https://s3.us-gov-west-1.amazonaws.com/rds-downloads/rds-ca-us-gov-east-1-2017-root.pem -o root.crt mkdir .postgresql mv root.crt .postgresql # Have to install pg11 from the official postgres repositories, but we'll remove it when we're done sudo yum install https://download.postgresql.org/pub/repos/yum/11/redhat/rhel-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm sudo yum install postgresql11 zcat restore.psql.gz | psql --set ON_ERROR_STOP=on --single-transaction postgresql://jira@jira2.csqclvntmsrg.us-gov-east-1.rds.amazonaws.com:5432/jira?sslmode=verify-full # No errors the first time, wtf? ``` To restore a second time: ``` echo 'DROP DATABASE jira' | psql postgresql://jira@jira2.csqclvntmsrg.us-gov-east-1.rds.amazonaws.com:5432/postgres?sslmode=verify-full echo 'CREATE DATABASE jira' | psql postgresql://jira@jira2.csqclvntmsrg.us-gov-east-1.rds.amazonaws.com:5432/postgres?sslmode=verify-full zcat restore.psql.gz | psql --set ON_ERROR_STOP=on --single-transaction postgresql://jira@jira2.csqclvntmsrg.us-gov-east-1.rds.amazonaws.com:5432/jira?sslmode=verify-full ``` ## 3. Install software on the new server: ``` wget https://www.atlassian.com/software/jira/downloads/binary/atlassian-jira-software-9.2.0.tar.gz sudo chmod 755 atlassian-jira-software-9.2.0-x64.bin sudo ./atlassian-jira-software-9.2.0-x64.bin # Yes to install fonts # 1 for express install # /opt/atlassian/jira (default) for install path # /opt/jira-data/jira (NOT THE DEFAULT) for data path # 1 for Default ports # y to service # Summary: # Installation Directory: /opt/atlassian/jira # Home Directory: /opt/jira-data/jira # HTTP Port: 8080 # RMI Port: 8005 # Install as service: Yes # no to not start jira now ``` ## 4. Update Configuration ``` sudo chown -R jira:jira /opt/atlassian /opt/jira-data vim /opt/jira-data/jira/dbconfig.xml # update the db connection string vim /opt/atlassian/jira/conf/server.xml # Update jira.mdr.defpoint.com to jira.xdr.accenturefederalcyber.com vim /opt/atlassian/jira/bin/setenv.sh # Set: # JVM_SUPPORT_RECOMMENDED_ARGS="-Dhttp.proxyHost=proxy.pvt.xdr.accenturefederalcyber.com -Dhttp.proxyPort=80 -Dhttps.proxyHost=proxy.pvt.xdr.accenturefederalcyber.com -Dhttps.proxyPort=80" ``` ``` vim /opt/atlassian/jira/conf/okta-config-jira.xml # copy from okta, remove the header ``` Login and resolve any issues, probably with basename Update mail server. NOTES: new jira db: jira2.csqclvntmsrg.us-gov-east-1.rds.amazonaws.com # Upgrade Notes - 5/2021, 6/29/2021 Manual upgrade, recommended for Linux: - [Upgrading Jira (manual)](https://confluence.atlassian.com/adminjiraserver0813/upgrading-jira-manual-1027137578.html) Tickets: - [MSOCI-1606 - FedRAMP High Upgrade Jira to Resolve Qualys Findings](https://jira.xdr.accenturefederalcyber.com/browse/MSOCI-1606) - [JRASERVER-71652 - CVE-2020-14184](https://jira.atlassian.com/browse/JRASERVER-71652) Current Version: v8.12.1#812002-sha1:7c28d59 Target Version: 5/2021: v8.13.6 (LTS) [Link](https://www.atlassian.com/software/jira/download) 6/2021: v8.13.8 (LTS) 8/2022: v9.2.0 (LTS) Download 'tar.gz' from [Link](https://www.atlassian.com/software/jira/download) Copy to server. ## Prep: Plenty of backups ``` sudo cat /opt/jira-data/jira/dbconfig.xml pg_dump postgresql://jira@jira2.csqclvntmsrg.us-gov-east-1.rds.amazonaws.com:5432/jira?sslmode=verify-full | gzip > backup.202105.psql.gz cd /opt sudo rsync -rav atlassian atlassian.bak.202105 sudo rsync -rav jira-data jira-data.202106 ``` Then in the GUI: 1) Take a snapshot of the instance (be sure to check the box to not restart it) 2) Take a snapshot of the database ## Prep: Extract and Configure ``` cd /opt/atlassian sudo tar xvzf ~/atlassian-jira-software-8.13.6.tar.gz sudo mv atlassian-jira-software-8.13.6-standalone jira-8.13.6 cd jira-8.13.6/ sudo chown -R jira:jira . sudo cp ../jira/atlassian-jira/WEB-INF/lib/okta-jira-3.1.3.jar ./atlassian-jira/WEB-INF/lib/okta-jira-3.1.3.jar sudo cp ../jira/atlassian-jira/okta_login.jsp ./atlassian-jira/okta_login.jsp sudo cp ../jira/conf/okta-config-jira.xml ./conf/okta-config-jira.xml sudo vim atlassian-jira/WEB-INF/classes/jira-application.properties # set jira.home to /opt/jira-data/jira sudo vim bin/setenv.sh # set JIRA_HOME (maybe? It wasn't set in the old one, but i'm setting it) # Copy JVM_SUPPORT_RECOMMENDED_ARGS from old copy # Note: previous setting JVM_MAXIMUM_MEMORY="768m" is now JVM_MAXIMUM_MEMORY="2048m" ``` In `atlassian-jira/WEB-INF/web.xml`, just before `THIS MUST BE THE LAST FILTER IN THE DEFINED CHAIN`, add the following: ``` OktaLoginFilter com.atlassian.jira.authenticator.okta.OktaLoginFilter OktaLoginFilter /* REQUEST FORWARD ``` In `conf/server.xml` in the first 'connector' clause, update the last two lines (only chagne on the first line is the `/>`):: ``` acceptCount="100" disableUploadTimeout="true" bindOnInit="false" proxyName="jira.xdr.accenturefederalcyber.com" proxyPort="443" scheme="https" secure="true"/> ``` In `atlassian-jira/WEB-INF/classes/seraph-config.xml`: 1. For `logout.url`, set: ``` https://mdr-multipass.okta.com ``` 2. Comment out the following: ``` ``` 3. After the last ``, add the following section (note the fixed path): ``` okta.config.file /opt/atlassian/jira-8.13.6/conf/okta-config-jira.xml ``` Update okta to latest (optional): 1. Log into okta admin 2. go to settings->downloads 3. Download the latest okta plugin 4. Copy to the server ``` ssh gc-prod-okta-server cd /opt/atlassian/jira-8.13.6 sudo rm atlassian-jira/WEB-INF/lib/okta-jira-3.1.3.jar sudo mv ~frederick_t_damstra/okta-jira-3.1.5.jar atlassian-jira/WEB-INF/lib/ sudo chown jira:jira atlassian-jira/WEB-INF/lib/okta-jira-3.1.5.jar sudo chmod 644 atlassian-jira/WEB-INF/lib/okta-jira-3.1.5.jar ``` ## Cutover: ``` sudo /etc/init.d/jira stop # In GUI, take db snapshot cd /opt sudo rsync -rav atlassian atlassian.bak.202105 sudo rsync -rav jira-data jira-data.202106 sudo mv /opt/atlassian/jira /opt/atlassian/jira-8.12.1 vim /etc/rc.d/init.d/jira # Update path to /opt/atlassian/jira/bin sudo /etc/init.d/jira start ``` Note, the 'check upgrade' step, suggested that these might have been modified, too: ``` jira-application.properties WEB-INF/web.xml seraph-config.xml ``` Also, if okta doesn't work, you can try placing okta back: ``` sudo cp /opt/atlassian/jira/atlassian-jira/WEB-INF/lib/okta-jira-3.1.3.jar /opt/atlassian/jira-8.13.6/atlassian-jira/WEB-INF/lib/okta-jira-3.1.3.jar ``` ## Problems during 4/27/2021 upgrade: ### Jira wouldn't start because JAVA_HOME was not yet. 1. edit bin/setenv.sh, search for JAVA_HOME and replace with: ``` JAVA_HOME="/opt/atlassian/jre/"; export JAVA_HOME ``` 2. Copy the old JRE (and I bet this needs to be updated) ### Jira started, but database won't connect. This was because of SSL. 1. temporarily removed the verify-ssl command from /opt/jira-data/jira/dbconfig.xml 2. It started. 3. Stopped it after it initialized 1. copied /home/jira/.posgres to /opt/jira-data/jira/.postgres (this contains the root cert) no change Tried /opt/jira-data/.postgres, too. Left it with `sslmode=require`. ### Noisy Catalina error: `The encoding [binary] is not recognised by the JRE`, with full stack trace: see [JRASERVER - 71265 - The encoding [binary] is not recognised by the JRE](https://jira.atlassian.com/browse/JRASERVER-71265) added 3 lines to `/opt/atlassian/jira-8.13.6/conf/logging.properties`: ``` # per https://jira.atlassian.com/browse/JRASERVER-71265 # Surpress 'The encoding [binary] is not recognised by the JRE' org.apache.catalina.connector.Response.level = ERROR ``` ## Problems during 6/29/2021 Upgrade The startup says `Unsupported collation error` after update. This page [Unsupported collation error thrown in Jira server using PostgreSQL](https://confluence.atlassian.com/jirakb/unsupported-collation-error-thrown-in-jira-server-using-postgresql-776657961.html) says it's right. We have `en_US.UTF-8` instead of one of the supported ones. However, I was able to resolve by editing `/opt/jira-data/jira/dbconfig.xml` and temporarily removing `?sslmode=require` from teh connection string. After stopping and starting, I re-added this option and stopped nad started again, and the error was gone. ## If all is well do a little dance # Updating JRE If Jira was installed via the tarball (as in the Govcloud move above), Java is not updated as part of the package and may need to be updated separately. To update: 1. Download from AdoptOpenJDK from [Eclipse Temurin version 11](https://adoptium.net/temurin/releases/?version=11) a. Do *NOT* download from oracle.com, as that requires a commercial license. b. Select "Temurin 11 (LTS)", "Linux", "x64", and download the JRE, not the JDK 2. Copy the file to the server. E.g.: ``` tshp scp OpenJDK11U-jre_x64_linux_hotspot_11.0.16.1_1.tar.gz jira-server.pvt.xdr.accenturefederalcyber.com: ``` 3. Extract and copy the JRE into a temporary location ``` tshp jira-server tar xvzf OpenJDK11U-jre_x64_linux_hotspot_11.0.16.1_1.tar.gz sudo mv jdk-11.0.16.1+1-jre/ /opt/atlassian/jre-11.0.16.1+1 sudo chown -R jira:jira /opt/atlassian/jre-11.0.16.1+1 sudo find /opt/atlassian/jre-11.0.16.1+1 -type d -exec chmod o+rx {} \; sudo find /opt/atlassian/jre-11.0.16.1+1 -type f -exec chmod o+r {} \; sudo find /opt/atlassian/jre-11.0.16.1+1/bin -type f -exec chmod o+rx {} \; /opt/atlassian/jre-11.0.16.1+1/bin/java -version ``` 4. Back up the old jre: ``` cd /opt/atlassian sudo cp -a jre jre.20220830 ``` 5. Install the updated JRE ``` cd /opt/atlassian sudo /etc/rc.d/init.d/jira stop sudo mv jre jre.old sudo mv jre-11.0.16.1+1 jre sudo /etc/rc.d/init.d/jira start ``` # Upgrading A salt state can be used for upgrading. See `msoc-infrastructure/salt/fileroots/jira/README.md`. ## Prepare 1. Go to Jira, click on teh Options > Applications option. 1. Click on "Plan your upgrade" 1. Review anything that looks dangerous. ## Staging the Update The Salt State will install the update into a new directory `/opt/atlassian/jira-{version}`, but will not stop/start it. You should be able to safely run this at any time (still, follow best practices: make backups first, run salt with `test=true`). 1. Download the latest Latest Release from [Jira Software](https://www.atlassian.com/software/jira/download-journey). Download in .tar.gz a. upgrade to latest release b. Server c. Latest Release d. tar.gz archive 1. Copy to afsxdr-binaries S3 bucket: `aws --profile mdr-common-services-gov s3 cp atlassian-jira-software-9.2.0.tar.gz s3://afsxdr-binaries/jira/` 1. Update `msoc-infrastructure/salt/fileroots/jira/init.sls` with the latest version and sha-256 Hash. 1. Optionally, update the `okta_jar` using the same steps as above (download from Okta Admin > Settings > Downloads > Okta Atlassian Authenticators > Okta Jira Authenticator). 1. Optionally, update the `xmlsec` jar using the same steps as above (download from url in `init.sls`). 1. Do a PR to develop. 1. Run the state in test, follow cutover below. `salt jira\* state.sls jira --output-diff test=true` # not useful; will show lots of errors, since it doesn't download the update `salt jira\* state.sls jira --output-diff test=false` 1. Test in test ( https://jira.xdrtest.accenturefederalcyber.com/ ; Configuration is incomplete -- can't even log in ) 1. PR to master 1. Apply, cutover, and test in prod ## Cutover The salt state does not change to the new version. To cut over: 1. Judiciously remove older files in `/opt/jira-data/jira/export` 1. Create backups of jira and jira-data ``` cd /opt/ sudo tar cvzf jira-backup.20220829.tar.gz atlassian jira-data ``` 1. Run `sudo systemctl stop jira` 1. Edit the Jira Linux service controller script - `/etc/init.d/jira` 1. Change the path to the updated jira version 1. `sudo systemctl daemon-reload` 1. Run `sudo systemctl start jira && sudo tail -F /opt/atlassian/jira-VERSION/logs/catalina.out` 1. Check the Jira Home page and if it warns "Upgrade: Custom changes have not been carried over", click 'ignore and continue'. If it returns to teh same screen, stop and start jira. The saltstate carried over these changes. `sudo systemctl restart jira && sudo tail -F /opt/atlassian/jira-VERSION/logs/catalina.out` 1. Test Jira may redirect you to PROD Jira. Watch your URL! ## Cleanup Once everything is running, there are a few things left: 1. Clean out the old directory to remove any warnings. ``` cd /opt/atlassian rm -rf jira-VERSION # or whatever previous version was ``` 1. Go to the web interface, to the gear > Jira Administration > Manage Apps 1. Click on 'Manage Apps' from the new menu 1. Click on 'Update' on each application that has an update ... ## Java Updates No support in this state yet for updating/installing the JRE. See `infrastructure-notes/Jira\ Notes.md` for detailed steps, or quick reference: ## Error after Upgrading to 8.22.2 There wasn't a lot of useful information in the logs. In `/opt/atlassian/jira-8.22.2/logs/catalina.out`: ``` 12-May-2022 13:54:49.205 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal One or more Filters failed to start. Full details will be found in the appropriate container log file 12-May-2022 13:54:49.210 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal Context [] startup failed due to previous errors ``` ``` 2022-05-12 13:54:59,133+0000 JIRA-Bootstrap ERROR [c.a.jira.upgrade.PluginSystemLauncher] A fatal error occured during initialisation. JIRA has been locked. com.atlassian.jira.InfrastructureException: Error occurred while starting Plugin Manager. null ``` ``` 2022-05-12 13:54:59,420+0000 JIRA-Bootstrap ERROR [c.a.jira.startup.DefaultJiraLauncher] JIRA has failed to start because of the following errors: [(Event: Level = (EventLevel: fatal) , Key = (EventType: startup-unexpected) , Desc = We couldn't start JIRA , Exception = An error occurred while trying to start JIRA. We can't give you any more detail right now, we suggest checking the logs for more detail and contacting our support team.
See our documentation for more information on contacting our support team and creating a support zip.), (Event: Level = (EventLevel: fatal) , Key = (EventType: database) , Desc = Could not execute health check, DatabaseConfigurationManager not available. , Exception = ] ``` Solution was here: [Jira integrated with OKTA fails to start after upgrading to 8.22.2](https://confluence.atlassian.com/jirakb/jira-integrated-with-okta-fails-to-start-after-upgraging-to-8-22-2-1127254525.html) **NOTE** YOU DO NOT NEED TO DO THIS AS IT IS NOW HANDLED BY SALT: ``` # Updated Repo information for xmlsec as of 29 AUG 2022 "Due to rising energy prices and inflation this website will shut down at the end of this year. Indexing of new Maven dependencies and their versions has been stopped already. It's been fun for the last 7 years, but now it's time to go back to mvnrepository.com or search.maven.org." ```