Merge branch 'master' of github.xdr.accenturefederalcyber.com:mdr-engineering/infrastructure-notes

Jira Notes.md

@@ -71,7 +71,7 @@ openssl s_client -starttls postgres -connect my.postgres.host:5432 # etc...
 
 # Proxy setup
 
-In `JIRA_HOME/bin/setenv.sh`
+In ~`JIRA_HOME/bin/setenv.sh`~ `<jira binaries directory>/bin/setenv.sh`
 
 ```
 JVM_SUPPORT_RECOMMENDED_ARGS=" -Dhttp.proxyHost=proxy.msoc.defpoint.local -Dhttp.proxyPort=80 -Dhttps.proxyHost=proxy.msoc.defpoint.local -Dhttps.proxyPort=80 -Dhttp.nonProxyHosts='*.defpoint.local|localhost|127.0.0.1|169.254.169.254|*.amazonaws.com'"
@@ -245,3 +245,170 @@ NOTES:
 new jira db:
 jira2.csqclvntmsrg.us-gov-east-1.rds.amazonaws.com
 
+# Upgrade Notes - 5/2021
+
+Manual upgrade, recommended for Linux: https://confluence.atlassian.com/adminjiraserver0813/upgrading-jira-manual-1027137578.html
+
+Tickets: https://jira.xdr.accenturefederalcyber.com/browse/MSOCI-1606
+         https://jira.atlassian.com/browse/JRASERVER-71652
+
+Current Version: v8.12.1#812002-sha1:7c28d59
+Target Version:  v8.13.6 (LTS)  https://www.atlassian.com/software/jira/download
+
+Download 'tar.gz' from https://www.atlassian.com/software/jira/download
+Copy to server.
+
+## Prep: Plenty of backups
+```
+sudo cat /opt/jira-data/jira/dbconfig.xml
+pg_dump postgresql://jira@jira2.csqclvntmsrg.us-gov-east-1.rds.amazonaws.com:5432/jira?sslmode=verify-full | gzip > backup.202105.psql.gz
+cd /opt
+sudo rsync -rav atlassian atlassian.bak.202105
+sudo rsync -rav jira-data jira-data.202106
+```
+
+Then in the GUI:
+1) Take a snapshot of the instance (be sure to check the box to not restart it)
+2) Take a snapshot of the database
+
+## Prep: Extract and Configure
+
+```
+cd /opt/atlassian
+sudo tar xvzf ~/atlassian-jira-software-8.13.6.tar.gz
+sudo mv atlassian-jira-software-8.13.6-standalone jira-8.13.6
+cd jira-8.13.6/
+sudo chown -R jira:jira .
+sudo cp ../jira/atlassian-jira/WEB-INF/lib/okta-jira-3.1.3.jar ./atlassian-jira/WEB-INF/lib/okta-jira-3.1.3.jar
+sudo cp ../jira/atlassian-jira/okta_login.jsp ./atlassian-jira/okta_login.jsp
+sudo cp ../jira/conf/okta-config-jira.xml ./conf/okta-config-jira.xml
+
+sudo vim atlassian-jira/WEB-INF/classes/jira-application.properties
+# set jira.home to /opt/jira-data/jira
+sudo vim bin/setenv.sh
+# set JIRA_HOME (maybe? It wasn't set in the old one, but i'm setting it)
+# Copy JVM_SUPPORT_RECOMMENDED_ARGS from old copy
+# Note: previous setting JVM_MAXIMUM_MEMORY="768m" is now JVM_MAXIMUM_MEMORY="2048m"
+```
+
+In `atlassian-jira/WEB-INF/web.xml`, just before `THIS MUST BE THE LAST FILTER IN THE DEFINED CHAIN`, add the following:
+```
+    <filter>
+        <filter-name>OktaLoginFilter</filter-name>
+        <filter-class>com.atlassian.jira.authenticator.okta.OktaLoginFilter</filter-class>
+    </filter>
+
+    <filter-mapping>
+        <filter-name>OktaLoginFilter</filter-name>
+        <url-pattern>/*</url-pattern>
+        <dispatcher>REQUEST</dispatcher>
+        <dispatcher>FORWARD</dispatcher>
+    </filter-mapping>
+```
+
+In `conf/server.xml` in the first 'connector' clause, update the last two lines (only chagne on the first line is the `/>`):: 
+```
+acceptCount="100" disableUploadTimeout="true" bindOnInit="false"
+proxyName="jira.xdr.accenturefederalcyber.com" proxyPort="443" scheme="https" secure="true"/>
+```
+
+In `atlassian-jira/WEB-INF/classes/seraph-config.xml`:
+
+1. For `<param-name>logout.url</param-name>`, set:
+```
+<param-value>https://mdr-multipass.okta.com</param-value>
+```
+
+2. Comment out the following:
+```
+    <!-- <authenticator class="com.atlassian.jira.security.login.JiraSeraphAuthenticator"/> -->
+```
+
+3. After the last `<!-- CROWD:END -->`, add the following section (note the fixed path):
+```
+    <!-- MDR OKTA:START -->
+
+        <authenticator class="com.atlassian.jira.authenticator.okta.OktaJiraAuthenticator30">
+            <init-param>
+                <param-name>okta.config.file</param-name>
+                <param-value>/opt/atlassian/jira-8.13.6/conf/okta-config-jira.xml</param-value>
+            </init-param>
+        </authenticator>
+
+    <!-- MDR OKTA:END -->
+```
+
+Update okta to latest (optional):
+1. Log into okta admin
+2. go to settings->downloads
+3. Download the latest okta plugin
+4. Copy to the server
+
+```
+ssh gc-prod-okta-server
+cd /opt/atlassian/jira-8.13.6
+sudo rm atlassian-jira/WEB-INF/lib/okta-jira-3.1.3.jar
+sudo mv ~frederick_t_damstra/okta-jira-3.1.5.jar atlassian-jira/WEB-INF/lib/
+sudo chown jira:jira atlassian-jira/WEB-INF/lib/okta-jira-3.1.5.jar
+sudo chmod 644 atlassian-jira/WEB-INF/lib/okta-jira-3.1.5.jar
+```
+
+## Cutover:
+
+```
+sudo /etc/init.d/jira stop
+# In GUI, take db snapshot
+cd /opt
+sudo rsync -rav atlassian atlassian.bak.202105
+sudo rsync -rav jira-data jira-data.202106
+sudo mv /opt/atlassian/jira /opt/atlassian/jira-8.12.1
+vim /etc/rc.d/init.d/jira
+# Update path to /opt/atlassian/jira/bin
+sudo /etc/init.d/jira start
+```
+
+Note, the 'check upgrade' step, suggested that these might have been modified, too:
+```
+jira-application.properties
+WEB-INF/web.xml
+seraph-config.xml
+``` 
+
+Also, if okta doesn't work, you can try placing okta back:
+```
+sudo cp /opt/atlassian/jira/atlassian-jira/WEB-INF/lib/okta-jira-3.1.3.jar /opt/atlassian/jira-8.13.6/atlassian-jira/WEB-INF/lib/okta-jira-3.1.3.jar
+```
+
+## Problems during 4/27/2021 upgrade:
+
+### Jira wouldn't start because JAVA_HOME was not yet. 
+1. edit bin/setenv.sh, search for JAVA_HOME and replace with:
+```
+JAVA_HOME="/opt/atlassian/jre/"; export JAVA_HOME
+```
+2. Copy the old JRE (and I bet this needs to be updated)
+
+### Jira started, but database won't connect. This was because of SSL.
+1. temporarily removed the verify-ssl command from /opt/jira-data/jira/dbconfig.xml
+2. It started.
+3. Stopped it after it initialized
+
+1. copied /home/jira/.posgres to /opt/jira-data/jira/.postgres (this contains the root cert)
+no change
+Tried /opt/jira-data/.postgres, too.
+
+Left it with `sslmode=require`.
+
+### Noisy catalina error: `The encoding [binary] is not recognised by the JRE`, with full stack trace:
+
+see https://jira.atlassian.com/browse/JRASERVER-71265
+added 3 lines to /opt/atlassian/jira-8.13.6/conf/logging.properties:
+```
+# per https://jira.atlassian.com/browse/JRASERVER-71265
+# Surpress 'The encoding [binary] is not recognised by the JRE'
+org.apache.catalina.connector.Response.level = ERROR
+```
+ 
+## If all is well
+
+do a little dance

Phantom Notes.md

@@ -217,7 +217,6 @@ sudo phenv python3 /opt/phantom/bin/ibackup.pyc --setup
 1. Stop phantom and create the last backup
 ```
 ssh dev-phantom
-/opt/phantom/bin/start_phantom.sh
 time sudo phenv python3 /opt/phantom/bin/ibackup.pyc --backup
 sudo ls -l /opt/phantom/data/backup/
 sudo /opt/phantom/bin/stop_phantom.sh
@@ -268,14 +267,14 @@ go to administration->app settings, update the proxy to `http://proxy.pvt.xdrtes
 administration->user management->authentication->saml2
   Record original values:
 ```
-    SSO Url: https://mdr-multipass.okta.com/app/mdrmultipass_testphantom_1/exk1oe0itkmsUtNoB297/sso/saml
-    New URL: https://mdr-multipass.okta.com/app/mdr-multipass_phantomtestgc_1/exk7wltztzufef0f1297/sso/saml
-    Issuer ID: http://www.okta.com/exk1oe0itkmsUtNoB297
-    New ID   : http://www.okta.com/exk7wltztzufef0f1297
+    SSO Url: https://mdr-multipass.okta.com/app/mdrmultipass_mdrphantom_1/exk1m6x7ri1WgvXCB297/sso/saml
+    New URL: 
+    Issuer ID: http://www.okta.com/exk1m6x7ri1WgvXCB297
+    New ID   : 
     Base URL: https://phantom.msoc.defpoint.local
-    New URL:  https://phantom.pvt.xdrtest.accenturefederalcyber.com
+    New URL:  
     Metadata:
-<?xml version="1.0" encoding="UTF-8"?><md:EntityDescriptor xmlns:md="urn:oasis:names:tc:SAML:2.0:metadata" entityID="http://www.okta.com/exk1oe0itkmsUtNoB297"><md:IDPSSODescriptor WantAuthnRequestsSigned="false" protocolSupportEnumeration="urn:oasis:names:tc:SAML:2.0:protocol"><md:KeyDescriptor use="signing"><ds:KeyInfo xmlns:ds="http://www.w3.org/2000/09/xmldsig#"><ds:X509Data><ds:X509Certificate>MIIDqjCCApKgAwIBAgIGAWrbB00GMA0GCSqGSIb3DQEBCwUAMIGVMQswCQYDVQQGEwJVUzETMBEG
+<?xml version="1.0" encoding="UTF-8"?><md:EntityDescriptor xmlns:md="urn:oasis:names:tc:SAML:2.0:metadata" entityID="http://www.okta.com/exk1m6x7ri1WgvXCB297"><md:IDPSSODescriptor WantAuthnRequestsSigned="false" protocolSupportEnumeration="urn:oasis:names:tc:SAML:2.0:protocol"><md:KeyDescriptor use="signing"><ds:KeyInfo xmlns:ds="http://www.w3.org/2000/09/xmldsig#"><ds:X509Data><ds:X509Certificate>MIIDqjCCApKgAwIBAgIGAWrbB00GMA0GCSqGSIb3DQEBCwUAMIGVMQswCQYDVQQGEwJVUzETMBEG
 A1UECAwKQ2FsaWZvcm5pYTEWMBQGA1UEBwwNU2FuIEZyYW5jaXNjbzENMAsGA1UECgwET2t0YTEU
 MBIGA1UECwwLU1NPUHJvdmlkZXIxFjAUBgNVBAMMDW1kci1tdWx0aXBhc3MxHDAaBgkqhkiG9w0B
 CQEWDWluZm9Ab2t0YS5jb20wHhcNMTkwNTIxMTUzMzA5WhcNMjkwNTIxMTUzNDA5WjCBlTELMAkG
@@ -291,12 +290,14 @@ AQCKqio8wrvhbkGRptCD6sEnRmC7/NBE133tIv7Z3R/Cve8DgO3GcKKrCUh+gZJLFV3eWw95FTWW
 MY7KrYEd353mKP8hL7mEc+qSmWuwfFw+6JePHsNDiFKCY2PfzbWgsG9nX7T6H7n8cn2hzVn4gBmb
 8TAXei+x0id9h24oSvtISZhMg+ED72c0BbO4wPZOQeisXPO4vugdRdbyB5wvIU2ILHb7WJnDNSai
 XSHqKUBigvQua2KSjh+GW7fMlvRbDkYxq3okj6sZlyCLN79IM4NZgKfCC4t8FoUA9ofIDUV9u70G
-+Utb6eeVogPzFlv4LuMRAEKbnV9G3yyDbxYsEcpY</ds:X509Certificate></ds:X509Data></ds:KeyInfo></md:KeyDescriptor><md:NameIDFormat>urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress</md:NameIDFormat><md:NameIDFormat>urn:oasis:names:tc:SAML:2.0:nameid-format:persistent</md:NameIDFormat><md:SingleSignOnService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST" Location="https://mdr-multipass.okta.com/app/mdrmultipass_testphantom_1/exk1oe0itkmsUtNoB297/sso/saml"/><md:SingleSignOnService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect" Location="https://mdr-multipass.okta.com/app/mdrmultipass_testphantom_1/exk1oe0itkmsUtNoB297/sso/saml"/></md:IDPSSODescriptor></md:EntityDescriptor>
++Utb6eeVogPzFlv4LuMRAEKbnV9G3yyDbxYsEcpY</ds:X509Certificate></ds:X509Data></ds:KeyInfo></md:KeyDescriptor><md:NameIDFormat>urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress</md:NameIDFormat><md:NameIDFormat>urn:oasis:names:tc:SAML:2.0:nameid-format:persistent</md:NameIDFormat><md:SingleSignOnService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST" Location="https://mdr-multipass.okta.com/app/mdrmultipass_mdrphantom_1/exk1m6x7ri1WgvXCB297/sso/saml"/><md:SingleSignOnService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect" Location="https://mdr-multipass.okta.com/app/mdrmultipass_mdrphantom_1/exk1m6x7ri1WgvXCB297/sso/saml"/></md:IDPSSODescriptor></md:EntityDescriptor>
 ```
 Update saml with settings from the saml provider metadata (available from okta, application, login settings)
 
 Log out and log back in via okta.
 
+Run the backup prep.
+
 ---
 I got 500: Server Error.
 
@@ -306,8 +307,6 @@ Double checked saml config
 Set hostname and fqdn in administration->company settings
 
 
-
-
 4.  Start phantom:
 ```
 sudo /opt/phantom/bin/stop_phantom.sh

Terragrunt Notes.md

@@ -3,11 +3,14 @@ aka "how to develop the terraform 12+ stuff"
 
 
 ## Local cache of providers
-helpful tip, speed up cache by adding the following to your `~./bashrc`:
-```
-export TF_PLUGIN_CACHE_DIR=~/.terraform.d/plugin-cache
-[[ -d "$TF_PLUGIN_CACHE_DIR" ]] || mkdir -p $TF_PLUGIN_CACHE_DIR
-```
+
+NOTE: this doesn't work well with provider locking in TF14+. I recommend you disable this if you've enabled it.
+
+~helpful tip, speed up cache by adding the following to your `~./bashrc`:~
+~```~
+~export TF_PLUGIN_CACHE_DIR=~/.terraform.d/plugin-cache~
+~[[ -d "$TF_PLUGIN_CACHE_DIR" ]] || mkdir -p $TF_PLUGIN_CACHE_DIR~
+~```~
 
 ## Renaming Directories/Resources
 
@@ -86,4 +89,47 @@ colby-williams taught me: cp -ar to copy symlinks correctly.
 ln -s ../../../../.tfswitch.toml .
 
 ls -larth
-.tfswitch.toml -> ../../../../.tfswitch.toml
+.tfswitch.toml -> ../../../../.tfswitch.toml
+
+### 2021-04-29: State Issues
+
+When running `terragrunt apply`, got the following:
+```
+Initializing the backend...
+Error refreshing state: state data in S3 does not have the expected content.
+
+This may be caused by unusually long delays in S3 processing a previous state
+update.  Please wait for a minute or two and try again. If this problem
+persists, and neither S3 nor DynamoDB are experiencing an outage, you may need
+to manually verify the remote state and update the Digest value stored in the
+DynamoDB table to the following value: ec9c9183a070f5ad59b9abd524810c06
+```
+
+The remote state looks uncorrupted:
+```
+cd ~/xdr-terraform-live/prod/aws-us-gov/mdr-prod-c2/160-splunk-indexer-cluster
+find .terragrunt-cache -name 'backend.tf'
+# Use the filename found and view the contents
+cat .terragrunt-cache/tC_aGEvkrKzsZjSw0YQum-A6YL8/Ipji28Trjy_fymLhd4EZgtAe8xg/base/splunk_servers/indexer_cluster/backend.tf
+# Use the bucket and key to from the s3 path:
+scp --profile mdr-common-services-gov cp s3://afsxdr-terraform-state/aws/prod/aws-us-gov/mdr-prod-c2/160-splunk-indexer-cluster/terraform.tfstate
+less -iS terraform.tfstate
+```
+
+To fix:
+1. go to the gui, log into AWS console to the mdr-common-services-gov account, service dynamodb
+2. Go to tables->items
+3. Change dropdown to 'query'
+4. Into lockId=, enter: afsxdr-terraform-state/aws/prod/aws-us-gov/mdr-prod-c2/160-splunk-indexer-cluster/terraform.tfstate-md5 (The key from above, with -md5 appended)
+5. Record the old digest: 9cb9cbfddaf100cfa8ae92ec79236175
+6. Insert the digest from the error message: ec9c9183a070f5ad59b9abd524810c06
+7. Run `terragrunt refresh`
+
+## TF 0.14 / The State lock File
+
+With tf14, terraform has added the creation of a 'provider state lock file' to prevent inadvertant drift of provider modules. This requires some addition management.
+
+* On first run of a module, create the provider lock file for multiple platforms by running `terragrunt-providers` (which is just a bash script that runs some cleanup and then runs `terragrunt providers lock -platform=darwin_amd64 -platform=linux_amd64 -platform=windows_amd64 -platform=linux_arm64`.
+* If you need an extra provider, you should override the generation of `required_providers.tf` in your `terragrunt.hcl` file for the module. This must include the modules from the root `terragrunt.chl` that are used within your module. For an example, see `xdr-terraform-live/common/aws-us-gov/afs-mdr-common-services-gov/085-codebuild-ecr-customer-portal/terragrunt.hcl`
+* To regenerate or upgrade modules, I guess you just delete it?
+* There is possible compatibility issues with `TF_PLUGIN_CACHE_DIR`. You can try disabling this if you have trouble getting hashes.