infrastructure-notes/Terragrunt Notes.md

Terragrunt Notes

aka "how to develop the terraform 12+ stuff"

Local cache of providers

NOTE: this doesn't work well with provider locking in TF14+. I recommend you disable this if you've enabled it.

~helpful tip, speed up cache by adding the following to your ~./bashrc:~ ~~ ~export TF_PLUGIN_CACHE_DIR=~/.terraform.d/plugin-cache~ ~[[ -d "$TF_PLUGIN_CACHE_DIR" ]] || mkdir -p $TF_PLUGIN_CACHE_DIR~ ~~

Renaming Directories/Resources

General process:

1. Make sure everything's up to date.
2. Move the remote state.
3. Update the configuration.
4. Rename the directory
5. Make sure terragrunt applies cleanly (But updates all the tags, so lots of changes to review)

For this example, I was renaming 010-standard-vpc to 010-vpc-splunk in test/aws-us-gov/mdr-test-modelclient.

cd 010-standard-vpc/
# clear out cache to make our lives easier
rm -rf .terragrunt-cache
# validate that we're on latest code
terragrunt-local apply
# Get the `bucket` and 'key' value
cat `find . -name 'backend.tf'`
# In this example:
#   bucket         = "afsxdr-terraform-state"
#   key            = "aws/test/aws-us-gov/mdr-test-modelclient/010-standard-vpc/terraform.tfstate"
aws --profile mdr-common-services-gov \
  s3 mv \
   s3://afsxdr-terraform-state/aws/test/aws-us-gov/mdr-test-modelclient/010-standard-vpc/terraform.tfstate \
   s3://afsxdr-terraform-state/aws/test/aws-us-gov/mdr-test-modelclient/010-vpc-splunk/terraform.tfstate
# move and rename
cd ..
git mv 010-standard-vpc 010-vpc-splunk
cd 010-vpc-splunk
# Apply again: NOTE: The only changes should be to the tags. Do not accept any other changes, or you will have extra resources
rm -rf .terragrunt-cache
terragrunt-local apply

If you get:

Error refreshing state: state data in S3 does not have the expected content.

You forgot to rename the directory you're working in.

GitFlow Notes

These notes will walk you through the Terragrunt git flow for making changes.

• Fork the Master branch to your branch
• change local xdr-terrafrom-live repo with expected new tag ( so you don't forget to do it when you are done. )
• make changes to xdr-terraform-modules
• make changes to xdr-terraform-live
• increment the ref=v0.x.x in your terragrunt.hcl
• use terragrunt-local to try the changes
• ( did you run the saml command to login?)
• use tgswitch to change versions
• rm -rf .terragrunt-cache to resolve "strange" errors
• push new branch to github
• get pr approved and merged in
• tag master to latest tag that is set in terragrunt.hcl
• verify it is working in TEST without terragrunt-local
• deploy to PROD
• delete github branch and close jira ticket

Destroy instances

TF_VAR_instance_termination_protection=false terragrunt apply
TF_VAR_instance_termination_protection=false terragrunt destroy

tfswitch.toml

colby-williams taught me: cp -ar to copy symlinks correctly.

ln -s ../../../../.tfswitch.toml .

ls -larth .tfswitch.toml -> ../../../../.tfswitch.toml

2021-04-29: State Issues

When running terragrunt apply, got the following:

Initializing the backend...
Error refreshing state: state data in S3 does not have the expected content.

This may be caused by unusually long delays in S3 processing a previous state
update.  Please wait for a minute or two and try again. If this problem
persists, and neither S3 nor DynamoDB are experiencing an outage, you may need
to manually verify the remote state and update the Digest value stored in the
DynamoDB table to the following value: ec9c9183a070f5ad59b9abd524810c06

The remote state looks uncorrupted:

cd ~/xdr-terraform-live/prod/aws-us-gov/mdr-prod-c2/160-splunk-indexer-cluster
find .terragrunt-cache -name 'backend.tf'
# Use the filename found and view the contents
cat .terragrunt-cache/tC_aGEvkrKzsZjSw0YQum-A6YL8/Ipji28Trjy_fymLhd4EZgtAe8xg/base/splunk_servers/indexer_cluster/backend.tf
# Use the bucket and key to from the s3 path:
scp --profile mdr-common-services-gov cp s3://afsxdr-terraform-state/aws/prod/aws-us-gov/mdr-prod-c2/160-splunk-indexer-cluster/terraform.tfstate
less -iS terraform.tfstate

To fix:

1. go to the gui, log into AWS console to the mdr-common-services-gov account, service dynamodb
2. Go to tables->items
3. Change dropdown to 'query'
4. Into lockId=, enter: afsxdr-terraform-state/aws/prod/aws-us-gov/mdr-prod-c2/160-splunk-indexer-cluster/terraform.tfstate-md5 (The key from above, with -md5 appended)
5. Record the old digest: 9cb9cbfdda
6. Insert the digest from the error message: ec9c9183a0
7. Run terragrunt refresh

TF 0.14 / The State lock File

With tf14, terraform has added the creation of a 'provider state lock file' to prevent inadvertant drift of provider modules. This requires some addition management.

• On first run of a module, create the provider lock file for multiple platforms by running terragrunt-providers (which is just a bash script that runs some cleanup and then runs terragrunt providers lock -platform=darwin_amd64 -platform=linux_amd64 -platform=windows_amd64 -platform=linux_arm64.
• If you need an extra provider, you should override the generation of required_providers.tf in your terragrunt.hcl file for the module. This must include the modules from the root terragrunt.chl that are used within your module. For an example, see xdr-terraform-live/common/aws-us-gov/afs-mdr-common-services-gov/085-codebuild-ecr-customer-portal/terragrunt.hcl
• To regenerate or upgrade modules, I guess you just delete it?
• There is possible compatibility issues with TF_PLUGIN_CACHE_DIR. You can try disabling this if you have trouble getting hashes.