Getting (fatal error: concurrent map writes) every now and then in GH Actions

[a-magdy]

Describe the bug

Sometimes we get the error

1093 fatal error: concurrent map writes

1095 goroutine 409 [running]:
1096 github.com/gruntwork-io/terragrunt/config.copyFieldsMetadata(...)
1097 /home/circleci/project/config/include.go:835
1098 github.com/gruntwork-io/terragrunt/config.(*TerragruntConfig).Merge(0xc00699a900, 0xc0049cbc20, 0x21137d8?)
1099 /home/circleci/project/config/include.go:297 +0xae8

in our github actions, it could happen during any of the steps, init, validate or plan

Steps To Reproduce

Can't really reproduce it, when we get it, we re-run the same workflow again, and it works

fatal error: concurrent map writes

goroutine 409 [running]:
github.com/gruntwork-io/terragrunt/config.copyFieldsMetadata(...)
	/home/circleci/project/config/include.go:835
github.com/gruntwork-io/terragrunt/config.(*TerragruntConfig).Merge(0xc00699a900, 0xc0049cbc20, 0x21137d8?)
	/home/circleci/project/config/include.go:297 +0xae8
github.com/gruntwork-io/terragrunt/config.handleInclude(0xc00061e770, 0xc003ad7680, 0x1)
	/home/circleci/project/config/include.go:133 +0x3a6
github.com/gruntwork-io/terragrunt/config.PartialParseConfig(0xc00061e150, 0xc002599780, 0x20fa846?)
	/home/circleci/project/config/config_partial.go:366 +0xb12
github.com/gruntwork-io/terragrunt/config.TerragruntConfigFromPartialConfig(0xc00061e150, 0xc002599780, 0x0)
	/home/circleci/project/config/config_partial.go:162 +0x356
github.com/gruntwork-io/terragrunt/config.PartialParseConfigFile(0xc00061e150, {0xc003902600, 0x7c}, 0xc005056c00?)
	/home/circleci/project/config/config_partial.go:142 +0x17f
github.com/gruntwork-io/terragrunt/config.decodeAndRetrieveOutputs(0xc000554460, 0xc0038a46c0)
	/home/circleci/project/config/dependency.go:193 +0x385
github.com/gruntwork-io/terragrunt/config.ParseConfig(0xc0007d8000, 0xc0038a46c0, 0xc003456000?)
	/home/circleci/project/config/config.go:795 +0x1ed
github.com/gruntwork-io/terragrunt/config.ParseConfigFile.func1({0x1c80f80?, 0x450f4f?})
	/home/circleci/project/config/config.go:722 +0x3c5

Expected behavior

Terragrunt to run without this random error

Nice to haves

• Terminal output
• Screenshots

Versions

• Terragrunt version: 0.55.21
• OpenTofu/Terraform version: 1.5.7
• Environment details (Ubuntu 20.04, Windows 10, etc.): Github Actions (ubuntu-latest-16-cores)

[denis256]

Fixed in https://github.com/gruntwork-io/terragrunt/releases/tag/v0.58.4

[josh-thisisbud]

I'm getting something very similar to this in 0.58.4
Gist linked due to size of traceback. But below is the starting snippet

fatal error: concurrent map read and map write

goroutine 551 [running]:
github.com/gruntwork-io/terragrunt/util.(*KeyLocks).Unlock(0xc008c0f7c8?, {0xc007910b40?, 0xc008c0f770?})
        /home/circleci/project/util/locks.go:26 +0x2c
github.com/gruntwork-io/terragrunt/config.copyFieldsMetadata(0xc00e11fe60, 0xc0061a4d80)
        /home/circleci/project/config/include.go:844 +0x29b
github.com/gruntwork-io/terragrunt/config.(*TerragruntConfig).Merge(0xc0061a4d80, 0xc00e11fe60, 0x21d0c9f?)
        /home/circleci/project/config/include.go:299 +0x83e
github.com/gruntwork-io/terragrunt/config.handleInclude(0xc000a7eb60, 0xc00f9f8b40, 0x0)
        /home/circleci/project/config/include.go:135 +0x3a6
github.com/gruntwork-io/terragrunt/config.ParseConfig(0xc00c2d6000, 0xc00adbe0c0, 0xc007810000?)
        /home/circleci/project/config/config.go:827 +0x2b0
github.com/gruntwork-io/terragrunt/config.ParseConfigFile.func1({0x4?, 0x3?})
        /home/circleci/project/config/config.go:725 +0x3c5
github.com/gruntwork-io/terragrunt/telemetry.Time({0x2807df0?, 0xc00c2d6000?}, 0xc002ab4258?, {0x212a61e?, 0x1cf82e0?}, 0x1cf82e0?, 0xc002ab4258?)
        /home/circleci/project/telemetry/metrics.go:44 +0x462
github.com/gruntwork-io/terragrunt/config.ParseConfigFile.Telemetry.func2({0x2807df0?, 0xc00c2d6000?})
        /home/circleci/project/telemetry/telemetry.go:77 +0x31
github.com/gruntwork-io/terragrunt/telemetry.Trace({0x2807df0?, 0xc00c2d6000?}, 0x20fd577?, {0x212a61e?, 0x0?}, 0x0?, 0x0?)
        /home/circleci/project/telemetry/*****s.go:40 +0x169
github.com/gruntwork-io/terragrunt/telemetry.Telemetry(...)
        /home/circleci/project/telemetry/telemetry.go:76
github.com/gruntwork-io/terragrunt/config.ParseConfigFile(0xc001082a80, 0xc00c2d6000, {0xc000edaf00, 0x77}, 0x0)
        /home/circleci/project/config/config.go:692 +0x279
github.com/gruntwork-io/terragrunt/config.ReadTerragruntConfig(0xc001082a80)
        /home/circleci/project/config/config.go:682 +0x11c
github.com/gruntwork-io/terragrunt/cli/commands/terraform.runTerraform({0x2807a50?, 0xc000b46360}, 0xc001082a80, 0xc0059a7880)
        /home/circleci/project/cli/commands/terraform/action.go:91 +0x88
github.com/gruntwork-io/terragrunt/cli/commands/terraform.Run({0x2807a50?, 0xc000b46360?}, 0x8?)
        /home/circleci/project/cli/commands/terraform/action.go:79 +0x5c
github.com/gruntwork-io/terragrunt/cli/commands/run-all.NewCommand.action.func1.1({0x2807a50, 0xc000b46360}, 0xc001082a80)
        /home/circleci/project/cli/commands/run-all/command.go:59 +0x13e
github.com/gruntwork-io/terragrunt/configstack.(*runningModule).runNow(0xc002515e50, {0x2807a50, 0xc000b46360}, 0xc002515e50?)
        /home/circleci/project/configstack/running_module.go:263 +0x108
github.com/gruntwork-io/terragrunt/configstack.(*runningModule).runModuleWhenReady.func3({0x16d1154b?, 0x66434a39?})
        /home/circleci/project/configstack/running_module.go:225 +0x25
github.com/gruntwork-io/terragrunt/telemetry.Time({0x2807a50?, 0xc000b46360?}, 0x416c3c?, {0x20f3247?, 0x0?}, 0x0?, 0xc004e9abc0?)
        /home/circleci/project/telemetry/metrics.go:44 +0x462
github.com/gruntwork-io/terragrunt/configstack.(*runningModule).runModuleWhenReady.Telemetry.func5({0x2807a50?, 0xc000b46360?})
        /home/circleci/project/telemetry/telemetry.go:77 +0x31
github.com/gruntwork-io/terragrunt/telemetry.Trace({0x2807a50?, 0xc000b46360?}, 0x2124984?, {0x20f3247?, 0x0?}, 0x0?, 0x994f2c?)
        /home/circleci/project/telemetry/*****s.go:40 +0x169
github.com/gruntwork-io/terragrunt/telemetry.Telemetry(...)
        /home/circleci/project/telemetry/telemetry.go:76
github.com/gruntwork-io/terragrunt/configstack.(*runningModule).runModuleWhenReady(0xc002515e50, {0x2807a50?, 0xc000b46360}, 0xc000a9f500, 0xc0005ecfc0)
        /home/circleci/project/configstack/running_module.go:221 +0x4e5
github.com/gruntwork-io/terragrunt/configstack.runModules.func1(0x0?)
        /home/circleci/project/configstack/running_module.go:184 +0x59
created by github.com/gruntwork-io/terragrunt/configstack.runModules in goroutine 72
        /home/circleci/project/configstack/running_module.go:182 +0xb8

https://gist.github.com/josh-thisisbud/a8f2e781b158968f4a1f69d85ea3c434

[alandreasyans]

@denis256
+1 Getting a similar issue in the 0.58.4 version.
fatal error: concurrent map read and map write
12:25:18 
12:25:18  goroutine 1020 [running]:
12:25:18  github.com/gruntwork-io/terragrunt/util.(*KeyLocks).Unlock(0x4128b2?, {0xc00332ce10?, 0xc001b155f0?})
12:25:18  /home/circleci/project/util/locks.go:26 +0x2c
12:25:18  github.com/gruntwork-io/terragrunt/config.copyFieldsMetadata(0xc001106000, 0xc001511440)
12:25:18  /home/circleci/project/config/include.go:844 +0x29b
12:25:18  github.com/gruntwork-io/terragrunt/config.(*TerragruntConfig).Merge(0xc001511440, 0xc001106000, 0x21d0c9f?)
12:25:18  /home/circleci/project/config/include.go:299 +0x83e
12:25:18  github.com/gruntwork-io/terragrunt/config.handleInclude(0xc00063dd50, 0xc001106000, 0x0)
12:25:18  /home/circleci/project/config/include.go:135 +0x3a6
12:25:18  github.com/gruntwork-io/terragrunt/config.ParseConfig(0xc00054c230, 0xc000306620, 0xc000d24000?)
12:25:18  /home/circleci/project/config/config.go:827 +0x2b0
12:25:18  github.com/gruntwork-io/terragrunt/config.ParseConfigFile.func1({0x4?, 0x3?})
12:25:18  /home/circleci/project/config/config.go:725 +0x3c5
12:25:18  github.com/gruntwork-io/terragrunt/telemetry.Time({0x2807df0?, 0xc00054c230?}, 0xc0022cf458?, {0x212a61e?, 0x1cf82e0?}, 0x1cf82e0?, 0xc0022cf458?)
12:25:18  /home/circleci/project/telemetry/metrics.go:44 +0x462
12:25:18  github.com/gruntwork-io/terragrunt/config.ParseConfigFile.Telemetry.func2({0x2807df0?, 0xc00054c230?})
12:25:18  /home/circleci/project/telemetry/telemetry.go:77 +0x31
12:25:18  github.com/gruntwork-io/terragrunt/telemetry.Trace({0x2807df0?, 0xc00054c230?}, 0x20fd577?, {0x212a61e?, 0x0?}, 0x0?, 0x0?)
12:25:18  /home/circleci/project/telemetry/traces.go:40 +0x169
12:25:18  github.com/gruntwork-io/terragrunt/telemetry.Telemetry(...)
12:25:18  /home/circleci/project/telemetry/telemetry.go:76
12:25:18  github.com/gruntwork-io/terragrunt/config.ParseConfigFile(0xc0001dbc00, 0xc00054c230, {0xc000ad4100, 0x77}, 0x0)
12:25:18  /home/circleci/project/config/config.go:692 +0x279

and etc.

[denis256]

Hello,
published another fix in https://github.com/gruntwork-io/terragrunt/releases/tag/v0.58.5 can you check from your side if it is fixed

[josh-thisisbud]

Will do as soon as our upstream docker images update (probably by tomorrow?) my code is part of our Infra CI and I am required to use the packaged version of Terragrunt for it. @alandreasyans might have better luck testing it out today?

[alandreasyans]

@josh-thisisbud, @denis256 seems the issue is fixed. With a new version, I haven't encountered an issue yet.
Thanks !!