Repository navigation
feat(terraform): prune resources outside the target block closure - #74
Conversation
EvaluateAll evaluates every resource and module in a module on each call. Callers that only need a subset of a module's output (for example coder/preview, which computes a workspace's input parameters) pay to evaluate resources whose values can never affect that output. OptionWithResourceClosure(targetTypes) restricts root-module evaluation to the resource blocks reachable, via references, from the given target block types (matched on a block's type label, e.g. "coder_parameter"). Every non-resource block is retained and submodules are evaluated in full, so the computed values of the target blocks are unchanged; only resources that nothing in the target closure references are dropped. The closure is conservative: a reference that cannot be resolved to a concrete block keeps the matching blocks, so a resource is excluded only when nothing a target block reads can reference it. Empty targetTypes disables the behavior (default).
fa2a879 to
3916002
Compare
Preview evaluates the entire Terraform module graph on every call, even though rendering a workspace form only needs coder_parameter, coder_workspace_preset and coder_workspace_tags (and what they reference). The resources a workspace would create cannot feed those blocks, so evaluating them is wasted work that dominates request latency on large templates. Pass OptionWithResourceClosure with the three target block types so the parser drops root-module resources that nothing in that closure references. On a real template this cuts EvaluateAll from ~2s to ~0.16s (~12x) with byte-identical parameters, presets and tags. Depends on the OptionWithResourceClosure addition in the trivy fork (coder/trivy#74). The trivy replace is temporarily pinned to that PR's commit; it will be moved to the merged coder/trivy commit before this merges.
| // References that cannot be resolved to a concrete block simply fail to match | ||
| // and prune nothing extra, so ambiguity always errs toward keeping resources. |
A resource referenced only as my_resource.x[0] or my_resource.x["k"] is currently pruned because Reference.RefersTo treats the reference key as significant while the unexpanded resource block has none. This test fails on the PR head and documents the expected behavior.
Pruning runs before count/for_each expansion, so resource blocks carry no key while references like my_resource.x[0] do. Reference.RefersTo treats that mismatch as a different block, pruning resources the target closure depends on. Compare block type and labels only.
…arg references in closure pruning Extends the resource-closure test set with cases that reach a resource through reference shapes the pruner must not treat as a different block: - splat (my_resource.web[*].name) - variable index (my_resource.pool[var.idx].name) - a parameter's nested option block - a module input argument Each asserts the referenced resource survives pruning (the parameter default still resolves) while an unrelated orphan is pruned. All pass on the current pruner, guarding reference matching against regressions.
|
@Emyrk pulled your indexed-ref fix in, thanks 🙏 Added a few more closure tests on top to lock down the other reference shapes: splat (x[*]), variable index (x[var.i]), a nested block ref, and a module input arg. All green, so the indexed-key case looks like the only spot that was over-pruning. |
|
One more reference shape where the closure is not conservative:
Repro (parser package): {
"data": {"coder_parameter": {"p": {"name": "p", "default": "${local.v}"}}},
"locals": {"v": "${my_resource.a.name}-${my_resource.b.name}"},
"resource": {"my_resource": {"a": {"name": "A"}, "b": {"name": "B"}}}
}Without the option: 2 resources kept, default The merging in the JSON path predates this PR, but pruning is what turns it from a quirk into wrong output. Iterating
|
…runer The closure pruner repeated the "resource" string literal, which tips the file over goconst's threshold on new lines. Use a local constant instead.
JSON templates (.tf.json / .tofu.json) can merge multiple references in a single expression into one reference during extraction (see Attribute.referencesFromExpression), which would let the closure drop a resource a target depends on. Reference resolution there is not reliable enough to prune safely, so keep everything when any source file is JSON. Adds a regression test (parameter default referencing two resources in one expression must resolve identically to an unpruned run).
The PR's lint (new-vs-main) surfaces a pre-existing revive unused-parameter on this eval-hook callback. The parameter is part of the fixed OptionWithEvalHook signature, so keep the descriptive name and annotate with //nolint:revive rather than renaming to _.
|
Handled the .tf.json case 👍 Went the conservative route (your option B): the pruner now skips entirely when any source file is .tf.json/.tofu.json, with a regression test using your repro (two resources in one expression, default resolves to "A-B"). That keeps us out of the shared referencesFromExpression path. Option A (making the JSON path iterate Variables like hclsyntax does) is a real general fix, but it's your call on shared scanner code, and since our templates are all .tf we did not want to take on that blast radius for an opt-in optimization. Glad to help with A separately if you want the .tf.json speedup. Two lint things the PR's new-vs-main lint surfaced that I also cleaned up:
Local run is clean: full parser package tests pass and golangci-lint new-vs-main reports 0 issues. |
…223) Pins the parameter, preset, and tag values that flow from `resource` blocks, ahead of #221 (which evaluates only the parameter/preset/tag closure via coder/trivy#74). Everything passes on `main`, where nothing is pruned, so a failure after rebasing #221 means pruning dropped a resource a target depends on. - `testdata/resourceclosure`: the fixture from #221 plus conditional, `try()`, expression index, `option` and `dynamic "option"` blocks, `count` on a target, a data-source intermediary, tag -> resource -> parameter, preset `prebuilds`, and a local module. - `testdata/resourceclosurejson`: a `.tf.json` template with two resources in one expression. This currently fails against the #221 pin (second resource pruned, default unknown); see the [comment on coder/trivy#74](coder/trivy#74 (comment)). - `testdata/diskaccess`: revives #85. `file()` on an in-tree path is read; a path escaping the template root is unknown and flags the option as invalid. Not closure-related, but a cheap guard that resource pruning does not disturb function evaluation against the template fs. - `testdata/chain`: revives #164. Pins two-hop count chaining: a parameter count-gated on a count-gated parameter's `[0].value` is silently dropped today because its count is evaluated before the input flows in. The vector asserts current behavior and the comment says how to flip it once fixed. Prepared with Coder Agents assistance.
Follow-up to #223. Pins more of the evaluation paths a parameter, preset, or tag can take to reach a `resource`, ahead of #221. All pass on `main`, and on #221 with the trivy pin at coder/trivy#74's current head (`fc995ac`). - `testdata/resourceclosure2`: `validation` block text and regex from a resource (asserted with a failing input), `[for ...]` and `{for ...}` over `count` and `for_each` resources, a splat directly in a tag, a `count = 0` resource behind `try()`, a resource whose `for_each` is driven by a parameter value and is read by a tag, a computed attribute (stays unknown), a two-level `module.outer` -> `module.inner` chain, and three-hop locals. - `testdata/resourceclosuresubmod`: the only parameter is in a submodule and reads a root resource through a module input; the root has no target block at all. Prepared with Coder Agents assistance.
coder/trivy#74 (OptionWithResourceClosure) is merged. Flip the replace from the PushTheLimit fork to coder/trivy@15b94953 so preview builds against the upstreamed closure support.
Preview evaluates the entire Terraform module graph on every call, even though rendering a workspace form only needs coder_parameter, coder_workspace_preset and coder_workspace_tags (and what they reference). The resources a workspace would create cannot feed those blocks, so evaluating them is wasted work that dominates request latency on large templates. Pass OptionWithResourceClosure with the three target block types so the parser drops root-module resources that nothing in that closure references. On a real template this cuts EvaluateAll from ~2s to ~0.16s (~12x) with byte-identical parameters, presets and tags. Uses the OptionWithResourceClosure addition merged in coder/trivy#74; the trivy replace points at that merged commit. Correctness is guarded by the resource closure test corpus already in main (coder#223, coder#224).
…#226) Evaluates only the `coder_parameter` / `coder_workspace_preset` / `coder_workspace_tags` closure when rendering, with an opt-out. Supersedes #221 (thanks @PushTheLimit); the trivy side is coder/trivy#74. - Bumps the trivy pin to a commit with `OptionWithResourceClosure`, and passes it by default so root resources nothing in the closure references are skipped. On the motivating template `EvaluateAll` drops from ~2s to ~0.16s. - `Preview` takes `opts ...Option`. `OptionFullEvaluation()` evaluates every resource, so coderd can turn the optimization off behind a setting without a new preview release if a template misbehaves. - `Test_Extract` runs every vector in both modes against the same expectations, pinning that parameters, presets, tags, and variables are identical either way. `Test_OptionFullEvaluation` observes the difference through `Output.ModuleOutput`. Behavior notes: - `Output.ModuleOutput` may contain unknown values for root outputs that read pruned resources. Nothing in coderd consumes it. - Templates containing `.tf.json` files skip pruning entirely (trivy's JSON reference extraction is not reliable enough to prune on). Test coverage for the reference shapes the pruner must see through landed separately in #223 and #224. Prepared with Coder Agents assistance.
Bumps `github.com/coder/preview` to pick up resource closure evaluation (coder/preview#226) and its `OptionFullEvaluation` opt-out. The `coder/trivy` replace moves to the commit that provides `OptionWithResourceClosure` (coder/trivy#74), matching what preview pins. Behavior is unchanged in this PR: `preview.Preview` now prunes root resources outside the parameter/preset/tag closure by default, and coderd already only reads parameters, presets, and tags from it. The next PR in the stack adds a deployment-level escape hatch. Prepared with Coder Agents assistance.
…ion (#29356) **What changed** Dynamic parameter rendering now evaluates only the coder parameter blocks and what they reference. Other root-module resources are skipped. Parameter, preset, and tag values are unchanged; optimizing rendering on large templates. **How to revert** Set `CODER_DYNAMIC_PARAMETERS_FULL_EVALUATION=true` (or `--dynamic-parameters-full-evaluation`) on `coder server` to restore full evaluation. The flag is hidden and needs no restart of workspaces. If you need it, please [open an issue](https://github.com/coder/coder/issues/new) with the template so we can fix the root cause. **Troubleshooting** Symptoms would be a parameter default, option list, preset value, or workspace tag that shows as unknown or empty after upgrading, when it was populated before. Confirm by setting the flag above; if the value comes back, report it. Templates written in `.tf.json` are not affected (they always use full evaluation). **Why** Rendering re-ran the entire template graph on every parameter change in the workspace form, which dominated latency on large templates. Resources a workspace would create cannot feed a parameter, so evaluating them is wasted work. The optimization lives in coder/preview#226 (coder/trivy#74); this PR adds the deployment-level escape hatch. --- - `CODER_DYNAMIC_PARAMETERS_FULL_EVALUATION` / `--dynamic-parameters-full-evaluation` (hidden, default `false`) on `DeploymentValues`. - `dynamicparameters.PreviewOptions(dv)` maps it to `preview.OptionFullEvaluation()`; `dynamicparameters.WithPreviewOptions` threads it through the loader to `preview.Preview`. - Wired at the three render paths that hold deployment values: the template version tags extractor, the dynamic parameters websocket, and `wsbuilder` builds created from `coderd` (which call `.DeploymentValues(...)`). Not covered: the autobuild lifecycle executor and the enterprise prebuilds reconciler construct `wsbuilder` without deployment values, so builds started from those paths use preview's default regardless of the flag. Those only re-render for builds, not the keystroke websocket path the optimization targets. Prepared with Coder Agents assistance.
What
Add
OptionWithResourceClosure(targetTypes []string)to the Terraform parser. When set, root-moduleresourceblocks that are not reachable (via references) from the given target block types are dropped before evaluation. Default off, so existing callers are unaffected.Why
EvaluateAllevaluates every resource and module in a module on each call. Callers that only need a subset of a module's output pay to evaluate resources whose values can never affect that output.The concrete driver is
coder/previewrendering a workspace form: it only needscoder_parameter/coder_workspace_preset/coder_workspace_tagsand what they reference, but today the whole module graph is evaluated on every keystroke over the dynamic-parameters websocket. On a real Altana template this is ~2s ofEvaluateAllper request; with this option it drops to ~0.16s locally (~12x), and the computed parameters, presets and tags are byte-identical.How
resourcenor anoutput(variables, locals, data sources, providers, module arguments, and the target blocks themselves).resourcereachable from that frontier, following references through resources that are themselves retained.Safety
The retained set is the transitive reference closure of everything whose value can flow into a target block, so a resource is excluded only when nothing a target block reads can reference it: the evaluated values of the target blocks are unchanged. The closure is conservative, a reference that cannot be resolved to a concrete block keeps the matching blocks. If no target block is present, nothing is pruned.
Testing
resource_closure_test.go: an orphan resource is pruned, a resource referenced by a parameter transitively through alocalis retained, and the parameter's computed default is unchanged. Also covers default-off and no-target-present.pkg/iac/scanners/terraform/parsersuite passes unchanged.Base branch
Targets
coder/preview_v0_69(the linecoder/previewcurrently pins) so it can flow into the deployed preview build; happy to also land it onmain.