elastic/schema-coverage

Schema Coverage (Terraform resource vs acceptance tests)

Goal

Given a Terraform resource, compare its schema attributes/blocks to what the acceptance tests configure and assert, then highlight opportunities to improve coverage.

Report findings in this order:

1. Attributes with no coverage
2. Attributes with poor coverage

Inputs (infer if not provided)

• Resource name (e.g. elasticstack_elasticsearch_security_api_key)
• Or the schema file / acceptance test file path
• Or the Go package directory containing the resource and *_acc_test.go

If the user has an acceptance test open, infer the resource under test from:

• resource.Test(...) names like resourceName := "elasticstack_..."
• resource.ParallelTest(...) steps referencing a single resource
• config builder function names like testAcc...Resource...

Workflow

1) Locate and parse the schema

Find the resource schema definition and capture all schema keys:

• Top-level attributes in Schema: map[string]*schema.Schema{ ... }
• Nested block schemas in:
  • Elem: &schema.Resource{ Schema: ... }
  • lists/sets/maps of resources and objects
  • nested blocks referenced via helpers

For each attribute/block, record:

• Path: terraform attribute path (top-level foo, nested block.0.bar, etc.)
• Schema metadata: Required/Optional/Computed, ForceNew, type (TypeString, TypeList, etc.), MaxItems/MinItems

2) Locate acceptance tests and collect attribute usage

Scan the acceptance tests for two independent signals:

• Configured attributes: attributes explicitly set in HCL test configs (including nested blocks).
• Asserted attributes: attributes referenced in checks, including:
  • value assertions (e.g. TestCheckResourceAttr)
  • set-only assertions (e.g. TestCheckResourceAttrSet)
  • absence assertions (e.g. TestCheckNoResourceAttr)
  • collection assertions (e.g. type-set element checks, list length checks)

Also detect update coverage by identifying multiple resource.TestStep{ Config: ... } steps for the same resource and whether attribute values change between steps.

3) Build a coverage matrix

For each schema attribute path, compute:

• Configured? (ever set in any test config)
• Asserted? (ever referenced by any check)
• Assertion quality:
  • value-specific: asserts the exact value (preferred)
  • set-only: only checks “is set” (weaker)
  • absence: checks unset / removed (good for optional fields)
• Value diversity: count distinct values across steps/configs (e.g. name="a" vs name="b")
• Optional-unset coverage: optional field has a test step where it is deliberately omitted, plus an assertion that it is absent (or defaults appropriately)
• Empty-collection coverage: collection has a step where it is empty (or omitted if optional), plus assertions validating the empty state
• Update coverage: attribute value changes across steps, and a post-update assertion verifies the new value

4) Produce the report (strict ordering)

Use the template below.

Report template

## Schema coverage report: <resource>

### Scope
- **Schema**: <file(s) or function(s)>
- **Acceptance tests**: <test file(s)>

### 1) Attributes with no coverage
These schema attributes/blocks are not referenced in acceptance tests (neither configured nor asserted):
- `<attr_path>`: <Required/Optional/Computed>, <type>. **Gap**: not configured, not asserted.

### 2) Attributes with poor coverage
These attributes appear in tests but the coverage is weak:
- `<attr_path>`: <schema flags/type>
  - **Observed**: <how it’s currently used (configured/asserted), example values>
  - **Gaps**:
    - <one or more of: set-only assertion, single value only, no unset coverage, no empty collection coverage, no update coverage>
  - **Suggested improvements**:
    - <concrete test step or assertion to add>

### Suggested next steps (smallest diffs first)
1. Add value-specific assertions for set-only checks
2. Add an “unset optional” step + `TestCheckNoResourceAttr`
3. Add an “empty collection” step + collection assertions
4. Add an “update” step changing the attribute + post-update assertions

Rules of thumb (for prioritization)

• Prefer value-specific assertions over “is set”.
• For Optional attributes, include at least one step where the attribute is omitted, and assert absence/default behavior.
• For collections (list/set/map), add a case for empty (or omitted if optional), and assert the expected empty state.
• For Update behavior, ensure there is at least one multi-step test where the attribute changes and the test asserts the new state.
• For Computed-only attributes, it’s acceptable to use set-only assertions when exact values are not deterministic, but prefer deterministic assertions when possible.

Notes / limitations

• Some schemas are built via helpers; follow helper references until all attribute keys are accounted for.
• Attribute paths in checks often use block.0.attr indexing; normalize consistently when matching to schema blocks.
• Don’t mark an attribute as “covered” solely because it appears in raw HCL—prefer it being asserted. Treat “configured but never asserted” as poor coverage, not good coverage.