santosomar/config-consistency-checker

Config Consistency Checker

"Works in staging, broken in prod" is almost always a config diff. Find it before deploy, not after.

What to compare

| Axis | Compare | Looking for | | --------------------------------- | -------------------------------------------------------- | ------------------------------------------------- | | Across environments | staging.yaml ↔ production.yaml | Keys present in one, missing in other | | Config file ↔ code | Config keys ↔ config.get(...) / os.environ[...] calls | Dead keys (in config, never read) and missing keys (read, never defined) | | Config file ↔ deployment | App config ↔ K8s ConfigMap/Deployment env | Config expects FOO; deployment sets FOO_URL | | Config file ↔ schema/defaults | Loaded values ↔ type annotations / validators | "30" where the code expects 30 | | Across replicas (runtime) | Live config on pod A ↔ pod B | Partial rollout, stale ConfigMap mount |

The key-diff algorithm

1. Flatten each config to dotted.key.path → value pairs.
2. Set-diff the key sets.
3. For keys in both, value-diff.
4. Classify each difference.

staging_keys − production_keys   → "missing in prod" (usually a bug)
production_keys − staging_keys   → "missing in staging" (usually a bug, unless it's a prod-only integration)
staging[k] ≠ production[k]       → "differs" — intentional or drift?

Classifying differences: intentional vs. drift

Not every difference is a bug. The signal is whether the difference is explicable:

| Difference class | Intentional? | | ----------------------------------- | --------------------------------------------------------- | | Database host differs | Intentional — they're different databases | | Pool size differs | Maybe — prod should be ≥ staging; if prod is smaller, bug | | Log level differs | Intentional (prod=INFO, dev=DEBUG). If prod=DEBUG, bug | | Feature flag differs | Depends entirely on the flag — ask | | Key exists in one, not the other | Almost always drift — someone forgot to add it to both | | Typo'd key (close string distance) | Always drift — databse_host is reading the default |

Worked example

Files: config/staging.yaml, config/production.yaml

Flattened:

staging:
  db.host                = staging-db.internal
  db.pool_size           = 20
  cache.ttl_seconds      = 300
  feature.new_checkout   = true
  log.level              = debug

production:
  db.host                = prod-db.internal
  db.pool_size           = 10
  cache.ttl_seconds      = 300
  feature.new_checkout   = true
  log.level              = info
  sentry.dsn             = https://...

Diff:

| Key | Staging | Prod | Verdict | | ----------------------- | ------- | ------------ | ---------------------------------------- | | db.host | differs | differs | ✅ Intentional — different databases | | db.pool_size | 20 | 10 | ⚠️ Prod pool smaller than staging — probably wrong | | cache.ttl_seconds | 300 | 300 | ✅ Match | | feature.new_checkout | true | true | ✅ Match | | log.level | debug | info | ✅ Intentional — expected split | | sentry.dsn | — | set | ⚠️ Missing in staging — errors from staging go nowhere |

Two flags, both worth raising.

Code ↔ config cross-check

Grep the codebase for config reads:

rg 'config\[|config\.get|os\.environ\[|getenv\(' --type py

Extract the key names. Compare against what's actually in the config files:

• In code, not in config: The app will hit a default or crash. Which one? Read the code.
• In config, not in code: Dead config. Delete it, or it's a typo for something the code does read.

Edge cases

• Secrets show as "differs" because they're redacted: Compare key presence, not value, for anything marked secret. password: *** ≠ password: *** by string comparison but they're both set — that's what matters.
• Array ordering differs: [a, b] vs [b, a] — depends on semantics. A list of allowed origins is a set (order irrelevant); a list of middleware is a sequence (order critical). You have to know which.
• One config is generated (Helm/Kustomize), the other is hand-written: Compare the rendered output, not the template. helm template / kustomize build first.
• The app has fallback defaults in code: A missing key isn't an error if the code has config.get("foo", sensible_default). But flag it — implicit defaults drift silently.

Do not

• Do not report "47 keys differ" without classifying. The user wants the 2 that are bugs, not the 45 that are intentional environment splits.
• Do not compare secret values. Compare presence. Never echo a secret value in the diff output.
• Do not trust the config file as source of truth. The code's reads are the truth; the config is what's supposed to satisfy them.
• Do not ignore near-miss key names. database_url in config and DATABASE_URI in code — Levenshtein distance 2, probably the bug.

Output format

## Missing keys
  <key>  — in <envA>, missing from <envB>  [<likely intentional | likely drift>]

## Suspicious value diffs
  <key>  <envA>=<val>  <envB>=<val>  — <why suspicious>

## Dead config (in file, never read by code)
  <key>

## Unbacked reads (in code, never in config — will default or crash)
  <file>:<line>  reads <key>  — default: <value | NONE — will crash>

## Clean
  <N> keys match across all environments