curiositech/data-quality-guardian

Data Quality Guardian

Implement comprehensive data quality frameworks with automated validation, anomaly detection, data contracts, and SLA monitoring.

Activation Triggers

Activate on: "data quality", "data validation", "Great Expectations", "data contract", "anomaly detection", "SLA", "freshness check", "schema validation", "data observability", "Soda", "elementary"

NOT for: dbt project layout → dbt-analytics-engineer | Schema evolution strategy → schema-evolution-manager | Data migration validation → data-migration-specialist

Quick Start

1. Define data contracts — agree on schema, freshness, volume, and value ranges with upstream producers
2. Implement tests — dbt tests for SQL models, Great Expectations for raw/external data
3. Monitor freshness — alert when source tables are stale beyond SLA
4. Detect anomalies — statistical checks for volume, distribution, and null rate changes
5. Build quality dashboard — centralized view of all quality metrics across pipelines

Core Capabilities

| Domain | Technologies | |--------|-------------| | Testing | dbt tests, Great Expectations 1.x, Soda Core 3.x | | Observability | elementary (dbt), Monte Carlo, Anomalo | | Contracts | Soda data contracts, dbt model contracts, Protobuf schemas | | Anomaly Detection | elementary anomaly monitors, custom z-score, Prophet | | Alerting | Slack/PagerDuty integration, SLA miss alerts |

Architecture Patterns

Multi-Layer Quality Checks

Raw Data (Landing)
    ↓
Layer 1: Schema Validation
    - Column types match contract
    - No unexpected NULLs in required fields
    - Row count within expected range
    ↓
Layer 2: Business Rule Validation
    - Referential integrity (FK relationships hold)
    - accepted_values constraints
    - Custom SQL assertions (e.g., revenue >= 0)
    ↓
Layer 3: Statistical Anomaly Detection
    - Row count deviation from 7-day rolling average
    - Null rate spike detection
    - Distribution shift (KL divergence)
    ↓
Layer 4: Freshness & SLA Monitoring
    - Source loaded within 2 hours
    - Downstream models built within 4 hours
    - Dashboard data <6 hours old

Data Contract YAML

# contracts/payments-contract.yml
contract:
  name: stripe_payments
  owner: payments-team
  version: "2.0"
  sla:
    freshness: 2h          # must be updated within 2 hours
    volume_min: 1000        # at least 1000 rows per day
    volume_max: 500000      # no more than 500k (anomaly if exceeded)
  schema:
    - name: payment_id
      type: string
      required: true
      unique: true
    - name: amount_cents
      type: integer
      required: true
      checks:
        - type: range
          min: 0
          max: 10000000     # $100k max
    - name: status
      type: string
      required: true
      checks:
        - type: accepted_values
          values: [succeeded, failed, pending, refunded]
    - name: created_at
      type: timestamp
      required: true
      checks:
        - type: not_in_future

Anomaly Detection with elementary

# dbt model with elementary anomaly monitors
version: 2
models:
  - name: fct_orders
    tests:
      - elementary.volume_anomalies:
          timestamp_column: created_at
          where: "created_at > dateadd(day, -30, current_date())"
          sensitivity: 3    # z-score threshold
      - elementary.column_anomalies:
          column_name: total_amount
          where: "created_at > dateadd(day, -30, current_date())"
      - elementary.freshness_anomalies:
          timestamp_column: _loaded_at
          sensitivity: 2

Anti-Patterns

1. Testing only in production — run quality checks in CI on sample data before merging; do not discover issues in production
2. Alert fatigue — too many low-severity alerts get ignored; tier alerts: P1 (data loss), P2 (quality regression), P3 (informational)
3. No contract with upstream — without an agreed contract, schema changes break pipelines silently
4. Static thresholds only — hardcoded "row count > 1000" breaks during holidays; use statistical anomaly detection
5. Quality checks at the end — validate at each layer (landing, staging, marts); catching errors early is cheaper

Quality Checklist

• [ ] Data contracts defined for all critical sources (schema, freshness, volume)
• [ ] dbt tests on every model: unique, not_null, accepted_values, relationships
• [ ] Freshness monitoring with SLA thresholds and alerting
• [ ] Anomaly detection on row counts, null rates, and value distributions
• [ ] Quality dashboard shows current status across all pipelines
• [ ] Alert severity tiered (P1/P2/P3) to prevent fatigue
• [ ] Quality checks run in CI (pre-merge) and production (post-load)
• [ ] Root cause analysis supported via data lineage integration
• [ ] Monthly data quality review with trending metrics
• [ ] Upstream teams notified within 15 minutes of contract violations