curiositech/data-lineage-tracker

Data Lineage Tracker

Implement end-to-end data lineage tracking using OpenLineage, DataHub, and Marquez for impact analysis, debugging, and compliance.

Activation Triggers

Activate on: "data lineage", "OpenLineage", "DataHub", "Marquez", "impact analysis", "data catalog", "column lineage", "data discovery", "data provenance", "downstream impact"

NOT for: Data quality checks → data-quality-guardian | dbt model documentation → dbt-analytics-engineer | Schema evolution → schema-evolution-manager

Quick Start

1. Choose platform — DataHub (full catalog + lineage), Marquez (lineage-focused), Amundsen, OpenMetadata
2. Instrument pipelines — integrate OpenLineage with Airflow, Spark, and dbt for automatic lineage
3. Capture column-level lineage — trace which source columns feed which target columns
4. Build impact analysis — before changing a table, see all downstream consumers
5. Integrate with incidents — link data quality alerts to lineage for root cause analysis

Core Capabilities

| Domain | Technologies | |--------|-------------| | Lineage Standard | OpenLineage 1.x (open standard, LFAI) | | Catalogs | DataHub 0.14+, OpenMetadata 1.5+, Amundsen | | Lineage Backend | Marquez (OpenLineage reference), DataHub lineage | | Integrations | Airflow OpenLineage, Spark OpenLineage, dbt, Great Expectations | | Visualization | DataHub lineage graph, Marquez UI, dbt docs |

Architecture Patterns

OpenLineage Event Flow

Airflow Task              Spark Job                 dbt Model
     │                         │                         │
     ├─ START event ──→        ├─ START event ──→        ├─ START event ──→
     │                         │                         │
     ├─ RUNNING event ─→       ├─ RUNNING event ─→       │
     │                         │                         │
     ├─ COMPLETE event ─→      ├─ COMPLETE event ─→      ├─ COMPLETE event ─→
     │  (with I/O datasets)    │  (with I/O datasets)    │  (with I/O datasets)
     ↓                         ↓                         ↓
              ┌────────────────────────────────────┐
              │     OpenLineage Backend            │
              │  (Marquez / DataHub / custom)      │
              │                                    │
              │  Stores: job runs, datasets,       │
              │  input/output relationships,       │
              │  column-level lineage              │
              └────────────────────────────────────┘

Impact Analysis Query

# DataHub GraphQL: find all downstream dependencies of a dataset
query = """
{
  dataset(urn: "urn:li:dataset:(urn:li:dataPlatform:snowflake,prod.fct_orders,PROD)") {
    downstream: relationships(
      input: { types: ["DownstreamOf"], direction: INCOMING, count: 50 }
    ) {
      relationships {
        entity {
          urn
          ... on Dataset {
            name
            platform { name }
            properties { description }
          }
        }
      }
    }
  }
}
"""

# Result: all dashboards, models, and exports that depend on fct_orders
# Use this BEFORE making schema changes

Column-Level Lineage

Source Tables                  Transformation              Target Table
─────────────                 ──────────────              ────────────
raw_payments.amount    ──→    SUM(amount)           ──→  fct_revenue.total_revenue
raw_payments.currency  ──→    exchange_rate_convert  ──→  fct_revenue.total_revenue_usd
raw_orders.order_id    ──→    JOIN key              ──→  fct_revenue.order_id
raw_customers.name     ──→    COALESCE(name, email) ──→  fct_revenue.customer_name

Column-level lineage answers:
  "Where does fct_revenue.total_revenue_usd come from?"
  → raw_payments.amount + raw_payments.currency via exchange_rate_convert

Anti-Patterns

1. Manual lineage documentation — manually maintained lineage diagrams go stale immediately; automate via OpenLineage
2. Table-level only — table lineage is not enough; column-level lineage is required for meaningful impact analysis
3. No integration with incidents — lineage is most valuable during incidents; link quality alerts to lineage graph
4. Lineage without ownership — every dataset in the catalog must have an owner team for accountability
5. Collecting without using — lineage data is useless if nobody queries it; embed impact analysis in CI and change management

Quality Checklist

• [ ] OpenLineage integrated with Airflow, Spark, and dbt
• [ ] Column-level lineage captured (not just table-level)
• [ ] All datasets have assigned owners in the catalog
• [ ] Impact analysis run before schema changes (CI check)
• [ ] Lineage graph accessible via UI (DataHub, Marquez)
• [ ] Data quality alerts link to lineage for root cause tracing
• [ ] Cross-platform lineage works (e.g., Kafka → Spark → Snowflake)
• [ ] Lineage metadata refreshed on every pipeline run (not periodic batch)
• [ ] Search and discovery enabled: analysts can find datasets by keyword
• [ ] Retention policy on lineage events (keep 90 days of run history)