starlake-ai/starflow-lineage-review

Data Lineage Review

Overview

Traces and documents data lineage across all pipeline stages — from source extraction through loading, transformation, and final consumption. Uses Starlake's built-in lineage capabilities and supplements with manual analysis where needed.

Role Guidance: Act as a Data Quality Engineer reviewing data lineage for completeness and correctness.

Lineage Dimensions

Table-Level Lineage

Trace the flow of data between tables:

• Source tables → Load targets → Transform outputs → Mart tables
• Use starlake lineage --task {domain}.{task} to generate lineage graphs
• Use starlake table-dependencies to list all dependencies

Column-Level Lineage

Trace individual columns through transformations:

• Source column → Load attribute → Transform expression → Output column
• Use starlake col-lineage --task {domain}.{task} --column {col} for column tracing
• Identify columns that undergo transformations (aggregation, type casting, privacy)

ACL/Permission Lineage

Trace access control through the pipeline:

• Source permissions → Table ACLs → View grants → Consumer access
• Use starlake acl-dependencies to map permission flows

Review Checklist

• [ ] Every output table has traceable inputs
• [ ] No orphaned tables (loaded but never consumed)
• [ ] No missing sources (referenced but not loaded)
• [ ] Privacy transforms applied consistently across lineage
• [ ] Column-level lineage documented for regulated data
• [ ] Cross-domain dependencies are explicit and documented

Starlake Commands

# Generate table lineage for a task
starlake lineage --task {domain}.{task}

# Trace column lineage
starlake col-lineage --task {domain}.{task} --column {column_name}

# List all table dependencies
starlake table-dependencies

# List ACL dependencies
starlake acl-dependencies

Output

Lineage documentation including:

• Visual dependency graph (text-based DAG representation)
• Table-level lineage matrix
• Column-level lineage for sensitive/regulated data
• Gap analysis (missing or broken lineage paths)

Related Starlake Skills

• Use the lineage skill for detailed starlake lineage command options
• Use the col-lineage skill for column-level lineage tracing
• Use the table-dependencies skill for dependency graph generation
• Use the acl-dependencies skill for permission lineage

Outcome

Complete data lineage documentation ensuring traceability from source to consumption for governance, debugging, and impact analysis.