a-green-hand-jack/data-pipeline-manager

Data Pipeline Manager

Make deliberate, documented data decisions before training. This skill prevents the most common experimental validity failures: undocumented preprocessing, leaky splits, contaminated test sets, and irreproducible data pipelines.

Use this skill when:

• deciding how to acquire, access, or download a dataset
• designing or auditing train/val/test split protocols
• preprocessing choices need documentation before they affect results
• data quality issues could corrupt baselines or ablations
• contamination between training and evaluation data is a reviewer risk
• dataset versions, seeds, and pipeline steps need to be pinned for reproducibility
• a paper claim depends on a specific data protocol and the protocol is not yet written down

Do not use this skill for running experiments — use run-experiment once the pipeline is validated. Do not use this skill to write result sections — use experiment-story-writer after results exist.

Pair this skill with:

• experiment-design-planner to ensure the split protocol matches the experimental design's evaluation criteria
• baseline-selection-audit when dataset choice or protocol affects which baselines are comparable
• research-project-memory to record data decisions as claims, evidence, and risks
• result-diagnosis when surprising results may trace to a data artifact rather than a method failure
• paper-evidence-board when data protocol details need to appear in the paper's methods and evidence slots
• init-python-project when the project's data/ layout and pipeline scripts need to be scaffolded

Skill Directory Layout

<installed-skill-dir>/
├── SKILL.md
├── references/
│   ├── split-protocols.md
│   └── contamination-guide.md
└── templates/
    └── pipeline-plan.md

Progressive Loading

• Always read references/split-protocols.md when designing or auditing split protocols.
• Read references/contamination-guide.md when handling LLM evaluation, pre-trained model probing, retrieval benchmarks, or any setting where the model may have seen test-adjacent data.
• Read templates/pipeline-plan.md before writing a pipeline plan.
• Read memory/claim-board.md and memory/evidence-board.md when data decisions need to be linked to paper claims.
• Read code/.agent/benchmark-plan.md or equivalent when a benchmark protocol already exists.

Core Principles

• Treat data decisions as experimental controls, not implementation details.
• Document every preprocessing step before it runs, not after results appear.
• Design splits from the evaluation goal backward: what question must the test set answer?
• Never let the model, hyperparameter search, or any paper-motivated decision touch the test set.
• Contamination is the paper's responsibility: assume reviewers will ask, and have an answer.
• Version the processed dataset alongside the code that created it.
• Seed-fix every shuffle, sampling step, and augmentation that affects reproducibility.
• Data quality issues found after training are ten times more expensive than those found before.

Step 1 — Gather Context

Locate:

• project root: look for data/, datasets/, code/data/, or user-provided path
• existing pipeline scripts: data_utils.py, preprocess.py, scripts/prepare_data.sh, or equivalent
• existing documentation: docs/data/, data/README.md, or any dataset description
• experiment design: code/.agent/benchmark-plan.md, memory/claim-board.md
• paper methods draft: paper/sections/experiments.tex or equivalent when present

Record:

• what dataset(s) are in use or planned
• whether a split already exists
• what preprocessing is already applied or planned
• whether any data quality audit has been done
• whether contamination has been checked

If nothing exists, produce a draft pipeline plan and mark uncertain fields as TBD.

Step 2 — Select Pipeline Mode

Choose based on the user's request or the current gap:

• audit: review an existing pipeline for completeness, reproducibility, and reviewer risk
• split: design or validate the train/val/test split protocol
• quality: run or plan a data quality audit (missing values, label noise, distribution shifts, class imbalance)
• contamination: check for data leakage between splits and pre-training / retrieved data
• versioning: pin dataset version, hashes, download scripts, and processing seeds
• plan: create a full pipeline plan covering all of the above for a new dataset

Default: use audit when a pipeline exists, plan when none exists.

Step 3 — Design or Audit the Split Protocol

Read references/split-protocols.md.

For every dataset split, record:

- Split: train | val | test | held-out
- Size: N examples (% of total)
- Stratification: class-stratified | time-ordered | random | grouped | domain-separated
- Overlap policy: none | explicit-allowed (state why)
- Leakage controls: [list controls]
- Assignment seed: <integer>
- Purpose: what hypothesis or metric this split answers
- Forbidden use: what must NOT touch this split before final evaluation

Flag any split designed after the method was partially tuned — this is a reviewer-visible risk. Flag any split where the validation set was used as a proxy test set.

Step 4 — Data Quality Audit

Check:

• Missing values: are they meaningful (masked) or errors? What is the imputation or exclusion policy?
• Label noise: known label error rate, labeling protocol, inter-annotator agreement if applicable
• Class imbalance: imbalance ratio, whether baselines handle it equivalently
• Distribution shift: between train and test, or between this dataset and cited prior work
• Duplicate or near-duplicate examples: deduplication policy and its effect on split boundaries
• Outliers: what is excluded and on what criteria — is this reproducible?
• Data leakage from preprocessing: normalization fit on train only; no augmentation at test time

For each finding, classify as:

• blocker: will cause invalid results or comparison failures
• risk: should be disclosed in the paper's limitations or experiment settings
• note: worth tracking but not blocking

Step 5 — Contamination Check

Read references/contamination-guide.md.

For LLM evaluation, retrieval, or pre-trained model probing:

• Check whether benchmark test examples appear in common pretraining corpora (e.g., via n-gram overlap, exact hash, or known contamination audits)
• Check whether model selection used test performance directly or indirectly
• Check whether retrieved context at test time could contain test-set answers
• Check whether data augmentation generated test-adjacent examples

For standard supervised tasks:

• Confirm no sample appears in both train and test after deduplication
• Confirm that grouping or stratification respects natural data boundaries (e.g., speaker IDs, document IDs, patient IDs)
• Confirm that any held-out reserve set was never used for threshold selection or model comparison

For every contamination finding, record the severity and the available mitigation.

Step 6 — Version and Reproducibility Record

For reproducible experiments, the pipeline plan must include:

- Dataset name and version: <name>==<version> or commit/hash
- Download source: <url or repo> (verified checksum or hash)
- Raw data hash: sha256=<hash>
- Processing script: <path/to/script> at git commit <sha>
- Processing seed: <integer> for every random step
- Processed data hash: sha256=<hash> of final splits
- Storage location: <path or remote>
- Excluded samples: <rule and count>
- Known issues: <list>

If any of these fields cannot be filled, mark as TODO and add to memory/action-board.md.

Step 7 — Write the Pipeline Plan

Use templates/pipeline-plan.md.

Save to:

code/.agent/data-pipeline-plan.md

If no code/ directory exists, save to:

.agent/data-pipeline-plan.md

When updating, preserve stable decisions and add a compact change note.

Step 8 — Memory Writeback

After completing a pipeline plan or audit:

• Write data decisions that affect claims to memory/claim-board.md
• Write contamination risks to memory/risk-board.md
• Write unresolved data quality issues as actions to memory/action-board.md
• Write evidence about dataset characteristics to memory/evidence-board.md when they are cited in the paper
• Update memory/current-status.md when the dataset version or split protocol changes

Step 9 — Route Follow-Ups

• experiment-design-planner: align experimental design with the finalized split protocol
• baseline-selection-audit: recheck baseline comparability after split or preprocessing changes
• result-diagnosis: diagnose surprising results that may trace to a data issue
• paper-evidence-board: add dataset protocol details to the paper's evidence slots
• init-python-project: scaffold data/ layout and pipeline scripts

Final Sanity Check

Before finalizing:

• every split has a documented purpose and a forbidden-use policy
• every preprocessing step is reproducible with a fixed seed and a version-pinned script
• data quality findings are classified as blockers, risks, or notes
• contamination checks are explicitly done or explicitly deferred with a risk entry
• dataset version, hashes, and download source are recorded or marked TODO
• memory writeback covers affected claims, risks, and actions
• next action is clear