brycewang-stanford/universal-ma-codebook

Universal Meta-Analysis Codebook

Version: 2.2 Status: Production Codex Review: APPROVE WITH MINOR CHANGES (2026-01-26) Update: Context-specific extensions (2026-01-26)

Purpose

A universal, AI-Human collaboration codebook for meta-analysis that enables:

1. AI extraction from PDFs (RAG/OCR) with confidence tracking
2. Human verification of AI-extracted values
3. 100% human-verified data through structured workflow
4. Integration with Diverga C5/C6/C7 agents and Category I pipeline
5. Context-specific extensions for domain-specific moderator variables

Context-Specific Extensions

The Universal Codebook supports project-specific moderator layers that extend the base 4-layer structure. Each meta-analysis context may have unique moderator variables.

Extension Architecture

┌─────────────────────────────────────────────────────────────────────┐
│              UNIVERSAL CODEBOOK WITH CONTEXT EXTENSION              │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  LAYER 1: IDENTIFIERS + METADATA (10 fields) ← Universal           │
│  LAYER 2: CORE STATISTICAL VALUES (18 fields) ← Universal          │
│  LAYER 3: CONTEXT-SPECIFIC MODERATORS ← Project Extension          │
│  LAYER 4: AI EXTRACTION PROVENANCE ← Universal                     │
│  LAYER 5: HUMAN VERIFICATION (8 fields) ← Universal                │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

Available Context Extensions

| Context | Extension File | Moderator Count | |---------|----------------|-----------------| | GenAI-HE | GENAI_HE_CODEBOOK.md | 15 moderators | | Clinical Trials | CLINICAL_CODEBOOK.md | TBD | | Educational Tech | EDTECH_CODEBOOK.md | TBD |

Creating a Context Extension

1. Define moderator variables specific to your research domain
2. Create classification rules for categorical moderators
3. Write AI extraction prompts for each moderator
4. Configure C6 agent with the extension schema

# Example: Configure C6 for GenAI-HE context
c6.configure_extension(
    context="genai_he",
    moderators=[
        {"name": "genai_tool", "type": "categorical", "values": ["ChatGPT", "Claude", ...]},
        {"name": "blooms_level", "type": "ordinal", "values": ["remember", "understand", ...]},
        {"name": "study_design", "type": "categorical", "values": ["RCT", "quasi", ...]},
    ],
    extraction_prompts=GENAI_HE_PROMPTS
)

GenAI-HE Extension (Example)

Layer 3: GenAI-HE Moderator Variables (15 fields)

| Category | Fields | |----------|--------| | GenAI Tool | genai_tool, genai_tool_version, genai_access_type | | Educational Outcome | blooms_level, outcome_dimension, learning_domain | | Study Design | study_design, intervention_duration, intervention_type, control_condition | | Context | education_level, discipline, country, sample_size_total, publication_type |

See: GenAI-HE-Review-AIMC/docs/GENAI_HE_CODEBOOK.md for full specification

Architecture: Four-Layer Design

┌─────────────────────────────────────────────────────────────────────┐
│              UNIVERSAL META-ANALYSIS CODEBOOK v2.1                  │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  LAYER 1: IDENTIFIERS + METADATA (10 fields)                        │
│  study_id, es_id, citation, doi, year, design_type,                │
│  timepoint, arm_label_treat, arm_label_control, unit_of_analysis   │
│                                                                     │
│  LAYER 2: CORE STATISTICAL VALUES (18 fields)                       │
│  Primary: outcome_name → se_g (12)                                  │
│  Change-score: pre_mean_treat, pre_sd_treat, pre_post_corr (3)     │
│  Cluster: cluster_size, icc, n_clusters (3)                        │
│                                                                     │
│  LAYER 3: AI EXTRACTION PROVENANCE                                  │
│  Per-value: ai_value, source, method, confidence, derived_from     │
│  Stored as: ai_extraction_json                                     │
│                                                                     │
│  LAYER 4: HUMAN VERIFICATION (8 fields)                             │
│  verified_status, verified_by, verified_date, corrections_json,    │
│  disagreement_resolved, final_values_json, verification_notes,     │
│  sign_off                                                          │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

Workflow: AI-Human Collaboration

Phase 1: AI Extraction (Automated)

Triggered by: I3 RAG building completion or manual PDF upload

Agent: C6-DataIntegrityGuard

# C6 extracts statistical values from PDFs
extraction_result = c6.extract_with_provenance(
    pdf_folder="./pdfs",
    methods=["rag", "ocr"],
    reconciliation="hierarchy",
    log_all_candidates=True
)

Actions:

1. I3 builds RAG from PDFs
2. C6 queries for statistical values (M, SD, n)
3. Multiple extraction methods run in parallel
4. Conflict resolution applied (hierarchy + tolerance)
5. Provenance recorded for all extractions
6. Hedges' g calculated where inputs complete

Output: All rows → verified_status = PENDING

Phase 2: Triage (Automated)

Agent: C7-ErrorPreventionEngine

# C7 categorizes by effective confidence
triage_result = c7.triage_extractions(
    data=extraction_result,
    thresholds=CONFIGURABLE_THRESHOLDS
)

Categories: | Confidence | Status | Action | |------------|--------|--------| | HIGH (≥90%) | PROVISIONAL | Awaits sign-off | | MEDIUM (70-89%) | PENDING | Recommended review | | LOW (<70%) | PENDING | Required review (priority) | | CONFLICT | PENDING | Required review (top priority) |

Phase 3: Human Review (Mandatory)

Interface: Excel Review Queue or Web UI

Critical Rule: ALL rows require human verification

Priority Queue:

1. Conflicts detected (highest)
2. LOW confidence
3. MEDIUM confidence
4. HIGH confidence (spot check)

Human Actions:

• Verify AI extraction against PDF
• Correct errors, record reason
• Mark as VERIFIED or REJECTED
• Resolve conflicts

Phase 4: Final Sign-Off

Agent: C5-MetaAnalysisMaster

# C5 validates all gates pass
validation = c5.validate_final(
    data=verified_data,
    require_all_verified=True,
    require_all_signed_off=True
)

Requirements:

• All rows: verified_status = VERIFIED
• All rows: sign_off = True
• All gates pass (C5 validation)

Result: 100% Human-Verified Dataset

---

Field Specifications

Layer 1: Identifiers + Metadata

| Field | Type | Description | Example | |-------|------|-------------|---------| | study_id | str | Unique study identifier | "CHEN_2024" | | es_id | str | Effect size ID | "CHEN_2024_01" | | citation | str | Full APA citation | "Chen et al. (2024)..." | | doi | str | DOI | "10.1000/xyz" | | year | int | Publication year | 2024 | | design_type | str | RCT|QUASI|PRE_POST | "RCT" | | timepoint | str | Measurement timing | "post" | | arm_label_treat | str | Treatment label | "ChatGPT group" | | arm_label_control | str | Control label | "Traditional" | | unit_of_analysis | str | individual|cluster | "individual" |

Layer 2: Core Statistical Values

Primary Statistics

| Field | Type | Required | |-------|------|----------| | outcome_name | str | Yes | | outcome_unit | str | No | | es_type | str | Yes | | analysis_type | str | No | | n_treatment | int | Yes | | n_control | int | Yes | | m_treatment | float | Conditional | | sd_treatment | float | Conditional | | m_control | float | Conditional | | sd_control | float | Conditional | | hedges_g | float | Derived | | se_g | float | Derived |

Change-Score Fields (when es_type = CHANGE)

| Field | Type | Description | |-------|------|-------------| | pre_mean_treat | float | Pre-test mean | | pre_sd_treat | float | Pre-test SD | | pre_post_corr | float | Pre-post correlation (default 0.5) |

Cluster Fields (when unit_of_analysis = cluster)

| Field | Type | Description | |-------|------|-------------| | cluster_size | float | Average cluster size | | icc | float | Intra-class correlation | | n_clusters | int | Number of clusters |

Layer 3: AI Extraction Provenance

Stored in ai_extraction_json:

{
  "n_treatment": {
    "ai_value": 43,
    "source": "Table 2, p.8",
    "method": "OCR",
    "confidence": 85,
    "derived_from": null
  },
  "sd_treatment": {
    "ai_value": 12.5,
    "source": "Text p.11, 95% CI",
    "method": "CALCULATED",
    "confidence": 92,
    "derived_from": "CI_95: SE = (14.8-10.2)/3.92"
  }
}

Layer 4: Human Verification

| Field | Type | Values | |-------|------|--------| | verified_status | str | PENDING|PROVISIONAL|VERIFIED|REJECTED | | verified_by | str | Reviewer initials | | verified_date | date | Review date | | corrections_json | json | {field: {ai_value, final_value, reason}} | | disagreement_resolved | bool | Conflict resolved? | | final_values_json | json | Human-confirmed values | | verification_notes | str | Free text notes | | sign_off | bool | Final approval |

---

Confidence Thresholds (Configurable)

Per-Field Thresholds

| Field | HIGH | MEDIUM | LOW | |-------|------|--------|-----| | n (sample size) | ≥95% | 80-94% | <80% | | M (mean) | ≥90% | 70-89% | <70% | | SD | ≥85% | 65-84% | <65% | | hedges_g (derived) | ≥92% | 75-91% | <75% | | se_g (derived) | ≥92% | 75-91% | <75% | | pre_post_corr | ≥85% | 65-84% | <65% | | icc | ≥80% | 60-79% | <60% |

Per-Source Modifiers

| Source | Modifier | |--------|----------| | Structured table | +10% | | Semi-structured figure | +5% | | Unstructured text | 0% | | Abstract only | -15% | | OCR with artifacts | -20% |

Formula: effective_confidence = base_confidence + source_modifier

---

Conflict Resolution

Extraction Hierarchy

| Priority | Source | Weight | |----------|--------|--------| | 1 | Table cell | 1.0 | | 2 | Figure data | 0.9 | | 3 | In-text stats | 0.8 | | 4 | Abstract | 0.5 |

Tolerance Thresholds

| Value Type | Relative | Absolute | |------------|----------|----------| | n (sample size) | 5% | 2 | | M (mean) | 10% | 0.5 | | SD | 15% | 0.5 |

Rule: If disagreement exceeds EITHER threshold → Human review required

---

Derived Value Verification

For calculated values (hedges_g, se_g), human verification means:

1. All source values (M, SD, n) verified
2. Formula appropriate for es_type
3. If change-score: pre_post_corr verified or documented default
4. If cluster: ICC and cluster_size verified
5. Final values recalculated from verified inputs

---

Integration Points

Systematic Review Pipeline Integration

# After Stage 5 (RAG building)
# RAG query integration

rag = RAGQuery(project_path)
values = c6.extract_from_rag(
    rag=rag,
    fields=["n_treatment", "n_control", "m_treatment", "sd_treatment",
            "m_control", "sd_control"],
    fallback_to_ocr=True
)

C5/C6/C7 Agent Roles

| Agent | Role in Codebook | |-------|------------------| | C5-MetaAnalysisMaster | Final validation, gate enforcement | | C6-DataIntegrityGuard | Extraction, Hedges' g calculation | | C7-ErrorPreventionEngine | Triage, conflict detection, warnings |

---

Excel Template Structure

Sheet 1: Codebook

One-page reference with field definitions

Sheet 2: Data (Main)

41 columns (39 visible + 2 JSON)

Sheet 3: Review Queue

Priority-ordered list of rows needing human review

Sheet 4: Extraction Log

Audit trail of all AI extractions

---

Success Metrics

| Metric | Target | |--------|--------| | AI extraction rate | ≥85% | | AI confidence accuracy | ≥90% | | Conflict detection rate | ≥95% | | Human review completeness | 100% | | Final sign-off rate | 100% | | Data completeness (Hedges' g) | ≥95% |

---

Commands

# Initialize codebook for a project
diverga codebook init --project genai-he

# Import AI extractions
diverga codebook import --source rag --project genai-he

# Generate review queue
diverga codebook queue --project genai-he

# Validate final dataset
diverga codebook validate --project genai-he

---

References

• Plan document: docs/plans/META_ANALYSIS_CODEBOOK_PLAN_V2.md
• C5 agent: .claude/skills/C5-meta-analysis-master/SKILL.md
• C6 agent: .claude/skills/C6-data-integrity-guard/SKILL.md
• C7 agent: .claude/skills/C7-error-prevention-engine/SKILL.md
• Borenstein et al. (2021). Introduction to Meta-Analysis
• Cochrane Handbook Chapter 6: Extracting Data
• PRISMA 2020 Statement

---

Created: 2026-01-26 Codex Review: APPROVE WITH MINOR CHANGES Author: Claude Code