mims-harvard/tooluniverse-meta-analysis

Meta-Analysis & Evidence Synthesis

Pool quantitative results from multiple studies into one estimate, and judge how consistent the studies are. This is the statistical half of a systematic review (the literature-collection half is tooluniverse-literature-deep-research).

When to use this

• You have effect sizes from ≥2 studies/cohorts and want a single pooled estimate + CI.
• Synthesizing a systematic review, replicated experiments, multi-cohort GWAS, or multi-dataset associations.
• Deciding whether studies agree (low heterogeneity) or conflict (high heterogeneity).

Do NOT use it to find the studies — use tooluniverse-literature-deep-research / the literature tools for that, then bring the extracted numbers here. Before trusting any input study, consider checking it with Crossref_check_retraction.

The workflow

1. Extract (effect, SE) per study  ← THE ERROR-PRONE STEP
2. Pick fixed vs random effects
3. Pool: MetaAnalysis_run
4. Read heterogeneity (I², Q, τ²)
5. Forest plot + interpret

Step 1 — Convert each study to (effect_size, se) ← do this carefully

The pooling step needs an effect size on an additive scale and its standard error. Ratio measures (OR/RR/HR) must be log-transformed first. Most reported numbers give you a CI, not an SE — derive the SE from the CI.

| What the paper reports | effect_size | se | |---|---|---| | OR / RR / HR with 95% CI [L, U] | ln(point) | (ln(U) − ln(L)) / (2 × 1.96) | | OR / RR / HR with a p-value (no CI) | ln(point) | |ln(point)| / z_from_p (two-sided z) | | GWAS / regression β with SE | β (as reported) | the reported SE | | Two groups, means + SDs + n₁,n₂ | Hedges' g (see script) | SE of g (see script) | | Single proportion p, n | logit(p)=ln(p/(1−p)) | sqrt(1/(np) + 1/(n(1−p))) | | Pearson correlation r, n | Fisher z = atanh(r) | 1 / sqrt(n − 3) |

Critical rules

• Log-transform ratio measures. Pooling raw ORs is wrong; pool ln(OR) and exponentiate the pooled result back. The script does this for you.
• One direction. Make sure every study's effect points the same way (e.g. "exposure increases risk"); flip sign / invert the ratio for studies coded the opposite way.
• Same effect measure. Don't mix OR with HR with mean-difference in one pool.
• 2 × 1.96 assumes a 95% CI; use 2 × 1.645 for 90%, 2 × 2.576 for 99%.

The helper script does these conversions — prefer it over hand math:

python skills/tooluniverse-meta-analysis/scripts/meta_analysis.py --input studies.csv
# studies.csv columns (use the set that matches your data):
#   name, or, ci_low, ci_high            (ratio + CI)
#   name, beta, se                       (already on log/linear scale)
#   name, mean1, sd1, n1, mean2, sd2, n2 (two-group means -> Hedges' g)
#   name, r, n                           (correlation -> Fisher z)

Step 2 — Fixed vs random effects

| Use fixed-effects when | Use random-effects when | |---|---| | Studies estimate the same true effect (e.g. exact replications, one trial split by site) | Studies differ in population/design/dose (the usual real-world case) | | I² is low (<25%) | I² is moderate–high, or studies are clinically heterogeneous |

When unsure, report random-effects (DerSimonian–Laird) as primary — it is the conservative default and widens the CI to reflect between-study variance.

Step 3 — Pool with MetaAnalysis_run

tu run MetaAnalysis_run '{"method":"random","studies":[
  {"name":"Smith 2019","effect_size":0.41,"se":0.12},
  {"name":"Lee 2021","effect_size":0.67,"se":0.18},
  {"name":"Garcia 2023","effect_size":0.33,"se":0.10}]}'

Returns pooled_effect, pooled_se, pooled_ci_lower/upper, pooled_z, pooled_p_value, a heterogeneity block (Q, Q_df, Q_p_value, I_squared, tau_squared), and per_study weights + CIs.

Scale foot-gun — read this. MetaAnalysis_run pools whatever scale you hand it and has no idea your inputs were ratios. For an OR/RR/HR you MUST pass the log-transformed effect_size + se from Step 1 (e.g. ln(1.42)=0.351, not 1.42) — feeding raw ratios silently produces a wrong pooled value with no error. And the values it returns — including its prose interpretation string — are on that same log scale. So: ignore the tool's interpretation field for ratios, and exp() the pooled_effect and CI bounds back to the OR/RR/HR scale yourself before reporting. The helper script avoids all of this — it takes raw ORs, tracks the scale, and prints results already back-transformed.

Step 4 — Interpret heterogeneity (decides the story)

| I² | Heterogeneity | What it means | |---|---|---| | 0–25% | Low | Studies largely agree; fixed-effects is defensible | | 25–50% | Moderate | Prefer random-effects; note the variability | | 50–75% | Substantial | Random-effects; investigate sources (subgroup / meta-regression) | | >75% | Considerable | Pooling may be inappropriate — explain why studies differ instead |

• Q_p_value < 0.10 → statistically significant heterogeneity (Q is low-powered, so 0.10 not 0.05).
• tau_squared is the between-study variance on the effect scale; > 0 is what random-effects adds over fixed.
• Borderline I² (≈25–50%) with a non-significant Q (Q_p_value ≫ 0.10), especially with few studies: fixed and random-effects converge — report random-effects as primary and note that fixed-effects agrees. Don't agonize over the model choice when both give essentially the same pooled estimate.

Step 5 — Forest plot + report

The script prints a text forest plot (per-study effect, CI, weight%, and the pooled diamond). Report, in order:

1. Pooled estimate + 95% CI + p (on the interpretable scale — exponentiate ratios back).
2. Number of studies and total N.
3. Heterogeneity: I² + Q p-value + the model you chose and why.
4. Direction/consistency: do all studies point the same way?

Example: "Across 3 cohorts (N=4,210), the pooled OR was 1.51 (95% CI 1.33–1.72, p=3.4×10⁻⁶), random-effects. Heterogeneity was substantial (I²=55%, Q p=0.11), so the random-effects model is reported; all three studies showed the same direction of effect."

Honest limitations

• Garbage in, garbage out. Meta-analysis cannot fix biased primary studies; check input study quality (and retraction status via Crossref_check_retraction) first.
• Publication bias. A pooled estimate from only published studies is likely inflated. With ≥10 studies, inspect a funnel plot / Egger's test (the script notes this); with <10, state that small-study bias cannot be assessed.
• Ecological / aggregation issues. Pooling study-level summaries is not the same as pooling individual patient data.
• Don't over-pool. With I²>75% and clinically different studies, a single number can mislead — describe the variation instead.

Related skills

• tooluniverse-literature-deep-research — find and grade the studies to feed in.
• tooluniverse-statistical-modeling — single-study regression, Cox, ORs (see references/cox_regression.md for HR extraction).
• tooluniverse-gwas-study-explorer / tooluniverse-gwas-finemapping — GWAS-specific multi-cohort analysis.