brycewang-stanford/svy

svy Skill

svy: design-based analysis of complex survey data in Python. Covers survey design specification (strata, PSU, weights, FPC), variance estimation (Taylor linearization, BRR, jackknife, bootstrap), descriptive estimation (means, totals, proportions, ratios, medians), survey-weighted GLM regression (gaussian, binomial, Poisson), domain/subpopulation analysis, calibration, and survey data I/O (SAS, SPSS, Stata). Uses Polars DataFrames natively. Use when analyzing data from complex sample surveys (NHANES, CPS, ACS PUMS, MEPS, ECLS-K, BRFSS, DHS). For non-survey regression, use statsmodels; for fixed effects, use pyfixest; for panel/IV models, use linearmodels.

Comprehensive skill for complex survey data analysis with svy. Use decision trees below to find the right guidance, then load detailed references.

What is svy?

svy is the Python package for design-based analysis of complex survey data:

• Survey-aware estimation: Means, totals, proportions, ratios, medians with proper design-based standard errors
• GLM regression: Survey-weighted linear, logistic, and Poisson regression with design-adjusted inference
• Flexible variance estimation: Taylor linearization (default), bootstrap, BRR (including Fay's modification), and jackknife (JK1, JKn) replicate methods
• Domain estimation: Correct subpopulation analysis without pre-filtering (preserves design structure)
• Native Polars: Built on Polars DataFrames, not pandas
• Survey data I/O: Read SAS (.sas7bdat), SPSS (.sav), Stata (.dta), and CSV with metadata
• Calibration: Post-stratification, raking, and GREG calibration for weight adjustment
• Validated: Results numerically equivalent to R's survey package across all methods

Version Notes

This skill targets svy 0.13.0 (released 2026-03-25). svy supersedes samplics (archived 2026-03-10), an earlier library by the same author (Mamadou S. Diallo, Ph.D.). Key differences from samplics:

• Unified Sample object replaces separate TaylorEstimator / ReplicateEstimator classes
• Polars-native (samplics used numpy arrays)
• Expanded GLM support and data I/O module (svy.io)
• The API is substantially different from samplics — do not assume samplics patterns carry over

How to Use This Skill

Reference File Structure

| File | Purpose | When to Read | |------|---------|--------------| | estimation.md | Means, totals, proportions, ratios, medians, domain estimation, cross-tabs, hypothesis tests | Descriptive survey statistics | | regression.md | Survey-weighted OLS, logistic, Poisson regression; extracting results; diagnostics | Survey regression models | | design-weights.md | Design specification, replicate weights, weight manipulation, variance setup, survey data I/O, federal survey patterns | Setting up the survey design object |

Reading Order

1. New to svy? Start with design-weights.md then estimation.md
2. Need survey-weighted regression? Read design-weights.md then regression.md
3. Have replicate weights already? Read design-weights.md (replicate design section) then estimation.md or regression.md
4. Setting up a federal survey (NHANES, CPS, etc.)? Read design-weights.md (federal survey patterns table)
5. Coming from samplics? Read design-weights.md for the new API; the Sample object replaces TaylorEstimator/ReplicateEstimator

Related Skills

| Skill | Relationship | |-------|-------------| | data-scientist | Provides methodology guidance (especially survey-analysis.md); svy provides implementation. Load data-scientist for "when and why" to use survey methods | | statsmodels | Complement for non-survey regression (OLS, GLM, time series, diagnostics). WLS in statsmodels is NOT survey-weighted regression — it does not account for stratification or clustering | | pyfixest | Complement for fixed effects models and DiD. pyfixest does not handle complex survey designs; use svy for survey-weighted estimation, pyfixest for FE/DiD | | linearmodels | Complement for panel models (RE, FD, Fama-MacBeth) and IV/GMM. Does not handle survey designs | | polars | svy uses Polars DataFrames natively. Load polars skill for data preparation before passing to svy |

Quick Decision Trees

"I need to analyze survey data"

What task?
├─ Descriptive statistics (mean, total, proportion)
│   └─ ./references/estimation.md
├─ Regression model
│   ├─ Linear (continuous outcome) → ./references/regression.md
│   ├─ Logistic (binary outcome) → ./references/regression.md
│   └─ Poisson (count outcome) → ./references/regression.md
├─ Set up the survey design object
│   └─ ./references/design-weights.md
├─ Read survey data from SAS/SPSS/Stata
│   └─ ./references/design-weights.md
├─ Subpopulation / domain analysis
│   └─ ./references/estimation.md
└─ Cross-tabulation
    └─ ./references/estimation.md

"I need survey-weighted regression"

What model?
├─ Linear regression (continuous Y)
│   └─ family="gaussian" → ./references/regression.md
├─ Logistic regression (binary Y)
│   └─ family="binomial" → ./references/regression.md
├─ Poisson regression (count Y)
│   └─ family="poisson" → ./references/regression.md
├─ Ordinal logistic / Cox survival / IV
│   └─ Not in svy — use rpy2 + R survey package (see rpy2 bridge below)
└─ Fixed effects + survey weights
    └─ Not directly supported — see Boundaries below

"I need to set up variance estimation"

What do you have?
├─ Design variables (strata, PSU, weights)
│   └─ Taylor linearization → ./references/design-weights.md
├─ Pre-computed replicate weights
│   ├─ BRR weights → ./references/design-weights.md
│   ├─ Jackknife weights → ./references/design-weights.md
│   └─ Bootstrap weights → ./references/design-weights.md
├─ Need to create replicate weights from design
│   └─ ./references/design-weights.md
└─ Not sure what I have
    └─ Read survey documentation first → ./references/design-weights.md (federal survey table)

"I need descriptive statistics from a survey"

What statistic?
├─ Population mean → ./references/estimation.md
├─ Population total → ./references/estimation.md
├─ Proportion → ./references/estimation.md
├─ Ratio (Y/X) → ./references/estimation.md
├─ Median / quantile → ./references/estimation.md
├─ Cross-tabulation → ./references/estimation.md
├─ By subgroup (domain estimation) → ./references/estimation.md
└─ Hypothesis test (t-test) → ./references/estimation.md

Boundaries

svy covers:

• Design-based estimation (descriptive and regression) for complex surveys
• Taylor and replicate-weight variance estimation
• Domain/subpopulation analysis
• Calibration and weight adjustment
• Survey data I/O

svy does NOT cover (use other tools):

• Fixed effects models — use pyfixest (survey weights + FE is methodologically complex; consult data-scientist skill)
• Panel data models (RE, FD, between) — use linearmodels
• Difference-in-differences — use pyfixest
• Causal inference methods (IV, RD, synthetic control) — use pyfixest/linearmodels/statsmodels
• Time series analysis — use statsmodels
• Machine learning — use scikit-learn
• Ordinal logistic, Cox proportional hazards, negative binomial — use rpy2 + R survey package
• Survey sampling design and sample size calculation — use data-scientist skill for methodology

The rpy2 Bridge

For models svy does not support (ordinal logistic, survival models, negative binomial GLM, cumulative link models), fall back to R's survey package via rpy2:

Decision rule: If the model family is not "gaussian", "binomial", or "poisson", use rpy2.

The R survey package (survey::svyglm, survey::svyolr, survey::svycoxph) covers the full range of survey-weighted models. Set up the survey design in R using the same design variables you would pass to svy.Design. See R survey package documentation at r-survey.r-forge.r-project.org for API details.

Legacy: samplics

samplics (2020-2026) is archived. svy supersedes it with a cleaner API, Polars integration, and expanded methods. If working with legacy code that uses samplics:

• The API is substantially different — TaylorEstimator/ReplicateEstimator classes are replaced by svy.Sample
• samplics used numpy arrays; svy uses Polars DataFrames
• Consult samplics documentation at samplics-org.github.io/samplics/ for legacy reference
• Migration requires rewriting, not find-and-replace

File-First Execution in Research Workflows

Important: In data research pipelines (see CLAUDE.md), svy analyses are executed through script files, not interactively. This ensures auditability and reproducibility.

The pattern:

1. Write estimation/regression code to scripts/stage8_analysis/{step}_{task-name}.py
2. Execute via Bash with automatic output capture wrapper script
3. Validation results get automatically embedded in scripts as comments
4. If failed, create versioned copy for fixes

Closely read agent_reference/SCRIPT_EXECUTION_REFERENCE.md for the mandatory file-first execution protocol. All survey analysis scripts must follow the Inline Audit Trail (IAT) standard — document design specification choices (why these strata/PSU/weights, what variance method, domain definitions) with # INTENT:, # REASONING:, and # ASSUMES: comments.

---

Quick Reference

Essential Import

import svy

Core Workflow

# 1. Load data
data = svy.io.read_stata("nhanes.dta")

# 2. Specify design
design = svy.Design(stratum="sdmvstra", psu="sdmvpsu", wgt="wtmec2yr")

# 3. Create sample object
sample = svy.Sample(data=data, design=design)

# 4. Estimate
mean_bmi = sample.estimation.mean("bmxbmi")
model = sample.glm.fit(y="bmxbmi", x=["ridageyr", svy.Cat("riagendr")], family="gaussian")

Core Operations

| Operation | Code | |-----------|------| | Design (Taylor) | svy.Design(stratum="s", psu="p", wgt="w") | | Sample object | svy.Sample(data=df, design=design) | | Mean | sample.estimation.mean("var") | | Total | sample.estimation.total("var") | | Proportion | sample.estimation.prop("var") | | Ratio | sample.estimation.ratio(y="num", x="denom") | | Median | sample.estimation.median("var") | | Domain estimation | sample.estimation.mean("var", by="group") | | Linear regression | sample.glm.fit(y="y", x=[...], family="gaussian") | | Logistic regression | sample.glm.fit(y="y", x=[...], family="binomial") | | Poisson regression | sample.glm.fit(y="y", x=[...], family="poisson") | | Categorical predictor | svy.Cat("varname") | | Read Stata | svy.io.read_stata("file.dta") | | Read SAS | svy.io.read_sas("file.sas7bdat") | | Read SPSS | svy.io.read_spss("file.sav") |

Topic Index

| Topic | Reference File | |-------|---------------| | Survey design setup | ./references/design-weights.md | | Taylor linearization | ./references/design-weights.md | | Replicate weights (BRR, jackknife, bootstrap) | ./references/design-weights.md | | Fay's BRR modification | ./references/design-weights.md | | Weight types and handling | ./references/design-weights.md | | Federal survey design patterns | ./references/design-weights.md | | Singleton PSU handling | ./references/design-weights.md | | Calibration and post-stratification | ./references/design-weights.md | | Reading SAS/SPSS/Stata files | ./references/design-weights.md | | Population means | ./references/estimation.md | | Population totals | ./references/estimation.md | | Proportions | ./references/estimation.md | | Ratios | ./references/estimation.md | | Medians and quantiles | ./references/estimation.md | | Domain / subpopulation estimation | ./references/estimation.md | | Cross-tabulations | ./references/estimation.md | | Survey-weighted t-tests | ./references/estimation.md | | Design effects (DEFF) | ./references/estimation.md | | Survey-weighted OLS | ./references/regression.md | | Survey-weighted logistic regression | ./references/regression.md | | Survey-weighted Poisson regression | ./references/regression.md | | Extracting regression results | ./references/regression.md | | Survey regression vs. WLS vs. cluster-robust | ./references/regression.md | | Categorical predictors (svy.Cat) | ./references/regression.md | | Model diagnostics in survey context | ./references/regression.md | | rpy2 bridge to R survey package | ./references/regression.md | | samplics migration | ./references/design-weights.md | | Polars DataFrame integration | ./references/design-weights.md |

Citation

When this library is used as a primary analytical tool, include in the report's Software & Tools references:

Diallo, M.S. svy: Python package for complex survey sampling and analysis [Computer software]. (Formerly samplics.)

Cite when: svy is used for survey-weighted estimation with complex survey designs (strata, PSU, replicate weights). Do not cite when: Only imported but no survey estimation performed.

For method-specific citations (e.g., variance estimation techniques), consult the reference files in this skill and agent_reference/CITATION_REFERENCE.md.