brycewang-stanford/modeling-strategy-guide

Modeling Strategy Guide

A skill for strategic statistical modeling applied to academic research. Covers advanced modeling decisions, experimental design, causal inference, feature engineering, and the critical thinking required to move from data to defensible conclusions.

Overview

Senior data scientists distinguish themselves not by knowing more algorithms but by asking better questions, designing cleaner experiments, and being honest about what the data can and cannot tell them. This skill translates that professional discipline into a research context, helping academics apply modern data science practices to their empirical work. It covers the strategic decisions that matter most: when to use simple models versus complex ones, how to establish causality rather than mere correlation, and how to communicate uncertainty honestly.

The skill is particularly useful for researchers working with observational data who need causal inference techniques, those designing randomized experiments who need proper power calculations and analysis plans, and anyone building predictive models who needs to avoid common overfitting and leakage pitfalls.

Strategic Modeling Decisions

Model Selection Philosophy

Decision Framework:
1. Start with the simplest model that could answer your question
2. Add complexity only when diagnostics reveal inadequacy
3. Prefer interpretable models unless prediction accuracy is the sole goal
4. Always have a baseline (mean, majority class, last observation)

Model Complexity Ladder:
  Level 1: Descriptive statistics, cross-tabulations
  Level 2: Linear/logistic regression
  Level 3: Regularized regression (Lasso, Ridge, Elastic Net)
  Level 4: Tree ensembles (Random Forest, Gradient Boosting)
  Level 5: Deep learning (only with sufficient data and clear justification)

Feature Engineering Principles

import pandas as pd
import numpy as np

def engineer_features(df: pd.DataFrame, config: dict) -> pd.DataFrame:
    """
    Apply systematic feature engineering based on domain knowledge.

    config example:
    {
        'log_transform': ['income', 'citations'],
        'interactions': [('experience', 'education')],
        'polynomial': {'age': 2},
        'time_features': 'date_column',
        'lag_features': {'metric': [1, 7, 30]}
    }
    """
    df = df.copy()

    # Log transforms for right-skewed variables
    for col in config.get('log_transform', []):
        df[f'{col}_log'] = np.log1p(df[col])

    # Interaction terms
    for col_a, col_b in config.get('interactions', []):
        df[f'{col_a}_x_{col_b}'] = df[col_a] * df[col_b]

    # Polynomial features
    for col, degree in config.get('polynomial', {}).items():
        for d in range(2, degree + 1):
            df[f'{col}_pow{d}'] = df[col] ** d

    # Time-based features
    if 'time_features' in config:
        time_col = config['time_features']
        df[time_col] = pd.to_datetime(df[time_col])
        df[f'{time_col}_month'] = df[time_col].dt.month
        df[f'{time_col}_dayofweek'] = df[time_col].dt.dayofweek
        df[f'{time_col}_quarter'] = df[time_col].dt.quarter

    return df

Causal Inference Methods

Beyond Correlation

| Method | When to Use | Key Assumption | |--------|-----------|---------------| | Randomized experiment | You can randomly assign treatment | Proper randomization, no attrition | | Difference-in-differences | Policy change affects one group | Parallel trends pre-treatment | | Regression discontinuity | Treatment assigned by cutoff | No manipulation near cutoff | | Instrumental variables | Endogeneity present | Valid instrument (relevance + exclusion) | | Propensity score matching | Observational data, many confounders | No unobserved confounders | | Synthetic control | Single treated unit, many controls | Good pre-treatment fit |

Propensity Score Matching

from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import NearestNeighbors

def propensity_score_match(df, treatment_col, covariates, caliper=0.05):
    """
    Match treated and control units based on propensity scores.
    """
    # Estimate propensity scores
    X = df[covariates].values
    y = df[treatment_col].values

    lr = LogisticRegression(max_iter=1000, random_state=42)
    lr.fit(X, y)
    df['pscore'] = lr.predict_proba(X)[:, 1]

    # Match using nearest neighbor within caliper
    treated = df[df[treatment_col] == 1]
    control = df[df[treatment_col] == 0]

    nn = NearestNeighbors(n_neighbors=1, metric='euclidean')
    nn.fit(control[['pscore']].values)

    distances, indices = nn.kneighbors(treated[['pscore']].values)

    # Apply caliper
    valid = distances.flatten() < caliper
    matched_treated = treated[valid].index.tolist()
    matched_control = control.iloc[indices.flatten()[valid]].index.tolist()

    return {
        'matched_treated': matched_treated,
        'matched_control': matched_control,
        'n_matched': sum(valid),
        'n_unmatched': sum(~valid),
        'balance_check': 'Run standardized mean differences on covariates'
    }

Experimentation Design

A/B Testing for Research

from scipy import stats
import numpy as np

def design_experiment(baseline_rate, mde, alpha=0.05, power=0.80):
    """
    Calculate required sample size for a two-proportion z-test.

    Args:
        baseline_rate: Current conversion/success rate
        mde: Minimum detectable effect (absolute change)
        alpha: Significance level
        power: Statistical power
    """
    from statsmodels.stats.power import NormalIndPower
    effect_size = mde / np.sqrt(baseline_rate * (1 - baseline_rate))
    analysis = NormalIndPower()
    n = analysis.solve_power(
        effect_size=effect_size, alpha=alpha, power=power, ratio=1.0
    )
    return {
        'sample_size_per_group': int(np.ceil(n)),
        'total_sample_size': int(np.ceil(n)) * 2,
        'baseline_rate': baseline_rate,
        'minimum_detectable_effect': mde,
        'alpha': alpha,
        'power': power
    }

Pre-Analysis Plan Template

Before running any experiment, document:

1. Primary hypothesis: One clearly stated prediction.
2. Primary outcome metric: One pre-specified metric for the main test.
3. Sample size justification: Power calculation with assumptions.
4. Randomization procedure: How units are assigned to conditions.
5. Analysis method: Exact statistical test and model specification.
6. Multiple comparisons: How secondary analyses will be corrected.
7. Stopping rules: Conditions for early termination (if applicable).

Model Validation

Cross-Validation Strategy

| Data Type | Recommended CV | Rationale | |-----------|---------------|-----------| | i.i.d. data | Stratified K-fold (K=5 or 10) | Preserves class balance | | Time series | Time-series split (expanding window) | Prevents look-ahead bias | | Grouped data | Group K-fold | Prevents data leakage across groups | | Small dataset (n<200) | Leave-one-out or repeated K-fold | Maximizes training data | | Spatial data | Spatial blocking | Prevents spatial autocorrelation leakage |

Leakage Detection Checklist

• [ ] No future information used as features (check timestamps)
• [ ] No target-derived features (e.g., group means computed on full data)
• [ ] Train/test split performed before any preprocessing
• [ ] Cross-validation folds respect group structure
• [ ] Feature selection performed inside CV loop, not before
• [ ] If accuracy seems too good to be true, it probably is

Communication and Reporting

The Senior DS Reporting Standard

• Lead with the business/research question, not the algorithm.
• Report confidence intervals, not just point estimates.
• Show what you tried that did not work (negative results matter).
• Quantify uncertainty: "The model predicts X with a 95% interval of [a, b]."
• Be explicit about limitations and assumptions.
• Use visualizations that a domain expert (not a statistician) can interpret.

References

• Angrist, J. D. & Pischke, J.-S. (2009). Mostly Harmless Econometrics. Princeton University Press.
• Cunningham, S. (2021). Causal Inference: The Mixtape. Yale University Press.
• Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning (2nd ed.). Springer.