brycewang-stanford/assessment-design-guide

Assessment Design Guide

A skill for designing, validating, and analyzing educational assessments using modern psychometric methods. Covers classical test theory, item response theory, test construction, validity evidence, and computerized adaptive testing.

Classical Test Theory

Reliability Analysis

Classical test theory (CTT) models observed scores as the sum of a true score and error:

X = T + E

Key reliability coefficients:

| Coefficient | Method | Interpretation | |-------------|--------|----------------| | Cronbach's alpha | Internal consistency | Homogeneity of items | | Test-retest | Stability over time | Temporal consistency | | Parallel forms | Equivalent test versions | Form equivalence | | Split-half (Spearman-Brown) | Odd-even item split | Internal consistency | | Inter-rater (Cohen's kappa) | Multiple raters | Scoring agreement |

import numpy as np
import pandas as pd

def item_analysis(responses: pd.DataFrame, total_scores: pd.Series) -> pd.DataFrame:
    """
    Classical item analysis: difficulty, discrimination, point-biserial.
    responses: binary DataFrame (1=correct, 0=incorrect), items as columns.
    total_scores: total test score for each examinee.
    """
    results = []
    for item in responses.columns:
        scores = responses[item]
        difficulty = scores.mean()  # p-value (proportion correct)

        # Point-biserial correlation
        corr = scores.corr(total_scores)

        # Upper-lower discrimination (top/bottom 27%)
        n = len(total_scores)
        cutoff_high = total_scores.quantile(0.73)
        cutoff_low = total_scores.quantile(0.27)
        upper = scores[total_scores >= cutoff_high].mean()
        lower = scores[total_scores <= cutoff_low].mean()
        discrimination = upper - lower

        results.append({
            "item": item,
            "difficulty": round(difficulty, 3),
            "discrimination": round(discrimination, 3),
            "point_biserial": round(corr, 3),
            "flag": "review" if difficulty < 0.2 or difficulty > 0.9
                    or discrimination < 0.2 else "ok"
        })
    return pd.DataFrame(results)

Item Selection Guidelines

• Difficulty: Aim for p-values between 0.30 and 0.80 for maximum discrimination
• Discrimination: Items with D < 0.20 should be revised or removed
• Distractors: Each distractor should attract at least 5% of examinees
• Point-biserial: Should be positive and ideally above 0.25

Item Response Theory

The Three-Parameter Logistic Model

IRT provides a more rigorous framework than CTT by modeling the probability of a correct response as a function of ability and item parameters:

import numpy as np

def irt_3pl(theta: float, a: float, b: float, c: float) -> float:
    """
    Three-parameter logistic IRT model.
    theta: examinee ability (typically -3 to +3)
    a: discrimination parameter (slope, typically 0.5 to 2.5)
    b: difficulty parameter (location, same scale as theta)
    c: guessing parameter (lower asymptote, typically 0.0 to 0.35)
    Returns: probability of correct response
    """
    exponent = -a * (theta - b)
    return c + (1 - c) / (1 + np.exp(exponent))

# Item characteristic curves for three items
thetas = np.linspace(-3, 3, 100)
item_easy = [irt_3pl(t, a=1.0, b=-1.0, c=0.2) for t in thetas]
item_medium = [irt_3pl(t, a=1.5, b=0.0, c=0.2) for t in thetas]
item_hard = [irt_3pl(t, a=1.2, b=1.5, c=0.2) for t in thetas]

IRT Model Estimation

# Using the 'mirt' package in R (called via rpy2 or standalone)
# R code for fitting a 2PL model:
r_code = """
library(mirt)

# responses: binary matrix (examinees x items)
mod <- mirt(responses, model = 1, itemtype = "2PL")

# Item parameters
coef(mod, simplify = TRUE)

# Ability estimates (Expected A Posteriori)
theta_hat <- fscores(mod, method = "EAP")

# Model fit
M2(mod)  # limited-information fit statistic
itemfit(mod, fit_stats = "S_X2")
"""

Model Comparison

| Model | Parameters | Use Case | |-------|-----------|----------| | Rasch (1PL) | b only | Equal discrimination assumed; measurement-focused | | 2PL | a, b | Different discrimination; general purpose | | 3PL | a, b, c | Multiple choice with guessing | | Graded Response | a, b_k | Likert-scale or partial credit items | | Nominal Response | a_k, c_k | Multiple choice with informative distractors |

Validity Evidence

The Unified Validity Framework

Following the Standards for Educational and Psychological Testing (AERA/APA/NCME, 2014), validity is a unitary concept supported by five types of evidence:

1. Content evidence: Expert review confirms items represent the construct domain
2. Response process evidence: Think-aloud protocols confirm examinees engage intended cognitive processes
3. Internal structure evidence: Factor analysis confirms dimensionality matches the test blueprint
4. Relations to other variables: Correlations with external criteria (convergent, discriminant, predictive)
5. Consequences evidence: Test use leads to intended benefits without unintended harm

from factor_analyzer import FactorAnalyzer

# Confirmatory approach: check dimensionality
fa = FactorAnalyzer(n_factors=3, rotation="promax")
fa.fit(item_responses)

# Eigenvalues for scree plot
eigenvalues, _ = fa.get_eigenvalues()
print("Eigenvalues:", eigenvalues[:10])

# Factor loadings
loadings = pd.DataFrame(
    fa.loadings_,
    columns=["Factor1", "Factor2", "Factor3"],
    index=item_names
)
print(loadings.round(3))

Computerized Adaptive Testing

CAT Algorithm

Computerized adaptive testing selects items in real time to match examinee ability:

Initialize: theta_0 = 0 (prior mean)
For each item i = 1, 2, ..., until stopping rule met:
    1. Select item with maximum Fisher information at current theta
    2. Administer item, observe response
    3. Update theta estimate using maximum likelihood or Bayesian EAP
    4. Check stopping rule:
       - Fixed length (e.g., 30 items)
       - SE(theta) < threshold (e.g., 0.30)
       - Maximum time reached
Return: final theta estimate and standard error

Item Exposure Control

To prevent overuse of high-quality items and maintain test security:

• Sympson-Hetter method: Set maximum exposure rates per item (e.g., 0.25)
• a-stratified method: Divide item bank into strata by discrimination, sample within strata
• Shadow test approach: Assemble full shadow tests at each step, administer the optimal item from the shadow test

Tools and Software

• R mirt package: Full-featured IRT estimation, DIF analysis, CAT simulation
• Python irt library (py-irt): Bayesian IRT models using PyTorch
• jMetrik: Open-source Java application for classical and IRT analysis
• TAO (Testing Assistee par Ordinateur): Open-source assessment delivery platform
• Concerto: Open-source adaptive testing platform from Cambridge

Key References

• Embretson, S.E. and Reise, S.P. (2000). Item Response Theory for Psychologists. Lawrence Erlbaum.
• de Ayala, R.J. (2022). The Theory and Practice of Item Response Theory (2nd ed.). Guilford Press.
• AERA, APA, and NCME (2014). Standards for Educational and Psychological Testing.