haoxuanlithuai/parameter-recovery-checker

Parameter Recovery Checker

Purpose

This skill encodes expert methodological knowledge for conducting parameter recovery studies -- a critical validation step before interpreting fitted model parameters. Parameter recovery determines whether a model's parameters are identifiable given the experimental design and sample size. A general-purpose programmer unfamiliar with computational modeling would not know that fitting a model is insufficient validation, or how to diagnose parameter tradeoffs and non-identifiability.

When to Use This Skill

• Before trusting fitted parameter values from any computational cognitive model
• When developing a new model and assessing whether parameters can be distinguished from data
• When planning an experiment and determining the minimum trial count for reliable parameter estimation
• When a reviewer asks for evidence of model identifiability
• When comparing models and needing to ensure each model can be distinguished (model recovery)
• When fitted parameters produce suspiciously extreme values or hit bounds

Research Planning Protocol

Before executing the domain-specific steps below, you MUST:

1. State the research question -- What specific question is this analysis/paradigm addressing?
2. Justify the method choice -- Why is this approach appropriate? What alternatives were considered?
3. Declare expected outcomes -- What results would support vs. refute the hypothesis?
4. Note assumptions and limitations -- What does this method assume? Where could it mislead?
5. Present the plan to the user and WAIT for confirmation before proceeding.

For detailed methodology guidance, see the research-literacy skill.

⚠️ Verification Notice

This skill was generated by AI from academic literature. All parameters, thresholds, and citations require independent verification before use in research. If you find errors, please open an issue.

Why Parameter Recovery Matters

Fitting a model to data and obtaining parameter estimates does NOT guarantee those estimates are meaningful (Wilson & Collins, 2019; Navarro, 2019). Common failure modes:

1. Non-identifiability: Multiple parameter combinations produce identical model predictions (e.g., drift rate and boundary in DDM trade off; Ratcliff & Tuerlinckx, 2002)
2. Insufficient data: Too few trials for the fitting procedure to recover true values
3. Local minima: Optimization converges to wrong parameter values
4. Model misspecification: The fitting procedure recovers parameters that do not reflect the assumed cognitive process

Parameter recovery is the standard diagnostic for these problems (Heathcote et al., 2015; Wilson & Collins, 2019).

Step-by-Step Recovery Procedure

Step 1: Define the Parameter Space

Choose ground-truth parameter values that span the plausible range for each parameter.

How many parameter sets to simulate?
 |
 +-- Minimum: 100 parameter sets (Wilson & Collins, 2019)
 |
 +-- Recommended: 500-1000 parameter sets for smooth recovery landscapes
 |
 +-- For publication: 1000+ parameter sets (Heathcote et al., 2015)

Sampling strategy:

| Strategy | When to Use | Source | |----------|-------------|--------| | Uniform grid | Few parameters (1-2), want complete coverage | Standard practice | | Latin hypercube | 3+ parameters, want space-filling without excessive samples | McKay et al., 1979 | | Random uniform | Simple, adequate for many parameters | Wilson & Collins, 2019 | | Prior-based sampling | Have informative priors on parameter ranges | Palestro et al., 2018 |

Range selection: Use ranges from published parameter estimates in the domain. For example:

• DDM drift rate v: 0.5 -- 4.0 (Ratcliff & McKoon, 2008)
• DDM boundary a: 0.5 -- 2.5 (Ratcliff & McKoon, 2008)
• DDM non-decision time Ter: 0.1 -- 0.5 s (Ratcliff & McKoon, 2008)
• ACT-R activation noise s: 0.1 -- 0.8 (Anderson, 2007)

Step 2: Simulate Data

For each ground-truth parameter set:

1. Match the experimental design exactly -- Same number of trials, conditions, and structure as the real experiment
2. Use the same model -- The generative model must be identical to the model you will fit
3. Include realistic noise -- Use the model's noise mechanism (do not add external noise)
4. Store the ground-truth parameters for later comparison

Critical: The number of simulated trials per participant must match the actual experiment. Recovery with 10,000 trials tells you nothing about recovery with 100 trials (Wilson & Collins, 2019).

Step 3: Fit the Model to Simulated Data

Apply the exact same fitting procedure you use for real data:

• Same optimization algorithm (e.g., MLE, Bayesian, chi-square minimization)
• Same parameter bounds and constraints
• Same starting values or initialization strategy
• Same convergence criteria

Multiple starting points: Run the optimizer from at least 5-10 random starting points per simulated dataset to avoid local minima (Heathcote et al., 2015).

Step 4: Evaluate Recovery Quality

Compare recovered parameters to true (ground-truth) parameters using multiple metrics.

Primary Metrics

| Metric | Formula | Good | Acceptable | Concerning | Source | |--------|---------|------|------------|------------|--------| | Pearson correlation (r) | cor(true, recovered) | r > 0.9 | r > 0.8 | r < 0.7 | Heathcote et al., 2015; rough benchmarks | | Bias | mean(recovered - true) | Near 0 | < 10% of range | > 20% of range | Wilson & Collins, 2019 | | RMSE | sqrt(mean((recovered - true)^2)) | Small relative to range | -- | Large relative to range | Standard | | Coverage | % of 95% CIs containing true value | ~95% | 85-100% | < 80% | Bayesian recovery |

Visualization (essential)

1. Scatter plot: Recovered vs. true for each parameter (identity line = perfect recovery)
2. Bland-Altman plot: Difference vs. mean (detect range-dependent bias)
3. Parameter correlation matrix: Off-diagonal correlations reveal tradeoffs

See references/recovery-diagnostics.md for visualization templates.

Step 5: Check Parameter Tradeoffs

Correlation between recovered parameters:

Are any pairs of recovered parameters correlated |r| > 0.5?
 |
 +-- YES --> These parameters trade off. Consider:
 | - Fixing one to a theoretically motivated value
 | - Reparameterizing the model
 | - Collecting more data to improve identifiability
 | - Reporting the tradeoff and interpreting cautiously
 |
 +-- NO --> Parameters are identifiable given this design

Common parameter tradeoffs in cognitive models:

| Model | Correlated Parameters | Nature of Tradeoff | Source | |-------|----------------------|-------------------|--------| | DDM | Drift rate (v) and boundary (a) | Speed-accuracy tradeoff | Ratcliff & Tuerlinckx, 2002 | | DDM | Non-decision time (Ter) and boundary (a) | Boundary absorbs timing variance | Ratcliff & Tuerlinckx, 2002 | | ACT-R | Noise (s) and threshold (tau) | Both affect retrieval probability | Anderson, 2007 | | RL models | Learning rate (alpha) and inverse temperature (beta) | Both control exploitation | Daw, 2011 | | Signal detection | d-prime and criterion (c) | Criterion shift mimics sensitivity change | Macmillan & Creelman, 2005 |

Model Recovery (Confusion Matrix)

Model recovery extends parameter recovery to test whether the correct model can be identified from data (Wagenmakers et al., 2004).

Procedure

1. For each candidate model M_k (k = 1, ..., K): a. Simulate data from M_k with representative parameters b. Fit ALL candidate models to the simulated data c. Select the best-fitting model using your comparison metric (AIC, BIC, Bayes factor)
2. Construct a K x K confusion matrix: rows = generating model, columns = selected model
3. Diagonal entries should dominate (correct model selected)

Quality Criteria

| Metric | Good | Concerning | Source | |--------|------|------------|--------| | Diagonal proportion | > 90% correct | < 70% correct | Wagenmakers et al., 2004 | | Off-diagonal patterns | Symmetric confusion | Asymmetric (one model always "wins") | Wilson & Collins, 2019 |

Warning: If model A is selected when data are generated from model B more than 20% of the time, those models are not distinguishable with your experimental design (Wilson & Collins, 2019).

Sample Size Effects

How Trial Count Affects Recovery

Recovery quality improves with more trials per participant. Test recovery at multiple trial counts:

| Trial Count | Expected Recovery | Recommendation | |-------------|-------------------|----------------| | < 50 trials | Often poor (r < 0.7) | Increase trials or simplify model | | 50-100 trials | Marginal for simple models | May suffice for 2-3 parameter models | | 100-200 trials | Adequate for most models | Standard for DDM (Ratcliff & McKoon, 2008) | | 200-500 trials | Good for complex models | Recommended for models with > 4 parameters | | 500+ trials | Excellent for most models | Required for hierarchical models |

Source: Wilson & Collins (2019); Ratcliff & Tuerlinckx (2002) for DDM-specific guidance.

Recovery as a Function of N

Plot recovery metrics (r, RMSE) as a function of trial count to determine the minimum viable N for your specific model and paradigm.

Landscape Analysis

Parameter Sensitivity Surfaces

For 1-2 key parameters, compute and visualize the objective function surface:

1. Fix all parameters except the target parameter(s)
2. Evaluate the objective function (e.g., negative log-likelihood) at a grid of values
3. Plot the surface (1D: line; 2D: contour or heatmap)

What to look for:

| Surface Feature | Interpretation | Action | |-----------------|---------------|--------| | Single sharp minimum | Well-identified parameter | Proceed with confidence | | Broad flat minimum | Parameter poorly constrained | Widen prior or collect more data | | Multiple minima | Non-convex; local minima risk | Use multiple starting points; consider reparameterization | | Ridge (elongated valley) | Parameter tradeoff | Two parameters are correlated; consider fixing one |

Reporting Standards

Minimum Reporting Checklist

When publishing a parameter recovery study:

• [ ] Number of simulated parameter sets (minimum 100; Wilson & Collins, 2019)
• [ ] Sampling strategy for ground-truth parameters (uniform, LHS, prior-based)
• [ ] Range of ground-truth values for each parameter (with justification)
• [ ] Number of simulated trials per dataset (must match real experiment)
• [ ] Fitting procedure used (same as for real data)
• [ ] Number of starting points for optimization
• [ ] Recovery metrics for each parameter: correlation (r), bias, RMSE
• [ ] Scatter plots: recovered vs. true for each parameter
• [ ] Parameter correlation matrix (recovered parameters)
• [ ] Model recovery confusion matrix (if performing model comparison)
• [ ] Recovery as a function of trial count (if applicable)

Where to Report

• Main text: Summary of recovery quality (r values, key plots)
• Supplementary: Full correlation matrices, all scatter plots, landscape analyses
• Parameter recovery is increasingly expected in top journals (Wilson & Collins, 2019; Navarro, 2019)

Common Pitfalls

1. Testing recovery with too many trials: Simulating 10,000 trials when the experiment has 100. Recovery will look excellent but is irrelevant to your actual data (Wilson & Collins, 2019).
2. Using different fitting procedures: The recovery study must use the identical optimization pipeline as the real-data analysis. Different starting values, bounds, or algorithms invalidate the test.
3. Ignoring parameter correlations: High marginal recovery (good r for each parameter) can coexist with strong parameter tradeoffs that distort interpretation. Always check the cross-parameter correlation matrix.
4. Reporting only correlation: Correlation measures rank-order recovery but ignores systematic bias. A parameter can have r = 0.95 but be consistently overestimated by 30%. Report bias and RMSE alongside r.
5. Sampling only near defaults: If ground-truth values cluster around typical defaults, recovery may look good only in that region. Sample across the full plausible range.
6. Neglecting model recovery: Good parameter recovery does not guarantee good model recovery. Two models can have recoverable parameters individually but be indistinguishable when competing (Wagenmakers et al., 2004).
7. Confusing identifiability with validity: A model can have perfectly recoverable parameters and still be a poor model of cognition. Recovery is necessary but not sufficient (Navarro, 2019).

References

• Anderson, J. R. (2007). How Can the Human Mind Occur in the Physical Universe? Oxford University Press.
• Daw, N. D. (2011). Trial-by-trial data analysis using computational models. In M. R. Delgado, E. A. Phelps, & T. W. Robbins (Eds.), Decision Making, Affect, and Learning. Oxford University Press.
• Heathcote, A., Brown, S. D., & Wagenmakers, E.-J. (2015). An introduction to good practices in cognitive modeling. In B. U. Forstmann & E.-J. Wagenmakers (Eds.), An Introduction to Model-Based Cognitive Neuroscience. Springer.
• Macmillan, N. A., & Creelman, C. D. (2005). Detection Theory: A User's Guide (2nd ed.). Lawrence Erlbaum Associates.
• McKay, M. D., Beckman, R. J., & Conover, W. J. (1979). A comparison of three methods for selecting values of input variables. Technometrics, 21(2), 239-245.
• Navarro, D. J. (2019). Between the devil and the deep blue sea: Tensions between scientific judgement and statistical model selection. Computational Brain & Behavior, 2(1), 28-34.
• Palestro, J. J., Sederberg, P. B., Osth, A. F., Van Zandt, T., & Turner, B. M. (2018). Likelihood-free methods for cognitive science. Springer.
• Ratcliff, R., & McKoon, G. (2008). The diffusion decision model: Theory and data for two-choice decision tasks. Neural Computation, 20(4), 873-922.
• Ratcliff, R., & Tuerlinckx, F. (2002). Estimating parameters of the diffusion model. Psychonomic Bulletin & Review, 9(3), 438-481.
• Wagenmakers, E.-J., Ratcliff, R., Gomez, P., & Iverson, G. J. (2004). Assessing model mimicry using the parametric bootstrap. Journal of Mathematical Psychology, 48(1), 28-50.
• Wilson, R. C., & Collins, A. G. (2019). Ten simple rules for the computational modeling of behavioral data. eLife, 8, e49547.

See references/ for diagnostic visualization templates and worked examples.