casemark/managing-model-risk

Managing Model Risk

Structures model validation with independent testing, limitation documentation, and ongoing monitoring for quantitative risk models used in pricing, credit decisioning, market risk measurement, and capital adequacy.

When To Use

• Performing or coordinating independent model validation (initial or periodic)
• Documenting model limitations, assumptions, and compensating controls
• Building or updating a model risk inventory / model registry
• Preparing for regulatory examination of model governance (OCC SR 11-7, Fed SR 15-18, ECB TRIM) [VERIFY jurisdiction-specific guidance]
• Assessing materiality tier assignments for new or modified models
• Reviewing model performance monitoring results and back-testing exceptions

Inputs To Gather

• Model documentation package: methodology paper, mathematical specification, implementation notes, change log
• Validation scope memo: model name/ID, tier classification, intended use, prior validation findings
• Performance data: back-testing results, benchmarking outputs, sensitivity analyses, P&L attribution reports
• Data lineage: source systems, transformations, proxy usage, missing-data treatments
• Governance artifacts: model owner sign-off, approval committee minutes, prior MRA/MRIA items
• Regulatory context: applicable supervisory guidance and any outstanding findings from exams [VERIFY against current regulatory expectations]

Workflow

1. Classify the model and confirm tier

   • Map the model to the firm's tiering framework (e.g., Tier 1 = material/critical, Tier 2 = significant, Tier 3 = limited impact)
   • Confirm the model's intended use, downstream consumers, and materiality to financial statements or capital ratios
   • Verify that model registration in the central inventory is current

2. Evaluate conceptual soundness

   • Review theoretical basis, mathematical derivation, and key assumptions
   • Assess whether the chosen methodology is appropriate for the portfolio/risk type
   • Identify known limitations of the approach (e.g., distributional assumptions, stationarity requirements, calibration window sensitivity)

3. Perform independent testing

   • Replicate key calculations or run parallel implementations where feasible
   • Execute back-testing against realized outcomes (e.g., VaR exceptions, PD/LGD accuracy ratios, stress-test hit rates)
   • Run sensitivity and stress analyses on critical inputs and parameters
   • Benchmark against challenger models or industry proxies

4. Assess data quality and lineage

   • Trace input data from source systems through transformations to model consumption
   • Flag any proxy variables, manual overrides, or gap-filling techniques
   • Evaluate the representativeness of calibration and validation data sets relative to the current portfolio

5. Document findings and limitations

   • Categorize findings by severity: MRA (Matter Requiring Attention), MRIA (Matter Requiring Immediate Attention), or observation
   • For each limitation, specify the condition under which it becomes material and any compensating controls in place
   • Draft a clear limitations section that model users can reference when interpreting outputs

6. Establish ongoing monitoring framework

   • Define KPIs and thresholds for model performance (e.g., back-test exception counts, stability indices, discriminatory power metrics)
   • Set monitoring frequency aligned with model tier (Tier 1: monthly/quarterly; Tier 2: quarterly/semi-annual; Tier 3: annual) [VERIFY against firm policy]
   • Specify escalation triggers: what level of deterioration requires ad-hoc re-validation vs. parameter recalibration vs. model replacement

7. Report and obtain governance approval

   • Present validation report to the model risk committee or equivalent governance body
   • Track open findings in the MRA/MRIA tracker with owners and remediation deadlines
   • Confirm that model approval status is updated in the inventory (approved / approved with conditions / rejected)

Output

The deliverable is a Model Validation Report containing:

• Executive summary with overall validation opinion (satisfactory / satisfactory with conditions / unsatisfactory)
• Tier classification and scope of review
• Conceptual soundness assessment
• Independent testing results with statistical evidence (tables, charts)
• Data quality evaluation
• Findings table: finding ID, severity, description, compensating control, remediation owner, target date
• Limitations inventory with materiality conditions
• Ongoing monitoring plan with KPIs, thresholds, and escalation paths
• Appendices: replication code references, data samples, benchmarking details

Quality Checks

• Every finding has a clear severity rating, an assigned owner, and a remediation deadline
• Limitations are expressed in terms of the conditions under which the model may underperform — not just abstract caveats
• Back-testing uses an adequate observation window (minimum 1 year for market risk; 5+ years for credit risk where available) [VERIFY against internal policy and regulatory minimums]
• The report distinguishes between model risk inherent in the methodology vs. implementation risk (coding, data feeds)
• Ongoing monitoring thresholds are calibrated to historical performance distributions, not arbitrary round numbers
• All regulatory references cite the correct guidance version and effective date [VERIFY current versions of OCC 2011-12, SR 11-7, and any local equivalents]
• The validation opinion is consistent with the severity and count of open findings — no "satisfactory" rating with outstanding MRIAs