lyndonkl/heuristics-and-checklists

Heuristics and Checklists

Table of Contents

• Workflow
• Common Patterns
• Guardrails
• Quick Reference

Workflow

Copy this checklist and track your progress:

Heuristics & Checklists Progress:
- [ ] Step 1: Identify decision or procedure
- [ ] Step 2: Choose approach (heuristic vs. checklist)
- [ ] Step 3: Design heuristic or checklist
- [ ] Step 4: Test and validate
- [ ] Step 5: Apply and monitor
- [ ] Step 6: Refine based on outcomes

Step 1: Identify decision or procedure

What decision or procedure needs simplification? Is it repetitive? Time-sensitive? Error-prone? See resources/template.md.

Step 2: Choose approach (heuristic vs. checklist)

Heuristic for decisions (choose option). Checklist for procedures (sequence of steps). See resources/methodology.md.

Step 3: Design heuristic or checklist

Heuristic: Define simple rule (recognition, take-the-best, satisficing threshold). Checklist: List critical steps, add READ-DO or DO-CONFIRM format. See resources/template.md and resources/template.md.

Step 4: Test and validate

Pilot test with sample cases. Check: Does heuristic produce good enough decisions? Does checklist catch errors? See resources/methodology.md.

Step 5: Apply and monitor

Use in real scenarios. Track outcomes: decision quality, error rate, time saved. See resources/template.md.

Step 6: Refine based on outcomes

Adjust rules based on data. If heuristic fails in specific contexts, add exception. If checklist too long, prioritize critical items. See resources/methodology.md.

Validate using resources/evaluators/rubric_heuristics_and_checklists.json. Minimum standard: Average score ≥ 3.5.

Common Patterns

Pattern 1: Recognition Heuristic

• Rule: Choose the option you recognize over the one you don't
• Best for: Choosing between brands, cities, experts when quality correlates with fame
• Example: "Which city is larger, Detroit or Milwaukee?" (Choose Detroit if only one recognized)
• When works: Stable environments where recognition predicts quality
• When fails: Advertising creates false recognition, niche quality unknown

Pattern 2: Take-the-Best Heuristic

• Rule: Identify single most important criterion, choose based on that alone
• Best for: Multi-attribute decisions with one dominant factor
• Example: Hiring - "What's their track record on [critical skill]?" Ignore other factors.
• When works: One factor predictive, others add little value
• When fails: Multiple factors equally important, interactions matter

Pattern 3: Satisficing (Good Enough Threshold)

• Rule: Set minimum acceptable criteria, choose first option that meets them
• Best for: Routine decisions, time pressure, diminishing returns from analysis
• Example: "Candidate meets 80% of requirements → hire, don't keep searching for 100%"
• When works: Searching costs high, good enough > perfect delayed
• When fails: Consequences of suboptimal choice severe

Pattern 4: Aviation Checklist (DO-CONFIRM)

• Format: Perform actions from memory, then confirm each with checklist
• Best for: Routine procedures with critical steps (pre-flight, pre-surgery, deployment)
• Example: Pilot flies from memory, then reviews checklist to confirm all done
• When works: Experts doing familiar procedures, flow state preferred
• When fails: Novices, unfamiliar procedures (use READ-DO instead)

Pattern 5: Surgical Checklist (READ-DO)

• Format: Read each step, then perform, one at a time
• Best for: Unfamiliar procedures, novices, high-stakes irreversible actions
• Example: Surgical team reads checklist aloud, confirms each step before proceeding
• When works: Unfamiliar context, learning mode, consequences of error high
• When fails: Expert routine tasks (feels tedious, adds overhead)

Pattern 6: Fast & Frugal Decision Tree

• Format: Simple decision tree with 1-3 questions, binary choices at each node
• Best for: Triage, classification, go/no-go decisions
• Example: "Is customer enterprise? Yes → Assign senior rep. No → Is deal >$10k? Yes → Assign mid-level. No → Self-serve."
• When works: Clear decision structure, limited information needed
• When fails: Nuanced decisions, exceptions common

Guardrails

Key requirements:

1. Know when heuristics work vs. fail: Heuristics excel in stable, familiar environments with time pressure. They fail in novel, deceptive contexts (adversarial, misleading information). Don't use recognition heuristic when advertising creates false signals.

2. Satisficing ≠ low standards: "Good enough" threshold must be calibrated. Set based on cost of continued search vs. value of better option. Too low → poor decisions. Too high → analysis paralysis.

3. Checklists for critical steps only: Don't list every trivial action. Focus on steps that (1) are skipped often, (2) have serious consequences if missed, (3) not immediately obvious. Short checklists used > long checklists ignored.

4. READ-DO for novices, DO-CONFIRM for experts: Match format to user expertise. Forcing experts into READ-DO creates resistance and abandonment. Let experts flow, confirm after.

5. Test heuristics empirically: Don't assume rule works. Test on historical cases. Compare heuristic decisions to optimal decisions. If accuracy <80%, refine or abandon.

6. Bias awareness is not bias elimination: Knowing availability bias exists doesn't prevent it. Heuristics are unconscious. Need external checks (checklists, peer review, base rates) to counteract biases.

7. Update heuristics when environment changes: Rules optimized for past may fail in new context. Market shifts, technology changes, competitor strategies evolve. Re-validate quarterly.

8. Forcing functions beat reminders: "Don't forget X" fails. "Can't proceed until X done" works. Build constraints (e.g., deployment script requires all tests pass) rather than relying on memory.

Common pitfalls:

• ❌ Heuristic as universal law: "Always choose recognized brand" fails when dealing with deceptive advertising or niche quality.
• ❌ Checklist too long: 30-item checklist gets skipped. Keep to 5-10 critical items max.
• ❌ Ignoring base rates: "This customer seems like they'll buy" (representativeness heuristic) vs. "Only 2% of leads convert" (base rate). Use base rates to calibrate intuition.
• ❌ Anchoring on first option: "First candidate seems good, let's hire" without considering alternatives. Set satisficing threshold, then evaluate multiple options.
• ❌ Checklist as blame shield: "I followed checklist, not my fault" ignores responsibility to think. Checklists augment judgment, don't replace it.
• ❌ Not testing heuristics: Assume rule works without validation. Test on past cases, measure accuracy.

Quick Reference

Common heuristics:

| Heuristic | Rule | Example | Best For | |-----------|------|---------|----------| | Recognition | Choose what you recognize | Detroit > Milwaukee (size) | Stable correlations between recognition and quality | | Take-the-best | Use single most important criterion | Hire based on track record alone | One dominant factor predicts outcome | | Satisficing | First option meeting threshold | Candidate meets 80% requirements → hire | Time pressure, search costs high | | Availability | Judge frequency by ease of recall | Plane crashes seem common (vivid) | Recent, vivid events (WARNING: bias) | | Representativeness | Judge by similarity to prototype | "Looks like successful startup founder" | Stereotypes exist (WARNING: bias) | | Anchoring | Adjust from initial value | First price shapes negotiation | Numerical estimates (WARNING: bias) |

Checklist formats:

| Format | When to Use | Process | Example | |--------|-------------|---------|---------| | READ-DO | Novices, unfamiliar, high-stakes | Read step → Do step → Repeat | Surgery (WHO checklist) | | DO-CONFIRM | Experts, routine, familiar | Do from memory → Confirm with checklist | Aviation pre-flight | | Challenge-Response | Two-person verification | One reads, other confirms | Nuclear launch procedures |

Checklist design principles:

1. Keep it short: 5-10 items max (critical steps only)
2. Use verb-first language: "Verify backups complete" not "Backups"
3. One step per line: Don't combine "Test and deploy"
4. Checkbox format: ☐ Clear visual confirmation
5. Pause points: Identify natural breaks (before start, after critical phase, before finish)
6. Killer items: Mark items that block proceeding (e.g., ⚠ Tests must pass)

When to use heuristics vs. checklists:

| Decision Type | Use Heuristic | Use Checklist | |---------------|---------------|---------------| | Choose between options | ✓ Recognition, take-the-best, satisficing | ✗ Not applicable | | Sequential procedure | ✗ Not applicable | ✓ Pre-flight, deployment, surgery | | Complex multi-step | ✗ Too simplified | ✓ Ensures nothing skipped | | Routine decision | ✓ Fast rule (satisficing) | ✗ Overkill | | Error-prone procedure | ✗ Doesn't prevent errors | ✓ Catches mistakes |

Cognitive biases (when heuristics fail):

| Bias | Heuristic | Failure Mode | Mitigation | |------|-----------|--------------|------------| | Availability | Recent/vivid events judged as frequent | Overestimate plane crashes (vivid), underestimate heart disease | Use base rates, statistical data | | Representativeness | Judge by stereotype similarity | "Looks like successful founder" ignores base rate of success | Check against actual base rates | | Anchoring | First number shapes estimate | Initial salary offer anchors negotiation | Set own anchor first, adjust deliberately | | Confirmation | Seek supporting evidence | Only notice confirming data | Actively seek disconfirming evidence | | Sunk cost | Continue due to past investment | "Already spent $100k, can't stop now" | Evaluate based on future value only |

Inputs required:

• Decision/procedure: What needs simplification or systematization?
• Historical data: Past cases to test heuristic accuracy
• Critical steps: Which steps, if skipped, cause failures?
• Error patterns: Where do mistakes happen most often?
• Time constraints: How quickly must decision be made?

Outputs produced:

• heuristic-rule.md: Defined heuristic with conditions and exceptions
• checklist.md: Structured checklist with critical steps
• validation-results.md: Test results on historical cases
• refinement-log.md: Iterations based on real-world performance