fatby/critical-evaluation

Critical Evaluation Skill

A systematic framework for analyzing arguments, detecting cognitive biases, and improving decision quality through structured critical thinking.

When to Use This Skill

1. Complex decisions with significant consequences
2. Evaluating arguments or debates
3. Reviewing proposals or plans
4. Analyzing conflicting information
5. Before making important recommendations
6. When cognitive biases may be influencing judgment

The Critical Evaluation Framework

Phase 1: Argument Deconstruction

Break down the argument or decision into core components:

1. Claim: What is being asserted?
2. Evidence: What supports the claim?
3. Reasoning: How does evidence connect to claim?
4. Assumptions: What is being taken for granted?
5. Counterarguments: What opposing views exist?

Phase 2: Cognitive Bias Detection

Check for common cognitive biases:

| Bias | Description | Questions to Ask | |------|-------------|------------------| | Confirmation Bias | Seeking information that confirms existing beliefs | "Am I ignoring contradictory evidence?" | | Anchoring Bias | Relying too heavily on first piece of information | "Did the initial information distort my judgment?" | | Availability Heuristic | Overestimating importance of recent/memorable info | "Am I overweighing vivid examples?" | | Dunning-Kruger Effect | Overestimating one's own competence | "Do I really have the expertise needed?" | | Sunk Cost Fallacy | Continuing because of past investment | "Am I letting past costs influence future decisions?" | | Groupthink | Conforming to group opinion | "Am I agreeing just to maintain harmony?" | | Framing Effect | Being influenced by how info is presented | "Would I decide differently if framed another way?" |

Phase 3: Logical Fallacy Detection

Identify common logical fallacies:

| Fallacy | Pattern | Example | |---------|---------|---------| | Ad Hominem | Attacking person, not argument | "His idea is bad because he's inexperienced" | | Straw Man | Misrepresenting opponent's position | "You want to destroy the economy" (from "regulate pollution") | | False Dilemma | Presenting only two options | "Either we cut costs or we go bankrupt" | | Slippery Slope | Assuming one step leads to catastrophe | "If we allow this, then everything will collapse" | | Appeal to Authority | Using authority as evidence | "Expert X says so, so it must be true" | | Correlation ≠ Causation | Assuming causation from correlation | "Ice cream sales increase with drownings" | | Hasty Generalization | Conclusion from insufficient evidence | "Two failures mean the whole approach is wrong" |

Phase 4: Evidence Quality Assessment

Evaluate the quality of evidence:

1. Source credibility: Expert, biased, reputable?
2. Recency: How current is the information?
3. Methodology: How was evidence gathered?
4. Sample size: Is it statistically significant?
5. Replicability: Can results be reproduced?
6. Conflict of interest: Are there ulterior motives?

Phase 5: Alternative Perspectives

Deliberately consider opposing views:

1. Devil's Advocate: Argue against your own position
2. Red Team: Attack the decision as an adversary would
3. Multiple Scenarios: Consider best/worst/most likely cases
4. Long-term View: How will this look in 5 years?
5. Outside View: How have similar situations played out?

Phase 6: Confidence Calibration

Assess and communicate uncertainty:

1. Confidence Level: High/Medium/Low with reasoning
2. Key Uncertainties: What we don't know
3. Information Gaps: What would help reduce uncertainty
4. Sensitivity Analysis: Which assumptions matter most

Application Templates

Decision Evaluation Template

## Decision: [Brief description]

### 1. Core Components
- Claim: 
- Evidence: 
- Reasoning: 
- Assumptions: 

### 2. Bias Check
- Confirmation bias risk: [High/Medium/Low]
- Anchoring bias risk: [High/Medium/Low]
- Other biases identified: 

### 3. Fallacy Check
- Logical fallacies found: 
- Argument weaknesses: 

### 4. Evidence Quality
- Source credibility: 
- Methodology strength: 
- Key limitations: 

### 5. Alternative Perspectives
- Devil's advocate view: 
- Red team concerns: 
- Scenario analysis: 

### 6. Recommendation
- Confidence level: [High/Medium/Low]
- Key uncertainties: 
- Final recommendation: 

Argument Analysis Template

## Argument Analysis

### Original Argument
[Quote or summarize]

### Deconstruction
1. **Claim**: 
2. **Evidence**: 
3. **Reasoning**: 
4. **Assumptions**: 

### Critical Assessment
- **Strengths**: 
- **Weaknesses**: 
- **Biases detected**: 
- **Fallacies identified**: 

### Improved Argument
[How to strengthen the argument]

Integration with Other Skills

With Structured Reasoning

Use critical evaluation as a validation phase in the structured reasoning flywheel:

1. Phase 5: Validation → Apply critical evaluation
2. Phase 6: Decision → Incorporate critical insights
3. Phase 7: Review → Include bias awareness

With Self-Improving Agent

Log critical evaluation insights:

• Decision patterns with bias risk
• Successful bias mitigation strategies
• Common fallacies in domain

With Diverse Ideation

Ensure ideation avoids:

• Groupthink in brainstorming
• Anchoring on first ideas
• Confirmation bias in idea selection

Examples

Example 1: Business Decision

Situation: "Should we launch Product X?" Critical Evaluation:

1. Deconstruct: Claim (launch will succeed), Evidence (market research), Assumptions (no competitor response)
2. Bias check: Confirmation bias (only researched positive outcomes), Sunk cost fallacy (already spent $500k)
3. Fallacy check: False dilemma (launch vs. abandon, no middle ground)
4. Evidence: Research methodology flaws (small sample, leading questions)
5. Alternatives: Soft launch, pivot, partner with existing player
6. Confidence: Medium (due to competitor uncertainty)

Example 2: Technical Architecture

Situation: "Microservices are always better than monoliths" Critical Evaluation:

1. Deconstruct: Absolute claim, evidence (scalability needs), assumptions (team ready for complexity)
2. Bias check: Availability heuristic (recent microservices success stories), Bandwagon effect (everyone's doing it)
3. Fallacy check: Hasty generalization (from few examples), False dilemma (only two options)
4. Evidence: Case studies lack context (team size, domain)
5. Alternatives: Modular monolith, hybrid approach
6. Confidence: Low claim (not "always"), High for specific context

Implementation Notes

When NOT to Use

• Simple factual questions
• Time-sensitive emergencies
• Low-stakes decisions
• When heuristic clearly applies

Skill Activation Cues

• "Critically evaluate..."
• "What are the weaknesses of..."
• "Play devil's advocate for..."
• "Identify biases in..."
• "What assumptions are we making..."

Tool Integration

• memory_search: Find similar past decisions
• web_search: Gather counter-evidence
• sessions_spawn: Create devil's advocate agent
• write_file: Document evaluation

Continuous Improvement

Track effectiveness:

1. Decision outcomes vs. critical evaluation predictions
2. Bias detection accuracy
3. Common patterns in your thinking
4. Adjust framework based on results

References

• Kahneman, D. (2011). Thinking, Fast and Slow
• Tetlock, P. (2015). Superforecasting
• Heath, C. & Heath, D. (2013). Decisive