nwave-ai/nw-dr-review-criteria

Documentation Review Criteria

Critique Dimensions

1. Classification Accuracy

Verify type assignment against DIVIO decision tree.

Questions: Do cited signals support assigned type? | Contradicting signals ignored? | Confidence appropriate? | Decision tree leads to same classification?

Verification: 1) Run decision tree independently 2) Check positive signals present 3) Check for red flags 4) Verify confidence matches signal strength

Severity: if wrong classification leads to wrong verdict = blocking.

2. Validation Completeness

Verify all type-specific criteria checked. Questions: All items checked? | Pass/fail correct? | Issues properly located? | Any criteria missed?

Tutorial (required): completable without external refs | steps numbered/sequential | verifiable outcomes | no assumed knowledge | builds confidence

How-to (required): clear goal | assumes fundamentals | single task | completion indicator | no basics teaching

Reference (required): all params documented | return values | error conditions | examples | no narrative

Explanation (required): addresses "why" | context/reasoning | alternatives considered | no task steps | conceptual model

3. Collapse Detection Correctness

Verify all five anti-patterns checked with accurate findings.

• Tutorial creep: explanation >20% | How-to bloat: teaching basics | Reference narrative: prose in entries
• Explanation task drift: steps in explanation | Hybrid horror: 3+ quadrants

Verification: independently scan, count lines per quadrant, compare to documentarist's findings, flag discrepancies.

4. Recommendation Quality

Criteria: Specific (exact what/where) | Actionable (author knows next step) | Prioritized (important first) | Justified (why it matters) | Root cause (underlying issue)

Bad: "Improve the documentation", "Make it clearer" Good: "Move explanation in section 3.2 (lines 45-60) to separate doc", "Add return value docs for login()"

5. Quality Score Accuracy

Verify six characteristics: Accuracy (factual claims verified?) | Completeness (gap analysis thorough?) | Clarity (Flesch 70-80?) | Consistency (style 95%+?) | Correctness (errors counted?) | Usability (structural assessment?)

Note: Documentarist cannot fully measure accuracy (needs expert) or usability (needs user testing). Verify limitations properly scoped.

6. Verdict Appropriateness

Verify verdict matches findings per decision matrix below.

Severity Framework

| Level | Definition | Action | |-------|-----------|--------| | Blocking | Wrong classification/verdict, missed collapse making doc unusable | Must fix | | High | Multiple criteria missed, collapse missed but usable | Should fix; may block | | Medium | Single criterion missed, miscalibrated confidence, false positive | Recommended | | Low | Format inconsistency, wording clarity | Optional |

Reject: any blocking | 3+ high | classification wrong | verdict contradicts findings Conditionally approve: 1-2 high not affecting verdict | multiple medium but core correct Approve: no blocking/high | medium noted but not blocking

Verdict Decision Matrix

• Approved: all checks pass or low-only failures | no collapse | quality gates met (Flesch 70-80, purity 80%+)
• Needs Revision: medium/low failures only | no collapse | fixable without restructuring
• Restructure Required: collapse detected | purity <80% | multiple user needs | requires splitting

Verification Algorithm

1. Count issues by severity 2. Check collapse_detection.clean 3. Check quality gates 4. Apply matrix 5. Compare to documentarist verdict 6. Flag discrepancy

Review Output Format

documentation_assessment_review:
  review_id: "doc_rev_{timestamp}"
  reviewer: "nw-documentarist-reviewer (Quill)"
  assessment_reviewed: "{path}"
  original_document: "{path}"

  classification_review:
    accurate: [boolean]
    confidence_appropriate: [boolean]
    independent_classification: "[your type]"
    match: [boolean]
    issues: [{issue, evidence, severity, recommendation}]

  validation_review:
    complete: [boolean]
    criteria_checked: "[X/Y required + Z/W additional]"
    missed_criteria: [list]
    issues: [{issue, severity, recommendation}]

  collapse_detection_review:
    accurate: [boolean]
    independent_findings: "[anti-patterns found]"
    false_positives: [count]
    missed_patterns: [list]
    issues: [{issue, severity, recommendation}]

  recommendation_review:
    quality: [high|medium|low]
    actionable: [boolean]
    properly_prioritized: [boolean]
    issues: [{issue, severity, improvement}]

  quality_score_review:
    accurate: [boolean]
    issues: [{score, issue, correction}]

  verdict_review:
    appropriate: [boolean]
    documentarist_verdict: "[their verdict]"
    recommended_verdict: "[your verdict]"
    verdict_match: [boolean]
    rationale: "{justification}"

  overall_assessment:
    assessment_quality: [high|medium|low]
    approval_status: [approved|rejected_pending_revisions|conditionally_approved|escalate_to_human]
    issue_summary: {blocking: N, high: N, medium: N, low: N}
    blocking_issues: [list]
    recommendations: [{priority, action}]

Review Iteration Limits

Maximum 2 revision cycles. After cycle 2: escalate to human, return approval_status: escalate_to_human with rationale.