curiositech/windags-evaluator

WinDAGs Evaluator

Receive completed node outputs. Produce a ReviewResult containing a QualityVector. Enforce the four-layer quality model as a runtime protocol gate. Run Stage 1 on every node. Escalate to Stage 2 only when the economic formula justifies it.

DECISION POINTS

Main Evaluation Flow

1. Node output received
   ├─ Floor check passes? 
   │  ├─ YES → Continue to Wall check
   │  └─ NO → Return floor_failed ReviewResult (skip Wall, Ceiling, Envelope)
   │
   ├─ Wall check passes?
   │  ├─ YES → Evaluate Stage 2 escalation
   │  └─ NO → Skip Ceiling, compute Envelope only
   │
   └─ Stage 2 escalation needed?
      ├─ YES (feeds human gate) → Run Stage 2 Ceiling
      ├─ YES (final deliverable) → Run Stage 2 Ceiling  
      ├─ YES (irreversible) → Run Stage 2 Ceiling
      ├─ YES (P(fail) * waste > cost) → Run Stage 2 Ceiling
      └─ NO → Stage 1 only, compute Envelope

Partial Wall Grade Resolution

Wall grade is 0.4 (partial pass):
├─ Grade >= 0.6? → Treat as PASS, continue to Stage 2 decision
├─ Grade < 0.3? → Treat as FAIL, skip Ceiling
└─ Grade 0.3-0.6? → Check downstream criticality:
   ├─ Node feeds critical path? → Treat as FAIL
   └─ Node feeds non-critical? → Treat as PASS with warning

Conflicting Evaluator Sources

Peer=0.8, Downstream=0.3 (high disagreement):
├─ Difference > 0.4? → Flag for human review
├─ Downstream consumer is critical? → Weight Downstream 0.5, Peer 0.2
├─ Peer has domain expertise? → Weight Peer 0.4, Downstream 0.25
├─ Historical Peer reliability < 0.7? → Trust Downstream more
└─ Default → Use standard weights: Peer 0.25, Downstream 0.35

Economic Escalation Formula

Cost-benefit calculation:
├─ downstreamWaste = sum(dependent_node_costs) * cascade_probability
├─ failureProbability = skill_failure_rate * stage_factor * breaker_penalty
├─ expectedWaste = failureProbability * downstreamWaste
└─ escalate = expectedWaste > stage2_cost

FAILURE MODES

Missing Envelope Syndrome

Symptoms: ReviewResult returned without envelope field populated Diagnosis: Floor check short-circuited the evaluation flow improperly Fix: Envelope computation is mandatory for ALL nodes regardless of Floor result; run envelope calculation before returning any ReviewResult

Rubber Stamp Review

Symptoms: All evaluations scoring 0.9+ with minimal variation across different node types Diagnosis: Evaluator not applying layer-specific criteria; treating all outputs as equally good Fix: Implement per-layer scoring rubrics; require justification for scores above 0.8; cross-check with historical distributions

Escalation Cascade Explosion

Symptoms: >60% of nodes escalating to Stage 2 when historical rate should be 15-20% Diagnosis: Economic formula over-weighting failure probability or downstream costs Fix: Recalibrate failure probability floor to 0.01; cap downstream waste calculation at 5x node cost; audit cascade_probability multiplier

Bias Amplification Loop

Symptoms: Position-swapped evaluations showing >0.3 variance consistently
Diagnosis: Neural evaluators not properly applying bias mitigation techniques Fix: Enforce position swapping on ALL pairwise comparisons; apply length normalization; flag and retry when position bias detected

Quality Vector Collapse

Symptoms: Multi-dimensional quality getting converted to single scalar scores in downstream processing Diagnosis: Consuming systems incorrectly aggregating QualityVector dimensions Fix: Audit all QualityVector consumers; enforce per-dimension access patterns; remove any automatic scalar conversion utilities

WORKED EXAMPLES

Example 1: Floor Failure Case

Scenario: Code generation node produces syntactically invalid Python

Input: "Generate a Flask route handler"
Output: "def handle_route(
    return jsonify({'status': 'ok')"  // Missing closing paren, incomplete

Stage 1 Floor Check:
1. Parse against Python AST → FAIL (SyntaxError)
2. Schema validation → FAIL (incomplete function)
3. Contract check → FAIL (not valid Python)

Decision: Floor failed → Skip Wall, Ceiling, Envelope
Action: Return ReviewResult{stage1: {passed: false, floor: {passed: false, violations: ["SyntaxError", "IncompleteFunction"]}}, envelope: null}

Expert catches: Floor failure should immediately terminate evaluation - don't waste compute on Wall/Ceiling for fundamentally broken output Novice misses: Trying to evaluate "style" or "approach" of syntactically invalid code

Example 2: Escalation Decision with Cost/Waste Trade-off

Scenario: API documentation generation for internal service

Node: doc-generator (skill success_rate: 0.85, novice stage)
Downstream: 3 dependent nodes worth $0.12 total cost
Stage 2 review cost: $0.045

Calculation:
- failureProbability = (1 - 0.85) * 1.5 * 1.0 = 0.225
- downstreamWaste = $0.12 * 0.6 = $0.072 
- expectedWaste = 0.225 * $0.072 = $0.016
- escalate? $0.016 < $0.045 → NO

Decision: Stage 1 only
Reasoning: Low downstream impact, novice penalty not enough to justify deep review

Expert catches: Economic formula must account for cascade probability (0.6) not just raw downstream costs Novice misses: Forgetting to apply stage factor for novice skills, leading to under-escalation

Example 3: Conflicting Peer vs Downstream Signals

Scenario: SQL query optimization node evaluated by peer optimizer and downstream execution engine

Peer evaluation: 0.85 ("Elegant use of window functions, good indexing strategy")
Downstream evaluation: 0.35 ("Query times out on large datasets, missing LIMIT clause")

Resolution Decision Tree:
1. Difference = |0.85 - 0.35| = 0.5 > 0.4 → Flag for human review
2. Downstream is execution engine (critical for performance) → Weight downstream higher
3. Check peer historical reliability: 0.82 (good)
4. Final weights: Peer 0.15, Downstream 0.45

Composite Score: (0.15 * 0.85) + (0.45 * 0.35) = 0.285
Action: Mark as failing Wall check due to performance issues despite elegant structure

Expert catches: Performance trumps elegance; downstream execution failures are more costly than aesthetic suboptimality Novice misses: Over-weighting peer review because it "sounds more technical"; ignoring real-world execution constraints

QUALITY GATES

• [ ] ReviewResult contains valid QualityVector with all required dimensions populated
• [ ] Envelope computed for every node regardless of Floor/Wall results
• [ ] Stage 2 escalation decision logged with economic calculation breakdown
• [ ] Position swapping applied to all neural evaluations with bias detection
• [ ] Self-evaluation outcome excluded from quality scoring (process self-check retained at 0.15 weight)
• [ ] Near-miss events detected and logged for quality margins < 10%, timeout margins < 20%
• [ ] Cognitive telemetry recorded for all failures, mutations, and Stage 2 escalations
• [ ] Multiple evaluator sources properly weighted according to trust levels
• [ ] Floor failure short-circuits Wall/Ceiling evaluation correctly
• [ ] Quality vectors stored per-dimension without automatic scalar collapse

NOT-FOR BOUNDARIES

Do not use windags-evaluator for:

• Real-time streaming evaluation (use stream-evaluator instead)
• Evaluating human-generated content (use human-content-reviewer instead)
• Binary pass/fail decisions without quality scoring (use simple-validator instead)
• Evaluating non-DAG workflow outputs (use workflow-evaluator instead)
• Cross-domain skill comparison without context (use skill-comparator instead)

Delegate to other skills when:

• Node output requires domain-specific validation → Use domain-validator-{domain}
• Evaluation needs human-in-the-loop verification → Use human-gate-evaluator
• Quality assessment needs historical trend analysis → Use quality-trend-analyzer
• Evaluation requires cross-model consensus → Use ensemble-evaluator
• Assessment needs explanation generation → Use explainable-evaluator