giuseppe-trisciuoglio/task-quality-kpi

Task Quality KPI Framework

Overview

The Task Quality KPI Framework provides objective, quantitative metrics for evaluating task implementation quality.

Key Architecture: KPIs are auto-generated by a hook - you read the results, not run scripts.

┌─────────────────────────────────────────────────────────────┐
│  HOOK (auto-executes)                                       │
│  Trigger: PostToolUse on TASK-*.md                          │
│  Script: task-kpi-analyzer.py                               │
│  Output: TASK-XXX--kpi.json                                 │
├─────────────────────────────────────────────────────────────┤
│  SKILL / AGENT (reads output)                               │
│  Input: TASK-XXX--kpi.json                                  │
│  Action: Make evaluation decisions                          │
└─────────────────────────────────────────────────────────────┘

Why This Architecture?

| Problem | Solution | |---------|----------| | Skills can't execute scripts | Hook auto-runs on file save | | Subjective review_status | Quantitative 0-10 scores | | "Looks good to me" | Evidence-based evaluation | | Binary pass/fail | Graduated quality levels |

KPI File Location

After any task file modification, find KPI data at:

docs/specs/[ID]/tasks/TASK-XXX--kpi.json

KPI Categories

┌─────────────────────────────────────────────────────────────┐
│                    OVERALL SCORE (0-10)                     │
├─────────────────────────────────────────────────────────────┤
│  Spec Compliance (30%)                                      │
│  ├── Acceptance Criteria Met (0-10)                         │
│  ├── Requirements Coverage (0-10)                           │
│  └── No Scope Creep (0-10)                                  │
├─────────────────────────────────────────────────────────────┤
│  Code Quality (25%)                                         │
│  ├── Static Analysis (0-10)                                 │
│  ├── Complexity (0-10)                                      │
│  └── Patterns Alignment (0-10)                              │
├─────────────────────────────────────────────────────────────┤
│  Test Coverage (25%)                                        │
│  ├── Unit Tests Present (0-10)                              │
│  ├── Test/Code Ratio (0-10)                                 │
│  └── Coverage Percentage (0-10)                             │
├─────────────────────────────────────────────────────────────┤
│  Contract Fulfillment (20%)                                 │
│  ├── Provides Verified (0-10)                               │
│  └── Expects Satisfied (0-10)                               │
└─────────────────────────────────────────────────────────────┘

Category Weights

| Category | Weight | Why | |----------|--------|-----| | Spec Compliance | 30% | Most important - did we build what was asked? | | Code Quality | 25% | Technical excellence | | Test Coverage | 25% | Verification and confidence | | Contract Fulfillment | 20% | Integration with other tasks |

When to Use

• Reading KPI data for task quality evaluation
• Understanding quality metrics and scoring breakdown
• Deciding whether to iterate or approve based on quantitative data
• Integrating KPI checks into automated loops (agents_loop.py)
• Generating evidence-based evaluation reports

Instructions

1. Reading KPI Data (Primary Use)

DO NOT run scripts - read the auto-generated file:

Read the KPI file:
  docs/specs/001-feature/tasks/TASK-001--kpi.json

2. Understanding the Data

The KPI file contains:

{
  "task_id": "TASK-001",
  "evaluated_at": "2026-01-15T10:30:00Z",
  "overall_score": 8.2,
  "passed_threshold": true,
  "threshold": 7.5,
  "kpi_scores": [
    {
      "category": "Spec Compliance",
      "weight": 30,
      "score": 8.5,
      "weighted_score": 2.55,
      "metrics": {
        "acceptance_criteria_met": 9.0,
        "requirements_coverage": 8.0,
        "no_scope_creep": 8.5
      },
      "evidence": [
        "Acceptance criteria: 9/10 checked",
        "Requirements coverage: 8/10"
      ]
    }
  ],
  "recommendations": [
    "Code Quality: Moderate improvements possible"
  ],
  "summary": "Score: 8.2/10 - PASSED"
}

3. Making Decisions

Use overall_score and passed_threshold:

IF passed_threshold == true:
  → Task meets quality standards
  → Approve and proceed

IF passed_threshold == false:
  → Task needs improvement
  → Check recommendations for specific targets
  → Create fix specification

Integration with Workflow

In Task Review (evaluator-agent)

## Review Process

1. Read KPI file: TASK-XXX--kpi.json
2. Extract overall_score and kpi_scores
3. Read task file to validate
4. Generate evaluation report
5. Decision based on passed_threshold

In agents_loop

# Check KPI file exists
kpi_path = spec_path / "tasks" / f"{task_id}--kpi.json"

if kpi_path.exists():
    kpi_data = json.loads(kpi_path.read_text())
    
    if kpi_data["passed_threshold"]:
        # Quality threshold met
        advance_state("update_done")
    else:
        # Need more work
        fix_targets = kpi_data["recommendations"]
        create_fix_task(fix_targets)
        advance_state("fix")
else:
    # KPI not generated yet - task may not be implemented
    log_warning("No KPI data found")

Multi-Iteration Loop

Instead of max 3 retries, iterate until quality threshold met:

Iteration 1: Score 6.2 → FAILED → Fix: Improve test coverage
Iteration 2: Score 7.1 → FAILED → Fix: Refactor complex functions  
Iteration 3: Score 7.8 → PASSED → Proceed

Each iteration updates the KPI file automatically on task save.

Threshold Guidelines

| Score | Quality Level | Action | |-------|---------------|--------| | 9.0-10.0 | Exceptional | Approve, document best practices | | 8.0-8.9 | Good | Approve with minor notes | | 7.0-7.9 | Acceptable | Approve (if threshold 7.5) | | 6.0-6.9 | Below Standard | Request specific improvements | | < 6.0 | Poor | Significant rework required |

Recommended Thresholds

| Project Type | Threshold | Rationale | |--------------|-----------|-----------| | Production MVP | 8.0 | High quality required | | Internal Tool | 7.0 | Good enough | | Prototype | 6.0 | Functional over perfect | | Critical System | 8.5 | No compromises |

Metric Details

Spec Compliance Metrics

Acceptance Criteria Met

• Calculates: (checked_criteria / total_criteria) * 10
• Source: Task file checkbox count
• Example: 9/10 checked = 9.0

Requirements Coverage

• Calculates: Count of REQ-IDs this task covers
• Source: traceability-matrix.md
• Example: 4 requirements covered = 8.0

No Scope Creep

• Calculates: (implemented_files / expected_files) * 10
• Source: Task "Files to Create" vs actual files
• Penalizes: Missing files or unexpected additions

Code Quality Metrics

Static Analysis

• Java: Maven Checkstyle
• TypeScript: ESLint
• Python: ruff
• Score: 10 if passes, 5 if issues found

Complexity

• Calculates: Functions >50 lines
• Score: 10 - (long_functions_ratio * 5)
• Penalizes: Large, complex functions

Patterns Alignment

• Checks: Knowledge Graph patterns
• Source: knowledge-graph.json
• Validates: Implementation follows project patterns

Test Coverage Metrics

Unit Tests Present

• Calculates: min(10, test_files * 5)
• 2 test files = maximum score
• Penalizes: Missing tests

Test/Code Ratio

• Calculates: (test_count / code_count) * 10
• 1:1 ratio = 10/10
• Ideal: At least 1 test file per code file

Coverage Percentage

• Source: Coverage reports (JaCoCo, lcov, etc.)
• Calculates: coverage_percent / 10
• 80% coverage = 8.0

Contract Fulfillment Metrics

Provides Verified

• Checks: Files exist and export expected symbols
• Source: Task provides frontmatter
• Validates: Contract satisfied

Expects Satisfied

• Checks: Dependencies provide required files/symbols
• Source: Task expects frontmatter
• Validates: Prerequisites met

When KPI File is Missing

If TASK-XXX--kpi.json doesn't exist:

1. Task was never modified - Hook runs on file save
2. Hook failed - Check Claude Code logs
3. Task is new - Save the file first to trigger hook

DO NOT try to calculate KPIs manually. The hook runs automatically when:

• Task file is saved (Write tool)
• Task file is edited (Edit tool)

Best Practices

1. Always Check KPI File Exists

Before evaluating:

Check if KPI file exists:
  docs/specs/[ID]/tasks/TASK-XXX--kpi.json

If missing:
  - Task may not be implemented yet
  - Ask user to save the task file first

2. Trust the Metrics

The KPIs are objective. Only override with documented evidence:

• Critical security issue not in metrics
• Logic error not caught by static analysis
• Exceptional quality not measured

3. Iterate on Low KPIs

Target specific categories:

❌ "Fix code quality issues"
✅ "Improve Code Quality KPI from 5.2 to 7.0:
    - Complexity: Refactor processData() (5→8)
    - Patterns: Add error handling (6→8)"

4. Track KPI Trends

Monitor quality over time:

Sprint 1: Average KPI 6.8
Sprint 2: Average KPI 7.3 (+0.5)
Sprint 3: Average KPI 7.9 (+0.6)

Troubleshooting

KPI File Not Generated

Check:

1. Hook enabled in hooks.json
2. Task file name matches pattern TASK-*.md
3. File was actually saved (not just viewed)

KPI Scores Seem Wrong

Validate:

1. Check evidence field for data sources
2. Verify files exist at expected paths
3. Some metrics need build tools (Maven, npm)

Low Scores Despite Good Code

Possible causes:

• Missing test files
• No coverage report generated
• Acceptance criteria not checked
• Lint rules too strict

Fix the root cause, not just the score.

Examples

Example 1: Reading KPI Data

Read the KPI file to evaluate task quality:
  docs/specs/001-feature/tasks/TASK-042--kpi.json

Based on the data:
- Overall score: 6.8/10 (below threshold)
- Lowest KPI: Test Coverage (5.0/10)
- Recommendation: Add unit tests

Decision: REQUEST FIXES - target Test Coverage improvement

Example 2: Iteration Decision

Iteration 1 KPI: Score 6.2 → FAILED
- Spec Compliance: 7.0 ✓
- Code Quality: 5.5 ✗
- Test Coverage: 6.0 ✗

Fix targets:
1. Refactor complex functions (Code Quality)
2. Add test coverage (Test Coverage)

Iteration 2 KPI: Score 7.8 → PASSED ✓

Example 3: agents_loop Integration

# In agents_loop, after implementation step
kpi_file = spec_dir / "tasks" / f"{task_id}--kpi.json"

if kpi_file.exists():
    kpi = json.loads(kpi_file.read_text())
    
    if kpi["passed_threshold"]:
        print(f"✅ Task passed quality check: {kpi['overall_score']}/10")
        advance_state("update_done")
    else:
        print(f"❌ Task failed quality check: {kpi['overall_score']}/10")
        print("Recommendations:")
        for rec in kpi["recommendations"]:
            print(f"  - {rec}")
        advance_state("fix")

References

• evaluator-agent.md - Agent that uses KPI data for evaluation
• hooks.json - Hook configuration for auto-generation
• task-kpi-analyzer.py - Hook script (do not execute directly)
• agents_loop.py - Orchestrator that reads KPI for decisions