kayba-ai/kayba-stage-1-api-analysis

Stage 1: Kayba API Analysis (Fetch-Only Mode)

Fetch pre-computed insights from the Kayba API. Traces have already been uploaded and analyzed — this stage only pulls results.

Inputs

• TRACES_FOLDER — passed by the orchestrator but ignored in this stage. Traces are already uploaded and analyzed on the Kayba side. Do NOT upload, validate, or read trace files.

Process

Step 1: Setup

Ensure eval/ directory exists at the project root.

Step 2: Fetch insights

kayba insights list --json > eval/insights.json

If kayba is not found in PATH, search common locations (.venv/bin/kayba, project virtualenvs). If found, use the full path. If not found anywhere, report the error and stop.

If KAYBA_API_KEY is not set, report the error and stop.

Step 3: Insight quality gate

Read eval/insights.json and run quality checks before building the summary:

1. Empty check: if the insights array is empty (0 insights returned), report this as a warning. Write a minimal summary noting "0 insights generated" and stop — downstream stages cannot proceed without insights.
2. Duplicate detection: compare insight content fields pairwise. If two insights cover substantially the same behavior (same section, overlapping evidence traces, similar corrective action), flag them as potential duplicates in the summary. Do not remove them — just annotate.
3. Evidence coverage: for each insight, check if the evidence field references specific traces (e.g., "task_7 turn 4"). Insights with no trace-specific evidence are lower quality — flag as "low-evidence" in the summary.
4. Vote signal: insights with status: "accepted" and helpful > 0 have been human-validated. Insights with status: "new" and helpful: 0, harmful: 0 are unvalidated — note this distinction in the summary.

Log the quality gate result: "Insight quality: {total} insights, {accepted} accepted, {new_unvalidated} unvalidated, {duplicates} potential duplicate pairs, {low_evidence} low-evidence"

Step 4: Build structured summary

Extract a structured summary of each insight:

• Insight ID and title/summary (use the section field as the title)
• Status
• Evidence citations — specific trace references, error strings, behavioral patterns the reflector identified
• Justification / reasoning chain — the reflector's full analysis of why this is a real pattern
• Confidence score if available
• Helpful/harmful counts if available
• Quality flags from Step 3 (potential duplicate, low-evidence, unvalidated)

Write the structured summary to eval/stage1_insights_summary.md using this format:

# Kayba Insights Summary

Generated from: Kayba API (pre-computed analysis)
Total insights: N
Quality: {accepted} accepted, {unvalidated} unvalidated, {duplicate_pairs} potential duplicate pairs, {low_evidence} low-evidence

## Insight: [ID] — [section title]
**Status:** [status] [quality flags if any, e.g., "[potential duplicate with ID]", "[low-evidence]", "[unvalidated]"]
**Confidence:** [score if available]
**Evidence:**
- [citation 1 — trace reference, error string, or behavioral pattern]
- [citation 2]
**Justification:** [reflector's reasoning for why this is a real pattern]
**Helpful/Harmful:** [counts if available]

---
[repeat for each insight]

Error handling

• If kayba is not found in PATH or common locations, report the error and stop
• If KAYBA_API_KEY is not set, report the error and stop
• If kayba insights list fails (network error, auth error), report the error and stop
• If 0 insights are returned, write a minimal summary and stop — downstream stages need insights

Outputs

• eval/insights.json — raw API response
• eval/stage1_insights_summary.md — structured summary with quality annotations for downstream stages