0xhoneyjar/feedback-observing

Feedback Observing

Read agent interaction logs across all packs, detect operational patterns (error rates, duration outliers, zero-invocation skills), and emit state-transition FeedbackEvents when patterns change.

This skill closes the agent feedback loop — agents report back what worked and what failed.

---

Core Principle

State-transition only emission. Only emit FeedbackEvents when a skill's operational state changes (healthy → concerning, concerning → healthy). This prevents recursive signal inflation from repeated analysis of the same stable state.

---

Triggers

/feedback-observe                    # Analyze all packs' agent logs
/feedback-observe --pack observer    # Specific pack only
/feedback-observe --since 7d         # Time range (default: 30d)

---

When to Use

• Periodically to check agent health across packs
• After deploying new skills or modifying existing ones
• When suspecting a skill is silently failing
• As part of a daily/weekly maintenance routine

---

Workflow

Phase 1: Collect Logs

Glob: grimoires/*/agent-logs/*.jsonl

For each JSONL file:
  Parse each line (skip malformed lines, log warning)
  Filter by --pack if specified
  Filter by --since if specified (compare ts field)

Collect all log entries into a unified dataset.

If no log files found: Report "No agent logs found. Skills need to emit logs — see agent-log-format.md" and exit.

Phase 2: Detect Patterns

Group entries by pack and skill, then compute:

For each (pack, skill) pair:

  1. ERROR RATE
     error_count = entries where status="error"
     total_count = all entries
     error_rate = error_count / total_count

     IF error_rate > 0.20 → state = "concerning"
     IF error_rate <= 0.10 → state = "healthy"
     IF 0.10 < error_rate <= 0.20 → state = "warning"

  2. DURATION OUTLIERS
     Compute median duration_ms across all entries for this skill
     IF any entry has duration_ms > 2x median → flag as outlier

     IF outlier_count / total_count > 0.20 → state = "slow"

  3. ZERO-INVOCATION SKILLS
     Compare skills found in logs against manifest.json skills arrays
     Skills in manifest but NOT in logs → "dormant" (potential discoverability problem)

  4. RETRY PATTERNS
     Detect same skill invoked 3+ times within 5 minutes → "friction"

  5. MISSING ARTIFACT PATTERNS
     Entries where error contains "not found" or "missing" → "missing dependency"

Phase 3: Compare with Previous Report

Read previous report: grimoires/observer/agent-feedback/report-latest.md

IF previous report exists:
  Parse previous state per skill
  Compare current state with previous state

  ONLY emit FeedbackEvents for STATE TRANSITIONS:
    - healthy → concerning (degradation)
    - concerning → healthy (recovery)
    - new skill appeared as "dormant"
    - skill crossed duration outlier threshold for first time

  DO NOT re-emit for unchanged states.

IF no previous report:
  Treat all current states as new (emit for any non-healthy state)

Phase 4: Emit State-Transition FeedbackEvents

For each state transition detected, emit via the Loa event bus:

source .claude/scripts/lib/event-bus.sh

emit_event "observer.state_transition" \
  '{
    "domain": "code",
    "target": {
      "type": "artifact",
      "selector": "{pack}/{skill}"
    },
    "signal": {
      "direction": "{negative for degradation, positive for recovery}",
      "weight": 0.5,
      "specificity": 0.7,
      "content": "{state_transition}: {description}",
      "kind": "behavioral"
    },
    "fingerprint": "{sha256 of '{pattern_type}:{pack}:{skill}:{window_start}:{window_end}'}"
  }' \
  "observer/feedback-observing" \
  "" "" \
  "{pack}/{skill}"

The bus auto-generates id, time, specversion in the CloudEvents envelope.

Fingerprint deduplication: The fingerprint field maps to the bus idempotency key. Same fingerprint = same finding for the same time window. The bus .idempotency/ store prevents duplicate processing.

Phase 5: Generate Report

Write to grimoires/observer/agent-feedback/report-{YYYY-MM-DD}.md:

---
type: agent-feedback-report
generated: "{RFC 3339 UTC}"
period: "{start} to {end}"
packs_analyzed: {N}
skills_analyzed: {N}
transitions_detected: {N}
---

# Agent Feedback Report: {date}

## Skill Health Matrix

| Pack | Skill | Invocations | Error Rate | Median Duration | State |
|------|-------|-------------|-----------|-----------------|-------|
| observer | observing-users | 12 | 8% | 340ms | healthy |
| artisan | inscribing-taste | 5 | 40% | 120ms | concerning |
| ... | ... | ... | ... | ... | ... |

## State Transitions (since last report)

| Skill | Previous | Current | Evidence |
|-------|----------|---------|----------|
| artisan/inscribing-taste | healthy | concerning | Error rate 8% → 40% |

## Duration Analysis

| Skill | Median | P95 | Outliers |
|-------|--------|-----|----------|
| ... | ... | ... | ... |

## Dormant Skills

| Pack | Skill | In Manifest | Invocations |
|------|-------|-------------|-------------|
| ... | ... | yes | 0 |

## Suggested Actions

- {actionable recommendation per concerning skill}

Also copy to grimoires/observer/agent-feedback/report-latest.md (overwrite).

Create grimoires/observer/agent-feedback/ directory if it doesn't exist.

---

Error Handling

| Error | Resolution | |-------|------------| | No log files found | Report empty state, exit 0 | | Malformed JSONL line | Skip line, log warning, continue | | No previous report | Treat all states as new | | Manifest not found | Skip dormant skill detection | | --pack not found in logs | Report "No logs for pack {name}" |

---

Integration Points

• Reads: grimoires/*/agent-logs/*.jsonl, grimoires/observer/agent-feedback/report-latest.md
• Writes: grimoires/observer/agent-feedback/report-{date}.md, grimoires/shared/feedback/events/{date}.jsonl
• Depends on: Skills emitting agent logs (Task 1.7)
• Consumed by: /artisan-patterns cross-domain analysis, /stale confidence system

---

Validation

• [ ] Reads all packs' agent logs via glob
• [ ] Correctly computes error rate, duration outliers
• [ ] Only emits FeedbackEvents on state transitions
• [ ] Fingerprint prevents duplicate emissions for same window
• [ ] Report includes all analyzed skills
• [ ] report-latest.md is updated
• [ ] --pack filter works correctly
• [ ] --since filter works correctly
• [ ] Handles empty/missing log directories gracefully