ariffazil/arifos-observability

arifos-observability (O_Ω Orchestration Layer)

Purpose

Generate and analyze structured telemetry for skill runs, compiling health dashboards and triggering performance warnings.

Use When

1. Parsing telemetry outputs (timing.json, grading.json, benchmark.json) generated by any L2 skill run.
2. Generating a live skill execution dashboard or summary report for the operator.
3. Troubleshooting skill paralysis, token bloat, or latency bottlenecks.
4. Auditing a skill's historical pass rates, activation precision, and cross-skill collision rates.
5. Raising alert triggers if a critical skill (e.g. arifos-governance) falls below baseline trust metrics.

Do Not Use When

1. Executing new parallel test runs or synthetic benchmarks (use arifos-evals instead).
2. Auditing broad folder references or package configuration dates (use arifos-recursive-audit instead).
3. Modifying the telemetry logging schemas of other skills.

Inputs

• Run Telemetry: Local JSON/JSONL logs located at /root/.agents/telemetry/ or specific workspace folders.
• Execution Logs: Direct step outputs containing token counts, latency, and commands executed.

Procedure

1. Parse Telemetry Records: Read all execution logs (timing.json, benchmark.json) generated during active sessions.
2. Aggregate Run Metrics: Calculate total input tokens, output tokens, total latency, commands executed, and the occurrence rate of 888_HOLD events.
3. Compute Skill Health Indicators: Update historical scorecards:
   • Activation Precision: Ratio of correct activations vs false triggers.
   • Rollback Safety: Verified success rate of F1 rollback commands.
   • Cross-skill Collision Rate: Frequency of multi-skill triggers for a single task.
4. Audit Postcondition Status: Confirm if postconditions were programmatically verified or bypassed.
5. Compile Dashboard: Generate a clean, human-readable markdown dashboard mapping active skill health.
6. Trigger Performance Warnings: If activation precision < 0.90, or rollback safety < 1.00, raise an alert ticket and recommend description tuning.

Postconditions

1. All parsed telemetry logs are validated and aggregated without loss.
2. An active skill health dashboard is outputted to the session workspace.
3. Trigger drift and latency alert flags are generated and logged.

Failure Modes & Escalation

• Corrupted Log Payload: Telemetry logs contain invalid JSON structure. Action: Flag record corruption, skip the line, log ERR_LOG_CORRUPTION, and alert the developer.
• Database Read Timeout: Redis or postgres connection is blocked. Action: Fall back to parsing local file-backed telemetry directories.

Telemetry per Run

{
  "skill_name": "arifos-observability",
  "version": "1.0.0",
  "trigger_phrase": "{{trigger_phrase}}",
  "selected_reason": "{{selected_reason}}",
  "latency_ms": 0,
  "token_in": 0,
  "token_out": 0,
  "commands_run": 0,
  "artifacts_written": 1,
  "postcondition_pass": false,
  "human_approval_required": false,
  "hold_code": "{{hold_code}}"
}

Recursive Scorecard

• Activation Precision: [0.0 - 1.0] (Target: >0.98)
• Task Completion Rate: [0.0 - 1.0] (Target: >0.95)
• Rollback Safety: [0.0 - 1.0] (Target: 1.00)
• Context Efficiency: [0.0 - 1.0] (Target: >0.95)
• Doc Freshness: [0.0 - 1.0] (Target: 1.00)
• Cross-Skill Collision Rate: [0.0 - 1.0] (Target: 0.00)
• Human Trust Score: [0.0 - 1.0] (Target: >0.98)