popup-studio-ai/bkit-evals

bkit Evals — Skill Quality Evaluation Runner

v2.1.11 Sprint β FR-β2. Wraps evals/runner.js with input validation, result persistence, and structured reporting. Replaces the bare node evals/runner.js <skill> invocation that previously required users to remember argv structure and ignored timeout / sandbox concerns.

Arguments

| Argument | Description | Example | |----------|-------------|---------| | run <skill> | Execute the eval suite for one skill | /bkit-evals run gap-detector | | list | List all skills that have an eval.yaml definition | /bkit-evals list |

If no argument is provided, render the same output as list.

Behavior

run <skill>

1. Validate skill against /^[a-z][a-z0-9-]{0,63}$/. Reject anything else (no shell metacharacters, no slashes, no spaces) — see Security below.
2. Spawn node evals/runner.js --skill <skill> via child_process.spawnSync (argv form, no shell). Default timeout 30 s, max 120 s. The --skill flag form is mandated by the runner CLI and locked by L3 contract test.
3. Capture stdout / stderr. Parse the trailing JSON block via balanced-brace fallback (string-aware).
4. Apply fail-closed defense: if parsed === null and stdout includes Usage:, return reason: 'argv_format_mismatch'; if parsed === null otherwise, return reason: 'parsed_null'. Exit code 0 alone NEVER implies success — the parsed JSON must be present.
5. Persist the structured result to .bkit/runtime/evals-{skill}-{ISO timestamp}.json with stdout/stderr tails (2000 chars each), parsed payload, and reason field.
6. Render a one-line summary in the chat:
   • exit code
   • parsed pass/fail counts (if available)
   • path of the persisted result file

list

1. Read evals/config.json to enumerate skill classifications.
2. For each classification (workflow, capability, hybrid), list skills that have evals/{classification}/{skill}/eval.yaml.
3. Render a category-grouped table with skill name + a one-line note from the eval YAML (description field if present).

Security

• Skill name regex prevents argument injection. Anything outside [a-z][a-z0-9-]{0,63} is rejected with reason: invalid_skill_name.
• argv-array spawn (no shell). No template-string concatenation into command lines.
• Result file path is composed from a hardcoded base + sanitized skill name + timestamp; no traversal possible.
• Subprocess timeout enforced (default 30 s, hard cap 120 s) so a buggy eval cannot block the session indefinitely.

Module Dependencies

| Module | Function | Usage | |--------|----------|-------| | lib/evals/runner-wrapper.js | invokeEvals(skill, opts) | Validate + spawn + persist | | lib/evals/runner-wrapper.js | isValidSkillName(name) | Regex pre-check shared with list | | evals/runner.js | (subprocess) | Existing eval execution engine |

Result Schema

.bkit/runtime/evals-{skill}-{timestamp}.json:

{
  "skill": "gap-detector",
  "invokedAt": "<ISO 8601>",
  "exitCode": 0,
  "timedOut": false,
  "stdoutTail": "...",
  "stderrTail": "...",
  "parsed": { /* whatever runner.js prints as JSON, or null */ }
}

Examples

# Single eval
/bkit-evals run gap-detector

# Discovery
/bkit-evals list

Related

• /control trust — eval results contribute to trust score
• /code-review — uses eval data when assessing skills
• /bkit explore (FR-β1) — explore evals as a category

ARGUMENTS: