0xhoneyjar/eval
.claude/skills/eval-running/SKILL.md
Run evaluation suites against the Loa framework
$ install --global
skillsauthnpx skillsauth add 0xhoneyjar/loa-finn evalInstall this skill globally with one command. Works with Claude Code, Cursor, and Windsurf.
Security Scan Results
3 of 9 scanners reported clean
Some scanners were skipped, did not run, or reported a non-clean status. Review each row below.
3
Scanners Passed
9
Scanners in report
Clean
TrivyContainer and dependency vulnerability scanner
95%
Clean
SemgrepStatic code analysis for vulnerabilities
95%
Clean
mcp-scan (Snyk)Model Context Protocol security validation
95%
Skipped
Snyk (dep)Open source security scanning
50%
Skipped
Socket.devSupply chain security analysis
50%
Skipped
VirusTotalMulti-engine malware detection
50%
Skipped
CrowdStrikeAdvanced threat intelligence
50%
Skipped
OSV-ScannerOpen Source Vulnerability database check
50%
Skipped
OWASP Dep-Check
50%
Last scanned: Apr 29, 2026, 3:04 PM15.3s1 file scanned
SKILL.md
- name:
- eval
- description:
- Run evaluation suites against the Loa framework
- allowed-tools:
- Read, Grep, Glob, Bash(evals/harness/*), Bash(bats tests/*)
- schema_version:
- 1
- read_files:
- true
- search_code:
- true
- write_files:
- false
- - command:
- bats
- args:
- ["tests/*"]
- deny_raw_shell:
- true
- web_access:
- false
- user_interaction:
- false
- agent_spawn:
- false
- task_management:
- false
- cost-profile:
- moderate
Eval Running Skill
Run evaluation suites against the Loa framework to detect regressions and benchmark skill quality.
Usage
# Run framework correctness suite
/eval --suite framework
# Run regression suite
/eval --suite regression
# Run a single task
/eval --task constraint-proc-001-enforced
# Run all tasks for a skill
/eval --skill implementing-tasks
# Update baselines
/eval --suite framework --update-baseline --reason "Post-refactor re-baseline"
How It Works
- Parses arguments from the
/evalcommand - Delegates to
evals/harness/run-eval.shwith appropriate flags - Reports results via CLI or JSON output
Execution
When invoked, translate the user's request into run-eval.sh arguments:
# Default: run all default suites
./evals/harness/run-eval.sh --suite framework --trusted
# With suite specified
./evals/harness/run-eval.sh --suite <suite> --trusted
# With task specified
./evals/harness/run-eval.sh --task <task-id> --trusted
# With skill filter
./evals/harness/run-eval.sh --skill <skill-name> --trusted
# Update baseline
./evals/harness/run-eval.sh --suite <suite> --update-baseline --reason "<reason>" --trusted
# JSON output for programmatic use
./evals/harness/run-eval.sh --suite <suite> --json --trusted
Note: --trusted flag is always added for local execution. In CI, the container sandbox provides isolation.
Exit Codes
| Code | Meaning | |------|---------| | 0 | All pass, no regressions | | 1 | Regressions detected | | 2 | Infrastructure error | | 3 | Configuration error |
Constraints
- C-EVAL-001: ALWAYS submit baseline updates as PRs with rationale
- C-EVAL-002: ALWAYS ensure code-based graders are deterministic
Related Skills
0xhoneyjar/tests/fixtures/registry/skills/test-vendor/valid-skill
testing
VerifiedTrustedCommunity
# valid-skill Test skill with valid license for unit testing. ## Purpose Used in test_constructs_loader.bats to verify correct handling of valid licenses.
2SKILL.mdUpdated Mar 24, 2026
0xhoneyjar/tests/fixtures/registry/skills/test-vendor/grace-skill
testing
VerifiedTrustedCommunity
# grace-skill Test skill in license grace period for unit testing. ## Purpose Used in test_constructs_loader.bats to verify correct handling of licenses in grace period.
2SKILL.mdUpdated Mar 24, 2026
0xhoneyjar/tests/fixtures/registry/skills/test-vendor/expired-skill
testing
VerifiedTrustedCommunity
# expired-skill Test skill with expired license for unit testing. ## Purpose Used in test_constructs_loader.bats to verify correct handling of expired licenses.
2SKILL.mdUpdated Mar 24, 2026
0xhoneyjar/tests/fixtures/registry/packs/test-pack/skills/skill-b
testing
VerifiedTrustedCommunity
# skill-b Test skill B from test-pack for unit testing. ## Purpose Used in test_pack_support.bats to verify pack validation and skill loading.
2SKILL.mdUpdated Mar 24, 2026