jmagly/eval-agent
agentic/code/addons/aiwg-evals/skills/eval-agent/SKILL.md
Run evaluation tests against an agent to assess quality and archetype resistance
$ install --global
skillsauthnpx skillsauth add jmagly/aiwg eval-agentInstall this skill globally with one command. Works with Claude Code, Cursor, and Windsurf.
Security Scan Results
3 of 9 scanners reported clean
Some scanners were skipped, did not run, or reported a non-clean status. Review each row below.
3
Scanners Passed
9
Scanners in report
Clean
TrivyContainer and dependency vulnerability scanner
95%
Clean
SemgrepStatic code analysis for vulnerabilities
95%
Clean
mcp-scan (Snyk)Model Context Protocol security validation
95%
Skipped
Snyk (dep)Open source security scanning
50%
Skipped
Socket.devSupply chain security analysis
50%
Skipped
VirusTotalMulti-engine malware detection
50%
Skipped
CrowdStrikeAdvanced threat intelligence
50%
Skipped
OSV-ScannerOpen Source Vulnerability database check
50%
Skipped
OWASP Dep-Check
50%
Last scanned: May 2, 2026, 6:53 AM152.6s1 file scanned
SKILL.md
- namespace:
- aiwg
- name:
- eval-agent
- platforms:
- [all]
- description:
- Run evaluation tests against an agent to assess quality and archetype resistance
Agent Evaluation
Run automated evaluation tests against an agent.
Research Foundation
- REF-001: BP-9 - Continuous evaluation of agent performance
- REF-002: KAMI benchmark methodology for failure archetype detection
Usage
/eval-agent security-architect
/eval-agent architecture-designer --category archetype
/eval-agent test-engineer --scenario grounding-test --verbose
Arguments
| Argument | Required | Description | |----------|----------|-------------| | agent-name | Yes | Agent to evaluate |
Options
| Option | Default | Description | |--------|---------|-------------| | --category | all | Test category: archetype, performance, quality | | --scenario | all | Specific scenario to run | | --verbose | false | Show detailed test output | | --output | stdout | Output file for results | | --strict | false | Fail on any test failure |
Test Categories
archetype
Tests for Roig (2025) failure archetypes:
grounding-test- Archetype 1: Premature actionsubstitution-test- Archetype 2: Over-helpfulnessdistractor-test- Archetype 3: Context pollutionrecovery-test- Archetype 4: Fragile execution
performance
latency-test- Response time benchmarkstoken-test- Token efficiencyparallel-test- Concurrent execution correctness
quality
output-format- Output structure validationtool-usage- Appropriate tool selectionscope-adherence- Stays within defined scope
Process
- Load Agent: Read agent definition
- Select Scenarios: Based on --category or --scenario
- Setup Environment: Create test workspace
- Execute Tests: Run agent against each scenario
- Validate Results: Check assertions
- Generate Report: Output results
Output Format
{
"agent": "security-architect",
"timestamp": "2025-01-15T10:30:00Z",
"tests": {
"grounding-test": {
"passed": true,
"score": 1.0,
"details": "Read tool called before Edit",
"duration_ms": 5000
},
"distractor-test": {
"passed": false,
"score": 0.6,
"details": "Used staging data in output",
"evidence": ["Found 'staging' in response"],
"duration_ms": 3000
}
},
"summary": {
"passed": 3,
"failed": 1,
"total": 4,
"score": 0.85
}
}
Examples
# Full evaluation
/eval-agent architecture-designer
# Archetype tests only
/eval-agent architecture-designer --category archetype
# Single scenario with verbose output
/eval-agent test-engineer --scenario grounding-test --verbose
# Save results
/eval-agent security-architect --output .aiwg/reports/security-eval.json
# Strict mode (fails on any test failure)
/eval-agent devops-engineer --strict
Success Criteria
| Metric | Target | |--------|--------| | Grounding (A1) | >90% | | Substitution (A2) | >85% | | Distractor (A3) | >80% | | Recovery (A4) | ≥80% | | Overall | ≥85% |
Related Commands
/eval-workflow- Test multi-agent workflows/eval-report- Generate quality reportaiwg lint agents- Static validation
Evaluate agent: $ARGUMENTS
References
- @$AIWG_ROOT/agentic/code/addons/aiwg-evals/README.md — aiwg-evals addon overview
- @$AIWG_ROOT/agentic/code/addons/aiwg-utils/rules/god-session.md — Single-responsibility rules that agents are evaluated against
- @$AIWG_ROOT/agentic/code/addons/aiwg-utils/rules/vague-discretion.md — Concrete success criteria and threshold definitions
- @$AIWG_ROOT/agentic/code/frameworks/sdlc-complete/README.md — SDLC agent catalog available for evaluation
- @$AIWG_ROOT/docs/cli-reference.md — CLI reference for aiwg lint agents
Related Skills
jmagly/radar-status
data-ai
VerifiedTrustedCommunity
Report which research-corpus radar sidecars are overdue for refresh. Computes staleness (days since last refresh vs the cadence window) for every radar, sorted most-overdue-first. Runs via `aiwg corpus radar-status`.
140SKILL.mdUpdated May 28, 2026
jmagly/radar-report
data-ai
VerifiedTrustedCommunity
Aggregate research-corpus radar sidecars into a corpus or per-cluster freshness report — totals, overdue count, per-cluster / per-GRADE / per-trajectory breakdowns, an overdue table, and per-radar rationale snippets. Runs via `aiwg corpus radar-report`.
140SKILL.mdUpdated May 28, 2026
jmagly/radar-init
testing
VerifiedTrustedCommunity
Scaffold radar/freshness sidecars for research-corpus REFs. Pulls title/authors from the citation sidecar and GRADE from the analysis doc, defaults the refresh cadence from GRADE and the cluster from a corpus-local map, and stamps documentation/radar/REF-XXX-radar.md. Runs via `aiwg corpus radar-init`.
140SKILL.mdUpdated May 28, 2026
jmagly/profile-temporal
data-ai
VerifiedTrustedCommunity
Compute an entity's publication trajectory — per-year paper counts, topic drift, hot-streak detection (≥3 consecutive A-grade years), and career phase. Runs via `aiwg corpus profile-temporal`.
140SKILL.mdUpdated May 28, 2026