jmagly/eval-workflow
agentic/code/addons/aiwg-evals/skills/eval-workflow/SKILL.md
Run evaluation tests against a multi-agent workflow to assess orchestration quality and failure archetype resistance
$ install --global
skillsauthnpx skillsauth add jmagly/aiwg eval-workflowInstall this skill globally with one command. Works with Claude Code, Cursor, and Windsurf.
Security Scan Results
3 of 9 scanners reported clean
Some scanners were skipped, did not run, or reported a non-clean status. Review each row below.
3
Scanners Passed
9
Scanners in report
Clean
TrivyContainer and dependency vulnerability scanner
95%
Clean
SemgrepStatic code analysis for vulnerabilities
95%
Clean
mcp-scan (Snyk)Model Context Protocol security validation
95%
Skipped
Snyk (dep)Open source security scanning
50%
Skipped
Socket.devSupply chain security analysis
50%
Skipped
VirusTotalMulti-engine malware detection
50%
Skipped
CrowdStrikeAdvanced threat intelligence
50%
Skipped
OSV-ScannerOpen Source Vulnerability database check
50%
Skipped
OWASP Dep-Check
50%
Last scanned: May 2, 2026, 6:52 AM69.0s1 file scanned
SKILL.md
- namespace:
- aiwg
- name:
- eval-workflow
- platforms:
- [all]
- description:
- Run evaluation tests against a multi-agent workflow to assess orchestration quality and failure archetype resistance
Workflow Evaluation
Run automated evaluation tests against a multi-agent workflow.
Research Foundation
- REF-001: BP-9 - Continuous evaluation of agent performance
- REF-002: KAMI benchmark methodology for real agentic task evaluation
Usage
/eval-workflow flow-security-review-cycle
/eval-workflow flow-inception-to-elaboration --scenario distractor-test
/eval-workflow flow-deploy-to-production --verbose --strict
Arguments
| Argument | Required | Description | |----------|----------|-------------| | workflow-name | Yes | Workflow (flow command) to evaluate |
Options
| Option | Default | Description | |--------|---------|-------------| | --scenario | all | Specific scenario to run | | --verbose | false | Show detailed test output | | --output | stdout | Output file for results | | --strict | false | Fail on any test failure | | --timeout | 300 | Maximum seconds per scenario |
What Gets Evaluated
Orchestration Quality
- Agent coordination: Parallel agents launched correctly in single message
- Handoff fidelity: Artifacts pass correctly between phases
- Gate enforcement: Phase gates checked before transition
Archetype Resistance
grounding-test— Archetype 1: Premature action without reading statedistractor-test— Archetype 3: Context pollution from irrelevant artifactsrecovery-test— Archetype 4: Fragile execution when subagent fails
Output Validation
- Required artifacts created in correct
.aiwg/paths - Document structure matches templates
- Traceability links intact
Process
- Load Workflow: Read flow command definition
- Select Scenarios: Based on --scenario flag or all applicable
- Setup Workspace: Create isolated
.aiwg/working/test space - Execute Flow: Run workflow against each scenario
- Validate Outputs: Check artifact presence, structure, and content
- Generate Report: Output results with pass/fail per assertion
- Cleanup: Remove test workspace
Output Format
{
"workflow": "flow-security-review-cycle",
"timestamp": "2026-04-01T10:30:00Z",
"scenarios": {
"grounding-test": {
"passed": true,
"score": 1.0,
"assertions": [
{"name": "threat-model-created", "passed": true},
{"name": "security-gate-run", "passed": true}
],
"duration_ms": 45000
},
"distractor-test": {
"passed": false,
"score": 0.7,
"assertions": [
{"name": "correct-assets-only", "passed": false, "evidence": "Distractor file referenced in output"}
],
"duration_ms": 38000
}
},
"summary": {
"passed": 4,
"failed": 1,
"total": 5,
"score": 0.80
}
}
Examples
# Full evaluation of a workflow
/eval-workflow flow-security-review-cycle
# Single scenario with verbose output
/eval-workflow flow-inception-to-elaboration --scenario grounding-test --verbose
# Strict mode with output saved
/eval-workflow flow-deploy-to-production --strict --output .aiwg/reports/deploy-eval.json
Success Criteria
| Metric | Target | |--------|--------| | Artifact creation | 100% | | Grounding compliance | >90% | | Distractor resistance | >80% | | Recovery success | ≥80% | | Overall | ≥85% |
Related Commands
/eval-agent- Test individual agents/eval-report- Generate aggregate quality reportaiwg lint agents- Static validation
Evaluate workflow: $ARGUMENTS
References
- @$AIWG_ROOT/agentic/code/addons/aiwg-evals/README.md — aiwg-evals addon overview
- @$AIWG_ROOT/agentic/code/addons/aiwg-utils/rules/subagent-scoping.md — Parallel agent coordination rules evaluated in workflows
- @$AIWG_ROOT/agentic/code/addons/aiwg-utils/rules/vague-discretion.md — Concrete success thresholds and test criteria
- @$AIWG_ROOT/agentic/code/frameworks/sdlc-complete/README.md — SDLC flow commands available for workflow evaluation
- @$AIWG_ROOT/docs/cli-reference.md — CLI reference for aiwg lint and eval commands
Related Skills
jmagly/radar-status
data-ai
VerifiedTrustedCommunity
Report which research-corpus radar sidecars are overdue for refresh. Computes staleness (days since last refresh vs the cadence window) for every radar, sorted most-overdue-first. Runs via `aiwg corpus radar-status`.
140SKILL.mdUpdated May 28, 2026
jmagly/radar-report
data-ai
VerifiedTrustedCommunity
Aggregate research-corpus radar sidecars into a corpus or per-cluster freshness report — totals, overdue count, per-cluster / per-GRADE / per-trajectory breakdowns, an overdue table, and per-radar rationale snippets. Runs via `aiwg corpus radar-report`.
140SKILL.mdUpdated May 28, 2026
jmagly/radar-init
testing
VerifiedTrustedCommunity
Scaffold radar/freshness sidecars for research-corpus REFs. Pulls title/authors from the citation sidecar and GRADE from the analysis doc, defaults the refresh cadence from GRADE and the cluster from a corpus-local map, and stamps documentation/radar/REF-XXX-radar.md. Runs via `aiwg corpus radar-init`.
140SKILL.mdUpdated May 28, 2026
jmagly/profile-temporal
data-ai
VerifiedTrustedCommunity
Compute an entity's publication trajectory — per-year paper counts, topic drift, hot-streak detection (≥3 consecutive A-grade years), and career phase. Runs via `aiwg corpus profile-temporal`.
140SKILL.mdUpdated May 28, 2026