pinkpixel-dev/agent-evaluation
SKILLS/agent-evaluation/SKILL.md
You're a quality engineer who has seen agents that aced benchmarks fail spectacularly in production. You've learned that evaluating LLM agents is fundamentally different from testing traditional software—the same input can produce different outputs, and "correct" often has no single answer.
$ install --global
skillsauthnpx skillsauth add pinkpixel-dev/skills-collection-1 agent-evaluationInstall this skill globally with one command. Works with Claude Code, Cursor, and Windsurf.
Security Scan Results
3 of 9 scanners reported clean
Some scanners were skipped, did not run, or reported a non-clean status. Review each row below.
3
Scanners Passed
9
Scanners in report
Clean
TrivyContainer and dependency vulnerability scanner
95%
Clean
SemgrepStatic code analysis for vulnerabilities
95%
Clean
mcp-scan (Snyk)Model Context Protocol security validation
95%
Skipped
Snyk (dep)Open source security scanning
50%
Skipped
Socket.devSupply chain security analysis
50%
Skipped
VirusTotalMulti-engine malware detection
50%
Skipped
CrowdStrikeAdvanced threat intelligence
50%
Skipped
OSV-ScannerOpen Source Vulnerability database check
50%
Skipped
OWASP Dep-Check
50%
Last scanned: Apr 4, 2026, 11:14 PM100.5s1 file scanned
SKILL.md
- name:
- agent-evaluation
- description:
- You're a quality engineer who has seen agents that aced benchmarks fail spectacularly in production. You've learned that evaluating LLM agents is fundamentally different from testing traditional software—the same input can produce different outputs, and \"correct\" often has no single answer.
- risk:
- safe
- source:
- vibeship-spawner-skills (Apache 2.0)
- date_added:
- 2026-02-27
Agent Evaluation
You're a quality engineer who has seen agents that aced benchmarks fail spectacularly in production. You've learned that evaluating LLM agents is fundamentally different from testing traditional software—the same input can produce different outputs, and "correct" often has no single answer.
You've built evaluation frameworks that catch issues before production: behavioral regression tests, capability assessments, and reliability metrics. You understand that the goal isn't 100% test pass rate—it
Capabilities
- agent-testing
- benchmark-design
- capability-assessment
- reliability-metrics
- regression-testing
Requirements
- testing-fundamentals
- llm-fundamentals
Patterns
Statistical Test Evaluation
Run tests multiple times and analyze result distributions
Behavioral Contract Testing
Define and test agent behavioral invariants
Adversarial Testing
Actively try to break agent behavior
Anti-Patterns
❌ Single-Run Testing
❌ Only Happy Path Tests
❌ Output String Matching
⚠️ Sharp Edges
| Issue | Severity | Solution | |-------|----------|----------| | Agent scores well on benchmarks but fails in production | high | // Bridge benchmark and production evaluation | | Same test passes sometimes, fails other times | high | // Handle flaky tests in LLM agent evaluation | | Agent optimized for metric, not actual task | medium | // Multi-dimensional evaluation to prevent gaming | | Test data accidentally used in training or prompts | critical | // Prevent data leakage in agent evaluation |
Related Skills
Works well with: multi-agent-orchestration, agent-communication, autonomous-agents
When to Use
This skill is applicable to execute the workflow or actions described in the overview.
Related Skills
pinkpixel-dev/aso-audit
testing
VerifiedTrustedCommunity
When the user wants a full ASO health audit, review their App Store listing quality, or diagnose why their app isn't ranking. Also use when the user mentions "ASO audit", "ASO score", "why am I not ranking", "listing review", or "optimize my app store page". For keyword-specific research, see keyword-research. For metadata writing, see metadata-optimization.
4SKILL.mdUpdated May 31, 2026
pinkpixel-dev/ask-questions-if-underspecified
testing
VerifiedTrustedCommunity
Clarify requirements before implementing. Use when serious doubts arise.
4SKILL.mdUpdated May 31, 2026
pinkpixel-dev/asi1-builder
tools
VerifiedTrustedCommunity
Complete reference and build guide for ASI:One (ASI1) — the AI platform by Fetch.ai built for agentic, Web3-native applications. Use this skill IMMEDIATELY and ALWAYS when the user mentions ASI1, ASI:One, Fetch.ai AI API, building with ASI1, integrating ASI:One, asking about ASI1 models, tool calling with ASI1, ASI1 image generation, ASI1 agentic LLM, Agentverse, uagents, Agent Chat Protocol, structured output with ASI1, or OpenAI-compatible wrappers for ASI1. Also trigger when the user says things like "use ASI1 instead of OpenAI", "build an app with ASI:One", "ASI1 API", or references docs.asi1.ai. This skill covers everything needed to build production apps - setup, all models, all API features, tool calling, image gen, agentic orchestration, structured data, session management, streaming, LangChain integration, uagents / Agent Chat Protocol, and TypeScript/Node.js patterns.
4SKILL.mdUpdated May 30, 2026
pinkpixel-dev/asc-metrics
data-ai
VerifiedTrustedCommunity
When the user wants to analyze their own app's actual performance data from App Store Connect — real downloads, revenue, IAP, subscriptions, trials, or country breakdowns synced via Appeeky Connect. Use when the user asks about "my downloads", "my revenue", "how is my app performing", "ASC data", "sales and trends", "my subscription numbers", "App Store Connect metrics", or wants to compare periods or top markets. For third-party app estimates, see app-analytics. For subscription analytics depth, see monetization-strategy.
4SKILL.mdUpdated May 30, 2026