microsoft/validate-skills

Validating Skills

Verify every executable command in a skill runs successfully on the current OS.

When to Use

• After creating or updating a skill that contains executable commands
• During skill review to catch stale or broken instructions
• When switching OS (e.g. Windows → Linux) to confirm cross-platform commands

Procedure

1. Detect Current OS

Determine which platform commands to extract:

# PowerShell (Windows)
$os = "Windows"

# Bash (Linux / macOS)
OS=$(uname -s)   # "Linux" or "Darwin"

2. Extract Commands

Parse the target skill's SKILL.md and list every shell command for the detected OS:

• Many skills document commands in tables with Windows and Linux / macOS columns. Pick the column matching your OS.
• If a command contains comments like # Windows or # Linux / macOS, only run the one for your OS.
• Placeholder substitution: Replace obvious placeholders (e.g. <path-to-csproj>, <skill-name>) with real values from the repo. If no sensible value exists, skip the command.
• Adapt cross-platform commands: Commands like ls -la should be adapted to PowerShell equivalents (Get-ChildItem) on Windows when no native Windows command is documented.

3. Track Results

Use the SQL tool to create a tracking table:

CREATE TABLE skill_commands (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  skill TEXT NOT NULL,
  command TEXT NOT NULL,
  expected_exit INTEGER DEFAULT 0,
  actual_exit INTEGER,
  status TEXT DEFAULT 'pending',
  notes TEXT
);

4. Run Each Command

For every extracted command:

1. Run it from the repo root
2. Record the exit code
3. Classify the result:
   • Exit 0 → PASS
   • Non-zero + environment issue (missing SDK, no internet) → ENV_ISSUE
   • Non-zero + command/docs wrong → ERROR

5. Safety Rules

CRITICAL: Never run unfiltered integration/acceptance tests. They take hours.

• test.sh --integrationTest or test.cmd -Integration MUST include a --filter or -p flag.
• test.sh -p smoke is acceptable (scoped to smoke tests), but expect it to be slow.

6. Report

After all commands finish, print a summary:

=== Skill Validation Report ===
Skill: <skill-name>
OS: <Linux|Darwin|Windows>
Commands tested: N
PASS: X
ENV_ISSUE: Y (list with reasons)
ERROR: Z (list failed commands with exit codes)

7. Fix or Flag

• ERROR (documentation bug): Update the skill's SKILL.md to fix the command.
• ENV_ISSUE: Add a troubleshooting note to the skill if the environment prerequisite is not already documented.
• PASS: No action needed.

Ordering Tips

• Run restore/build before tests (tests depend on build output)
• Run the cheapest commands first to fail fast
• Batch independent test commands in parallel when possible