jmagly/devkit-test

Test AIWG Package

Run comprehensive tests on an AIWG package to verify structure, functionality, and deployment readiness.

Process

1. Locate and Identify Package

Resolve the package path from $ARGUMENTS:

• If relative path, resolve from current directory
• If package name, look in addons/ and frameworks/

Identify package type from manifest.json:

• "type": "addon" → Addon tests
• "type": "framework" → Framework tests
• "type": "extension" → Extension tests

2. Run Structure Tests

Manifest Validation:

• [ ] manifest.json exists and is valid JSON
• [ ] Required fields present: id, type, name, version, description
• [ ] Entry paths exist and match actual directories
• [ ] Component arrays match actual files

Directory Structure:

• [ ] Expected directories exist (agents/, commands/, etc.)
• [ ] No orphaned files outside expected structure
• [ ] README.md exists and is non-empty

Component Files:

• [ ] All agents listed in manifest exist as .md files
• [ ] All commands listed in manifest exist as .md files
• [ ] All skills listed in manifest have SKILL.md
• [ ] All templates listed in manifest exist

3. Run Content Tests

Agent Validation: For each agent file:

• [ ] Valid YAML frontmatter with name, description, model, tools
• [ ] Model is valid (sonnet, opus, haiku)
• [ ] Tools list contains valid tool names
• [ ] Content sections present (Domain Expertise, Responsibilities, etc.)

Command Validation: For each command file:

• [ ] Valid YAML frontmatter with name, description
• [ ] Args defined with name and description
• [ ] Process section present
• [ ] No broken internal references

Skill Validation: For each skill:

• [ ] SKILL.md has valid frontmatter
• [ ] Trigger phrases defined
• [ ] Execution process documented
• [ ] References directory exists (if referenced)

4. Run Type-Specific Tests

Addon Tests:

• [ ] No requires field (addons are standalone)
• [ ] Can be deployed independently

Extension Tests:

• [ ] requires field present and valid
• [ ] Parent framework exists
• [ ] Located in correct extensions/ subdirectory

Framework Tests:

• [ ] Phases defined in manifest
• [ ] Flow documents exist for each phase
• [ ] actors-and-templates.md exists
• [ ] config/models.json exists
• [ ] metrics/tracking-catalog.md exists

5. Run Deployment Test

Simulate deployment to temporary directory:

# Create temp test directory
mkdir -p /tmp/aiwg-test-$$

# Attempt deployment
aiwg -deploy-agents --source <package-path> --target /tmp/aiwg-test-$$ --dry-run

# Check expected output structure

6. Auto-Fix (if --fix)

When --fix is specified, attempt to repair:

Fixable Issues:

• Missing directories → Create them
• Manifest component arrays out of sync → Update from files
• Missing README.md → Generate from manifest
• Invalid JSON formatting → Reformat

Non-Fixable Issues (report only):

• Missing required fields in manifest
• Invalid agent/command content
• Missing parent framework (extensions)

7. Generate Report

Summary Format:

Package Test: <package-id>
Type: <addon|extension|framework>
─────────────────────────────────

Structure Tests: <pass/fail count>
Content Tests: <pass/fail count>
Type Tests: <pass/fail count>
Deployment Test: <pass/fail>

Overall: <PASS|FAIL>

Detailed Format (--verbose):

Package Test: <package-id>
─────────────────────────────────

[Structure Tests]
  ✓ manifest.json exists
  ✓ manifest.json is valid JSON
  ✓ Required fields present
  ✓ agents/ directory exists
  ✗ commands/ directory missing
  ...

[Content Tests]
  ✓ agent-one.md: valid frontmatter
  ✓ agent-one.md: valid model (sonnet)
  ✗ agent-two.md: missing tools field
  ...

[Type Tests]
  ✓ No requires field (addon)
  ...

[Deployment Test]
  ✓ Dry-run deployment successful

[Issues Found]
  ERROR: commands/ directory missing
  WARNING: agent-two.md missing tools field

[Auto-Fixed] (if --fix)
  ✓ Created commands/ directory

Exit Codes

• 0: All tests passed
• 1: Tests failed (fixable issues remain)
• 2: Tests failed (non-fixable issues)

Examples

# Test addon
/devkit-test aiwg-utils

# Test with verbose output
/devkit-test sdlc-complete --verbose

# Test and auto-fix
/devkit-test my-addon --fix --verbose

# Test extension
/devkit-test sdlc-complete/extensions/hipaa

References

• @$AIWG_ROOT/agentic/code/addons/aiwg-dev/README.md — aiwg-dev addon overview
• @$AIWG_ROOT/docs/extensions/overview.md — Extension system architecture and package types
• @$AIWG_ROOT/docs/cli-reference.md — CLI reference including validate-metadata command
• @$AIWG_ROOT/agentic/code/addons/aiwg-utils/rules/research-before-decision.md — Verification-before-action pattern
• @$AIWG_ROOT/agentic/code/frameworks/sdlc-complete/README.md — SDLC framework as example test target