ratacat/documentation-scraper

Documentation Scraper with slurp-ai

Overview

slurp-ai scrapes documentation websites and compiles them into a single markdown file optimized for AI agent context. It is lightweight, fast, and deterministic - it does NOT use AI to scrape, it is FOR AI consumption.

CRITICAL: Run Outside Sandbox

All commands in this skill MUST be run outside the sandbox. Use dangerouslyDisableSandbox: true for all Bash commands including:

• which slurp (installation check)
• node analyze-sitemap.js (sitemap analysis)
• slurp (scraping)
• File inspection commands (wc, head, cat, etc.)

The sandbox blocks network access and file operations required for web scraping.

Pre-Flight: Check Installation

Before scraping, verify slurp-ai is installed:

which slurp || echo "NOT INSTALLED"

If not installed, ask the user to run:

npm install -g slurp-ai

Requires: Node.js v20+

Do NOT proceed with scraping until slurp-ai is confirmed installed.

Commands

| Command | Purpose | |---------|---------| | slurp <url> | Fetch and compile in one step | | slurp fetch <url> [version] | Download docs to partials only | | slurp compile | Compile partials into single file | | slurp read <package> [version] | Read local documentation |

Output: Creates slurp_compiled/compiled_docs.md from partials in slurp_partials/.

CRITICAL: Analyze Sitemap First

Before running slurp, ALWAYS analyze the sitemap. This reveals the complete site structure and informs your --base-path and --max decisions.

Step 1: Run Sitemap Analysis

Use the included analyze-sitemap.js script:

node analyze-sitemap.js https://docs.example.com

This outputs:

• Total page count (informs --max)
• URLs grouped by section (informs --base-path)
• Suggested slurp commands with appropriate flags
• Sample URLs to understand naming patterns

Step 2: Interpret the Output

Example output:

📊 Total URLs in sitemap: 247

📁 URLs by top-level section:
   /docs                          182 pages
   /api                            45 pages
   /blog                           20 pages

🎯 Suggested --base-path options:
   https://docs.example.com/docs/guides/     (67 pages)
   https://docs.example.com/docs/reference/  (52 pages)
   https://docs.example.com/api/             (45 pages)

💡 Recommended slurp commands:

   # Just "/docs/guides" section (67 pages)
   slurp https://docs.example.com/docs/guides/ --base-path https://docs.example.com/docs/guides/ --max 80

Step 3: Choose Scope Based on Analysis

| Sitemap Shows | Action | |---------------|--------| | < 50 pages total | Scrape entire site: slurp <url> --max 60 | | 50-200 pages | Scope to relevant section with --base-path | | 200+ pages | Must scope down - pick specific subsection | | No sitemap found | Start with --max 30, inspect partials, adjust |

Step 4: Frame the Slurp Command

With sitemap data, you can now set accurate parameters:

# From sitemap: /docs/api has 45 pages
slurp https://docs.example.com/docs/api/intro \
  --base-path https://docs.example.com/docs/api/ \
  --max 55

Key insight: Starting URL is where crawling begins. Base path filters which links get followed. They can differ (useful when base path itself returns 404).

Common Scraping Patterns

Library Documentation (versioned)

# Express.js 4.x docs
slurp https://expressjs.com/en/4x/api.html --base-path https://expressjs.com/en/4x/

# React docs (latest)
slurp https://react.dev/learn --base-path https://react.dev/learn

API Reference Only

slurp https://docs.example.com/api/introduction --base-path https://docs.example.com/api/

Full Documentation Site

slurp https://docs.example.com/

CLI Options

| Flag | Default | Purpose | |------|---------|---------| | --max <n> | 20 | Maximum pages to scrape | | --concurrency <n> | 5 | Parallel page requests | | --headless <bool> | true | Use headless browser | | --base-path <url> | start URL | Filter links to this prefix | | --output <dir> | ./slurp_partials | Output directory for partials | | --retry-count <n> | 3 | Retries for failed requests | | --retry-delay <ms> | 1000 | Delay between retries | | --yes | - | Skip confirmation prompts |

Compile Options

| Flag | Default | Purpose | |------|---------|---------| | --input <dir> | ./slurp_partials | Input directory | | --output <file> | ./slurp_compiled/compiled_docs.md | Output file | | --preserve-metadata | true | Keep metadata blocks | | --remove-navigation | true | Strip nav elements | | --remove-duplicates | true | Eliminate duplicates | | --exclude <json> | - | JSON array of regex patterns to exclude |

When to Disable Headless Mode

Use --headless false for:

• Static HTML documentation sites
• Faster scraping when JS rendering not needed

Default is headless (true) - works for most modern doc sites including SPAs.

Output Structure

slurp_partials/              # Intermediate files
  └── page1.md
  └── page2.md
slurp_compiled/              # Final output
  └── compiled_docs.md       # Compiled result

Quick Reference

# 1. ALWAYS analyze sitemap first
node analyze-sitemap.js https://docs.example.com

# 2. Scrape with informed parameters (from sitemap analysis)
slurp https://docs.example.com/docs/ --base-path https://docs.example.com/docs/ --max 80

# 3. Skip prompts for automation
slurp https://docs.example.com/ --yes

# 4. Check output
cat slurp_compiled/compiled_docs.md | head -100

Common Issues

| Problem | Cause | Solution | |---------|-------|----------| | Wrong --max value | Guessing page count | Run analyze-sitemap.js first | | Too few pages scraped | --max limit (default 20) | Set --max based on sitemap analysis | | Missing content | JS not rendering | Ensure --headless true (default) | | Crawl stuck/slow | Rate limiting | Reduce --concurrency 3 | | Duplicate sections | Similar content | Use --remove-duplicates (default) | | Wrong pages included | Base path too broad | Use sitemap to find correct --base-path | | Prompts blocking automation | Interactive mode | Add --yes flag |

Post-Scrape Usage

The output markdown is designed for AI context injection:

# Check file size (context budget)
wc -c slurp_compiled/compiled_docs.md

# Preview structure
grep "^#" slurp_compiled/compiled_docs.md | head -30

# Use with Claude Code - reference in prompt or via @file

When NOT to Use

• API specs in OpenAPI/Swagger: Use dedicated parsers instead
• GitHub READMEs: Fetch directly via raw.githubusercontent.com
• npm package docs: Often better to read source + README
• Frequently updated docs: Consider caching strategy