sampx/fc-local

fc-local — Local Web Scraping

CLI wrapper for web scraping and crawling.

Prerequisites

Core Commands

| Command | Description | |---------|-------------| | fc-cli | CLI for scrape/crawl operations | | my-fc | Service manager (start/stop/status/health/logs) | | chrome_remote | Chrome CDP manager (for anti-crawling fallback) |

fc-local Service

my-fc status          # Check container health
my-fc health          # Full check (API + scrape test)
my-fc start           # Start Docker services
my-fc logs api 100    # View API logs

CDP Fallback (Anti-crawling)

chrome_remote -b      # Start Chrome with remote debugging
chrome_remote status  # Check CDP status
chrome_remote stop    # Stop Chrome CDP

Verify dependencies:

which fc-cli my-fc chrome_remote agent-browser jq

Command Matrix

| Need | Command | Options | |------|---------|---------| | Single page | scrape <url> | --format, --clean🤖, --prompt🤖 | | Entire site | crawl <url> | --limit, --wait, --clean🤖 | | Multiple URLs | batch <file> | --wait | | Link discovery | map <url> | --limit, --filter | | Web search | search <query> | --limit | | LLMs.txt | llmstxt <path> | --full | | Job status | *-status <id> | --wait |

🤖 = AI feature, high cost. Use only when user explicitly requests.

Core Commands

scrape — Single Page

fc-cli scrape <url> [-o .wopal-space/INBOX/docs/scraped/<name>.md] [--format markdown|html|links] [--clean] [--prompt <text>]

• -o: Output file path (default: .wopal-space/INBOX/docs/scraped/<name>.md)
• --format: Output format (default: markdown)
• --clean🤖: AI removes nav, ads, sidebars
• --prompt🤖: Custom AI processing (implies --clean)

crawl — Website Crawling

fc-cli crawl <url> --limit <n> --wait [-o .wopal-space/INBOX/docs/scraped/<site>] [--clean] [--prompt <text>]

• --limit: Max pages to crawl
• --wait: Wait for completion
• -o: Output directory (default: .wopal-space/INBOX/docs/scraped/<site>)
• --clean🤖: AI content cleaning
• --prompt🤖: Custom AI processing

Output: Directory structure with .md files per page.

batch — Multiple URLs

# Input: one URL per line or JSON array
fc-cli batch urls.txt --wait [-o results.json]

map — Link Discovery

fc-cli map <url> [--limit <n>] [--filter <pattern>]

• --filter: Wildcard pattern (*api*, */docs/v1/*)

llmstxt — Generate LLMs.txt

# From URL
fc-cli llmstxt https://example.com

# From local directory
fc-cli llmstxt ./crawl-output [--full]

Async Job Pattern

Most commands return job ID immediately. Use --wait for sync execution:

# Async
fc-cli crawl https://example.com
# Returns: job_abc123

# Check status
fc-cli crawl-status job_abc123

# Or sync mode
fc-cli crawl https://example.com --wait

Status commands: crawl-status, batch-status

Global Options

| Option | Description | |--------|-------------| | --api-url <url> | Override API URL | | -o, --output <file> | Save to file/directory | | -v, --verbose | Detailed logging |

🚨 AI Features — STRICT RULES

🔴 NEVER use AI options (--clean, --prompt) unless user EXPLICITLY requests them.

| User says | Interpretation | Your action | |-----------|----------------|-------------| | "抓取这个页面" | Plain scrape | fc-cli scrape <url> | | "结构化获取" | Markdown output (already structured) | fc-cli scrape <url> | | "爬取整个网站" | Plain crawl | fc-cli crawl <url> | | "用 AI 清理内容" | Explicit AI request | fc-cli scrape <url> --clean ✅ |

Key principle: scrape already returns structured Markdown. "结构化" ≠ AI extraction.

AI options consume credits and are slower. Default to plain scrape/crawl.

Notes

• Output directory: Save scraped files to .wopal-space/INBOX/docs/scraped/ using -o option
• Speed: Playwright renders 2-5s per page
• Large sites: Test with small --limit first

Fallback: Agent-Browser CDP Mode

When fc-local fails due to anti-crawling (page keeps navigating, 403/500, captcha, empty content), use agent-browser with user's Chrome CDP.

Anti-crawling indicators:

• Playwright log: page is navigating and changing the content
• API returns 403/500 or captcha page
• Content is empty or shows "access denied"

Quick Fallback (Recommended)

# From skill directory:
./scripts/scrape-cdp.sh "<url>" [-o .wopal-space/INBOX/docs/scraped/<name>.md]

The script automatically:

1. Starts chrome_remote if not running
2. Opens URL and waits for load
3. Extracts main content (excludes nav/footer/ads)
4. Saves as Markdown with title and metadata

Manual Fallback

# 1. Start Chrome CDP
chrome_remote -b

# 2. Navigate and extract
agent-browser --cdp 9222 open "<url>"
agent-browser --cdp 9222 wait --load networkidle
agent-browser --cdp 9222 get text body > .wopal-space/INBOX/docs/scraped/<name>.md

# 3. Cleanup
chrome_remote stop

Why this works: chrome_remote uses user's real Chrome profile with cookies/login state, bypassing anti-bot detection.

Troubleshooting

When commands fail or return unexpected results:

# Quick diagnostics
my-fc status              # Container health
my-fc health              # Full API test
my-fc logs api 100        # Recent API logs
my-fc logs playwright-service 50  # Playwright errors

Common issues:

• Empty content → Check playwright logs for anti-crawling errors → Use CDP fallback
• API not responding → my-fc restart
• Timeout → Increase --timeout or check site speed

👉 Full guide: references/TROUBLESHOOTING.md

References

• Full API: references/API_REFERENCE.md
• Examples: references/EXAMPLES.md
• Troubleshooting: references/TROUBLESHOOTING.md