pim97/scrappey

Scrappey

Overview

Use Scrappey via https://publisher.scrappey.com/api/v1?key=.... Prefer the bundled wrapper in scripts/invoke_scrappey.py so requests are reproducible and the API key stays in SCRAPPEY_API_KEY.

Quick Start

Set the API key in the shell:

$env:SCRAPPEY_API_KEY="your_api_key"

Run a browser-backed GET request:

python .\scripts\invoke_scrappey.py --cmd request.get --url https://example.com

Run a POST request with additional fields:

python .\scripts\invoke_scrappey.py `
  --cmd request.post `
  --url https://httpbin.org/post `
  --data '{"customHeaders":{"content-type":"application/json"},"postData":{"hello":"world"}}'

Run a full payload file for browser actions or autoparse:

python .\scripts\invoke_scrappey.py --payload-file .\payload.json

Workflow

1. Choose the smallest request shape that can succeed: request.get, request.post, browser actions, or autoparse.
2. Start with only cmd and url, then add headers, cookies, referer setup, retries, proxy settings, or actions if the target requires them.
3. Execute through scripts/invoke_scrappey.py instead of embedding the full request logic in every task.
4. Inspect the raw JSON response first, especially fields like status, solution.statusCode, solution.currentUrl, and any returned body fields.
5. Load references/api-cheatsheet.md only when a request needs specific payload fields or examples.

Core Patterns

Browser-backed GET

Use for JS-rendered or dynamically loaded pages, or any target that a plain HTTP client cannot render correctly.

Browser-backed POST

Use request.post with postData. Set customHeaders.content-type when sending JSON.

Referer-aware flows

Use getUrl before a POST request when the target site expects a realistic referer or landing page.

Browser actions

Use browserActions when content appears only after UI interactions such as clicks, typing, waits, or scrolling.

Autoparse

Use autoparse: true with a structure schema when the user wants structured JSON instead of raw HTML.

Guardrails

• Only request content you are authorized to access, and follow each target site's terms of service, robots directives, rate limits, and applicable law.
• Require SCRAPPEY_API_KEY; do not hardcode or commit credentials.
• Keep payloads minimal because retries, proxies, video, and autoparse can increase cost.
• Prefer payload files for large browser action arrays or autoparse schemas.
• Capture the raw response before adjusting multiple settings at once.
• Treat this skill as an HTTP integration. It helps Codex call Scrappey, but it does not install a dedicated MCP server or native Codex tool.

Resources

• scripts/invoke_scrappey.py: CLI wrapper around the Scrappey API.
• references/api-cheatsheet.md: Request shapes and field reminders.
• README.md: User-facing setup and usage instructions.