scraperapi/scraperapi-n8n

ScraperAPI in n8n — Workflow Generator

This skill helps you assemble n8n workflows around the official ScraperAPI community node (n8n-nodes-scraperapi-official). It supports two output shapes, picked from the user's phrasing:

| User says … | Produce | |-------------|---------| | "give me the workflow JSON", "export", "I want to import this", "scaffold a .json" | A complete, importable workflow JSON file | | "walk me through it", "show me how to build", "in the n8n editor", "step by step" | A numbered build guide referencing exact resource / operation / parameter names | | Ambiguous | Default to JSON. It's reversible — they can either import it or follow it as a recipe. |

When you produce JSON, write it to a file in the working directory (e.g., scraperapi-workflow.json) and tell the user to import it via Workflows → Import from File in the n8n UI. When you produce a build guide, format the steps for the n8n editor (resource picker → operation picker → fields).

Prerequisites the user must have

1. An n8n instance (Cloud, self-hosted, or desktop) where they can install community nodes. On self-hosted instances, an instance owner must enable community nodes first.
2. The community node installed: Settings → Community Nodes → Install → n8n-nodes-scraperapi-official. After install, "ScraperAPI" appears in the node picker.
3. A ScraperAPI API key from https://dashboard.scraperapi.com/. They create the credential once via Credentials → New → ScraperAPI API and paste the key. The credential is named ScraperAPI API in the UI and scraperApi-Api in the workflow JSON.

If the user is on n8n Cloud and the workflow JSON references this node, importing will succeed only after the community node is installed in their instance. Mention this when handing over JSON.

Three resources — pick the right one

The ScraperAPI node has one operation surface (resource) with three branches. Choose based on the job:

| Job | Resource | Why | |-----|----------|-----| | Scrape an arbitrary URL and get HTML/Markdown/Text/JSON | api | General-purpose proxy. Hand it any URL. | | Get clean JSON from a supported site (Amazon, Google, eBay, Walmart, Redfin) | sde | Pre-parsed structured data. Saves a Set/Code node for HTML parsing and is more reliable across site changes. | | Crawl multiple pages from one starting URL, streaming results to a webhook | crawler | Multi-page jobs that follow links. Returns a jobId; results arrive at your webhook, not as the node's output. |

A common pattern: use a structured endpoint when one exists for the site, fall back to api (with apiRender: true or apiPremium: true) for anything else, and reach for crawler only when one-URL-at-a-time would need many sequential node executions.

Required JSON shape — minimal example

This is the smallest possible workflow that uses ScraperAPI. Everything else in this skill is variations on this shape.

{
  "name": "ScraperAPI minimal example",
  "nodes": [
    {
      "parameters": {},
      "type": "n8n-nodes-base.manualTrigger",
      "typeVersion": 1,
      "position": [240, 300],
      "id": "trigger-1",
      "name": "When clicking 'Test workflow'"
    },
    {
      "parameters": {
        "resource": "api",
        "operation": "apiRequest",
        "apiUrl": "https://example.com",
        "apiOptionalParameters": {
          "apiRender": true,
          "apiCountryCode": "us"
        }
      },
      "type": "n8n-nodes-scraperapi-official.scraperApi",
      "typeVersion": 1,
      "position": [480, 300],
      "id": "scraperapi-1",
      "name": "ScraperAPI",
      "credentials": {
        "scraperApi-Api": {
          "id": "REPLACE_WITH_CREDENTIAL_ID",
          "name": "ScraperAPI API"
        }
      }
    }
  ],
  "connections": {
    "When clicking 'Test workflow'": {
      "main": [
        [{ "node": "ScraperAPI", "type": "main", "index": 0 }]
      ]
    }
  },
  "settings": { "executionOrder": "v1" }
}

Things to get right in every workflow JSON

1. Node type: n8n-nodes-scraperapi-official.scraperApi — exact spelling, case-sensitive. Wrong type = "Unrecognized node type" on import.
2. Credential block: key is scraperApi-Api (not scraperApiApi). Leave id as a placeholder and tell the user to re-select the credential in the imported node. n8n cannot look up credential IDs across instances.
3. typeVersion: 1 for the ScraperAPI node.
4. Connections use node name, not id. If you rename a node, update the connections map too.
5. Position is [x, y] pixels in the editor. Anything is fine on import; n8n re-layouts on demand. Use ~240px spacing horizontally.
6. executionOrder: "v1" under settings — this is current default. Older "none"/"v0" are deprecated and behave differently with merges.

Expressions in node fields — the = prefix

n8n distinguishes a literal string from an expression by a leading = character on the field's value. This applies to every field of every node, including ScraperAPI's apiUrl, sdeAsin, sdeQuery, etc.

"apiUrl": "https://example.com"          // literal — always this URL
"apiUrl": "={{ $json.url }}"             // expression — URL from upstream item
"apiUrl": "{{ $json.url }}"              // ⚠ literal "{{ $json.url }}" string

A few practical patterns that come up constantly:

"sdeAsin":  "={{ $json.asin }}"
"sdeQuery": "={{ $json.body.search_term }}"           // webhook payload
"sdeUrl":   "=https://www.redfin.com/city/{{ $json.city_id }}"

When the value needs to be partly literal and partly dynamic, keep the = only at the start of the string. Everything inside {{ }} is a JS expression — $json, $node['Other'].json.x, JSON.stringify(...), inline arithmetic, etc.

For webhook-triggered workflows, remember user data lives under body (see "Common mistakes" below). The expression is ={{ $json.body.x }}, not ={{ $json.x }} — a single missing body. is the most frequent silent failure in n8n.

Parameter naming — the n8n trap

The node's JSON parameter names are not the same as ScraperAPI's HTTP query parameter names. n8n uses camelCase with a resource prefix; the API uses snake_case. When generating JSON, use the n8n names.

| ScraperAPI HTTP param | n8n field (api resource) | n8n field (crawler resource) | |-----------------------|--------------------------|------------------------------| | url | apiUrl | crawlerStartUrl | | render | apiOptionalParameters.apiRender | crawlerOptionalParameters.crawlerApiParameters.crawlerApiRender | | country_code | apiOptionalParameters.apiCountryCode | crawlerOptionalParameters.crawlerApiParameters.crawlerApiCountryCode | | premium | apiOptionalParameters.apiPremium | crawlerOptionalParameters.crawlerApiParameters.crawlerApiPremium | | ultra_premium | apiOptionalParameters.apiUltraPremium | crawlerOptionalParameters.crawlerApiParameters.crawlerApiUltraPremium | | autoparse | apiOptionalParameters.apiAutoparse | crawlerOptionalParameters.crawlerApiParameters.crawlerApiAutoparse | | output_format | apiOptionalParameters.apiOutputFormat | crawlerOptionalParameters.crawlerApiParameters.crawlerApiOutputFormat | | device (desktop) | apiOptionalParameters.apiDesktopDevice (bool) | crawlerOptionalParameters.crawlerApiParameters.crawlerApiDesktopDevice | | device (mobile) | apiOptionalParameters.apiMobileDevice (bool) | crawlerOptionalParameters.crawlerApiParameters.crawlerApiMobileDevice |

For SDE platforms, see references/node-reference.md — each platform/ operation has its own required field name (e.g., sdeAsin, sdeQuery, sdeProductId, sdeUrl, sdeCategory) and its own optional-parameters collection (e.g., sdeAmazonProductOptions, sdeGoogleSearchOptions).

If you're not sure of an exact field name, read the reference file before generating JSON. Mis-named fields are silently dropped on import.

Output of the ScraperAPI node

Every operation returns one item per input item with this shape:

{
  "resource": "api",
  "response": {
    "body": "...",
    "headers": { },
    "statusCode": 200,
    "statusMessage": "OK"
  }
}

For api and crawler, body is the raw text payload (HTML, Markdown, JSON-as-string depending on apiOutputFormat). For sde, body is the parsed JSON document from ScraperAPI as a string — downstream nodes need to parse it. The simplest pattern is a Code node:

return $input.all().map((item, i) => ({
  json: JSON.parse(item.json.response.body),
  pairedItem: { item: i },
}));

The pairedItem field tells n8n which input item each output belongs to. Skipping it works for linear flows, but the moment the workflow fans out (Loop Over Items, Merge, IF that re-converges), missing pairedItem causes downstream nodes to mismatch items — most often seen as "wrong row paired with wrong scrape result" in batch workflows. Always set it.

After this node, every field from the structured payload is accessible directly as ={{ $json.<field> }} downstream.

Wiring patterns by use case

The user usually has one of these shapes in mind. Don't invent a complex multi-node flow when one of these covers it.

Daily monitor — scheduled scrape → diff → notify

Schedule Trigger → [Sheets/DB read] → Loop Over Items
                                          ↓
                                       ScraperAPI (sde)
                                          ↓
                                       Code (parse body)
                                          ↓
                                       [Sheets/DB write history]
                                          ↓
                                       IF (changed?)
                                          ↓
                                       Email / Slack

When the user's request resembles a price-monitoring or daily-digest workflow (any combination of: scheduled trigger, list of products, ScraperAPI lookup, history table, email-on-change), don't hand-write the JSON from this shape. Fetch the canonical ScraperAPI template and adapt it:

GET https://api.n8n.io/api/templates/workflows/15609

The full importable workflow JSON is at response.workflow.workflow (it has nodes and connections at the top level, ready to drop into n8n's import dialog). The template has 29 nodes — Schedule Trigger, 3 ScraperAPI nodes (Amazon / Walmart / Google Shopping), Data Table for history, Code nodes for diffing, an HTML digest, plus both Gmail and SMTP send paths so the user can pick one.

Adapt rather than reproduce: change the product list, drop retailers the user doesn't need, swap the email path, simplify the digest. Cite the human-readable page so the user can preview before importing:

https://n8n.io/workflows/15609-send-daily-price-drop-digest-emails-for-amazon-walmart-and-google-via-scraperapi/

If the fetch fails (offline, rate-limited, API change), fall back to the shape diagrammed above and the simpler single-ASIN example in references/example-workflows.md.

Webhook enrichment — on-demand lookup

Webhook → ScraperAPI (sde or api) → Code (parse) → Respond to Webhook

Useful for lead enrichment, on-demand product lookups, or as an MCP-style tool exposed to an LLM. Set the Webhook's response to "Using 'Respond to Webhook' node" so you can return the parsed data.

Crawler + webhook callback — multi-page jobs

[Trigger] → ScraperAPI (crawler / Initiate) → ... waits ...
                                                ↓
                              (results arrive at separate webhook workflow)

Webhook (the callbackUrl) → process each page → store / index

The Initiate operation returns a jobId immediately. Pages are POSTed to crawlerCallbackUrl as they're scraped — this must be a different workflow with a Webhook trigger, not the same workflow that initiated the job. Always tell the user to copy the callback workflow's webhook URL into crawlerCallbackUrl before activating.

Batch from a list — CSV/Sheet fan-out

[Read CSV / Sheet] → Loop Over Items (batch size 5–10) → ScraperAPI → Code → Merge → Output

Use Loop Over Items (a.k.a. Split In Batches) with a small batch size to stay under ScraperAPI's concurrency limit. The Free plan allows 5 concurrent requests; paid plans go higher. If the user knows their plan, size the batch accordingly; otherwise default to 5.

AI Agent tool — give an LLM live web access

The ScraperAPI node declares usableAsTool: true, so it can be plugged into an n8n AI Agent node as a callable tool. The agent decides when to invoke it, with what URL/query, based on the user's prompt.

Chat Trigger → AI Agent ─┬─ Chat Model (OpenAI / Anthropic / etc.)
                          ├─ Memory (optional)
                          └─ Tool: ScraperAPI

Wiring details:

1. Add the ScraperAPI node to the canvas. Its operation/parameters act as the tool's signature — the agent fills in dynamic parameters from context. Pick the operation that matches the job you want the agent to be able to do (e.g., sde / googleSearch for "let the agent search the web", or api / apiRequest for "let the agent fetch any URL").
2. For dynamic fields, use ={{ $fromAI('parameter_name', 'description', 'string') }}. $fromAI is the n8n helper that exposes a parameter to the agent's tool schema; the agent picks the value at runtime.
3. Connect the ScraperAPI node into the AI Agent node via the Tool input (not the main input).
4. Give the node a clear, action-oriented name ("Search the web", "Scrape any URL") — the agent uses the node name as part of the tool description it sees.

{
  "resource": "sde",
  "sdePlatform": "google",
  "operation": "googleSearch",
  "sdeQuery": "={{ $fromAI('query', 'The user search query', 'string') }}",
  "sdeGoogleSearchOptions": {
    "tld": "com",
    "outputFormat": "json"
  }
}

The agent receives the parsed JSON body as the tool result and reasons over it. For best results add a Code node between the ScraperAPI tool and the agent to slim the payload — raw SDE responses can be large enough to blow the agent's context budget.

Credit budgeting — warn the user

ScraperAPI charges credits per successful request. Generated workflows that loop or schedule can burn credits silently. When the workflow scrapes more than ~50 URLs per run or runs more than hourly, add a note in your reply estimating the per-run and per-month cost.

| Setting | Credits per request | |---------|---------------------| | api standard | 1 | | api + apiRender: true | 10 | | api + apiPremium: true | 10 | | api + apiPremium: true + apiRender: true | 25 | | api + apiUltraPremium: true | 30 | | sde structured endpoint | 1–10 (per vertical; see public docs) |

Failed requests (non-200) are generally not charged, but retries on transient errors still consume credits if they eventually succeed.

For full and current pricing, link to https://docs.scraperapi.com/ from the response.

Common mistakes — ❌ wrong / ✅ correct

Missing = prefix turns the expression into literal text

❌ "apiUrl": "{{ $json.url }}"            // n8n scrapes the literal string "{{ $json.url }}"
✅ "apiUrl": "={{ $json.url }}"           // resolved to the current item's url field

Webhook data lives under body, not the root

❌ "sdeQuery": "={{ $json.search_term }}"           // undefined
✅ "sdeQuery": "={{ $json.body.search_term }}"      // works

The Webhook node wraps incoming data as { headers, params, query, body }. The user's payload is $json.body. Same applies to a Chat Trigger that receives a JSON body.

Code node returning a parsed object instead of n8n item shape

❌ return JSON.parse($json.response.body);
❌ return { data: parsed };
✅ return [{ json: JSON.parse($json.response.body), pairedItem: { item: 0 } }];

n8n expects an array of { json: {...}, pairedItem?: {...} }. Returning a raw object yields "Cannot read properties of undefined" downstream.

apiPremium and apiUltraPremium set together

❌ { "apiPremium": true, "apiUltraPremium": true }
✅ { "apiUltraPremium": true }                      // pick one; ultra implies premium

They are mutually exclusive in the node UI and have undefined behavior together.

Crawler callback routed back into the same workflow

❌ ScraperAPI (crawlerJobCreate) ───loops to itself───> processing nodes
✅ Workflow A:  Trigger → ScraperAPI (crawlerJobCreate, callbackUrl = "<webhook in Workflow B>")
   Workflow B:  Webhook trigger → process page → store

Crawler results arrive at the callbackUrl as HTTP POSTs from ScraperAPI's infrastructure. Only a workflow with a Webhook trigger can receive them — never the same workflow that initiated the job.

Hardcoded credential ID in exported JSON

❌ "credentials": { "scraperApi-Api": { "id": "abc123", "name": "ScraperAPI API" } }
✅ "credentials": { "scraperApi-Api": { "id": "REPLACE", "name": "ScraperAPI API" } }

Credential IDs are local to one n8n instance. An imported workflow with a real ID from another instance will fail with a "credential not found" error. Use a placeholder and ask the user to re-select the credential once after import.

Setting sdePlatform without a matching operation

❌ { "sdePlatform": "amazon", "operation": "googleSearch" }   // mismatched
✅ { "sdePlatform": "amazon", "operation": "amazonProduct" }

The operation field is platform-scoped. For sdePlatform: "amazon", valid operations are amazonProduct, amazonSearch, amazonOffers. For Amazon Reviews and Prices (available via the underlying API but not yet surfaced as separate node operations), fall back to api + apiAutoparse: true against the Amazon URL.

Skipping the Code-node parse step after sde

The structured response is a JSON string inside response.body. Downstream nodes referencing ={{ $json.pricing }} see undefined until a Code (or Set) node parses the body. See the parse snippet above.

Hardcoded API key in node parameters

The API key belongs in the credential record, never in node parameters or expressions. Putting it in JSON exposes it in workflow exports, version history, and audit logs.

When to read what

| If you need … | Read | |---------------|------| | The exact field name for an SDE operation | references/node-reference.md | | A larger worked example (price monitor, lead enricher) | references/example-workflows.md | | What a valid n8n workflow JSON looks like end-to-end | references/workflow-json-anatomy.md |

After producing a workflow JSON, sanity-check it:

1. node.type is exactly n8n-nodes-scraperapi-official.scraperApi
2. Every node referenced in connections exists in nodes
3. Field names match the prefix rule (api* / sde* / crawler*)
4. The credentials block uses key scraperApi-Api
5. No literal API key is anywhere in the JSON

Related skills and ecosystem

• scraperapi-mcp (sibling skill in this plugin) — use when the user wants Claude itself to call ScraperAPI as a tool, instead of wiring it into an n8n workflow. Different output: live answers vs. reusable JSON.
• n8n-mcp MCP server (https://github.com/czlonkowski/n8n-mcp) — third-party MCP server that exposes n8n's full node catalog, validation, and workflow CRUD to Claude. If the user has it installed, recommend validating the generated JSON via validate_workflow and deploying with n8n_create_workflow instead of having them import manually. The two skills compose: this one teaches the ScraperAPI node's shape; n8n-mcp teaches the surrounding n8n surface.

Documentation

• Node listing: https://n8n.io/integrations/scraperapi/
• Price-drop digest template (page): https://n8n.io/workflows/15609-send-daily-price-drop-digest-emails-for-amazon-walmart-and-google-via-scraperapi/
• Price-drop digest template (API, returns full workflow JSON): https://api.n8n.io/api/templates/workflows/15609
• npm package: https://www.npmjs.com/package/n8n-nodes-scraperapi-official
• ScraperAPI docs: https://docs.scraperapi.com/
• Dashboard: https://dashboard.scraperapi.com/