scraperapi/scraperapi-datapipeline

ScraperAPI DataPipeline

DataPipeline is a managed scraping product. You define a project (what to scrape, how often, where to send results), and ScraperAPI runs it on your schedule without you managing proxies, retries, or infrastructure.

When NOT to use DataPipeline

• One-off scrapes of a known URL list → use the Async API — faster, cheaper, no project setup.
• Exploring a site without known URLs → use the Crawler.
• Need results in real-time within your code → Async API is programmable; DataPipeline is scheduled.
• Free plan, need recurring execution → recurring schedules require a paid plan.

Use DataPipeline when: scraping runs on a fixed schedule, the input list is large (up to 100,000 items), results should flow to a webhook automatically, or you want email notifications on job completion.

Base URL and Auth

Base URL: https://datapipeline.scraperapi.com/api
Auth:     ?api_key=YOUR_KEY  (query parameter on every request)

Project Types

Set projectType in the create request to choose what to scrape:

| Type | Input | |------|-------| | urls | Raw HTML from any URL | | urls_with_js | Same but with JavaScript rendering | | google_search | Search queries | | google_news | Search queries | | google_jobs | Search queries | | google_shopping | Search queries | | google_maps | Search queries | | amazon_product | ASINs | | amazon_search | Search queries | | amazon_offers | ASINs | | walmart_product | Product IDs | | walmart_search | Search queries | | walmart_category | Category IDs | | walmart_reviews | Product IDs | | ebay_product | 12-digit product IDs | | ebay_search | Search queries | | redfin_listing_for_sale | Listing URLs | | redfin_listing_for_rent | Listing URLs | | redfin_listing_search | Search result URLs | | redfin_agent_details | Agent profile URLs |

Creating a Project

import os, requests

API_KEY = os.environ["SCRAPERAPI_API_KEY"]
BASE    = "https://datapipeline.scraperapi.com/api"

project = requests.post(
    f"{BASE}/projects",
    params={"api_key": API_KEY},
    json={
        "name":               "Weekly Amazon price monitor",
        "projectType":        "amazon_product",
        "schedulingEnabled":  True,
        "scrapingInterval":   "weekly",
        "scheduledAt":        "now",
        "projectInput": {
            "type": "list",
            "list": ["B09V3KXJPB", "B08N5WRWNW"]   # ASINs
        },
        "apiParams": {
            "country_code": "us"
        },
        "webhookOutput": {
            "url":             "https://yourapp.com/pipeline-results",
            "webhookEncoding": "multipart_form_data_encoding"
        },
        "notificationConfig": {
            "notifyOnSuccess": "with_every_run",
            "notifyOnFailure": "with_every_run"
        }
    }
).json()

print(f"Project created: id={project['id']}")

Create request fields

| Field | Required | Description | |-------|----------|-------------| | name | No | Human-readable project name | | projectType | Yes | What to scrape (see table above) | | schedulingEnabled | No | true to enable recurring schedule | | scrapingInterval | Yes (if scheduled) | See scheduling options below | | scheduledAt | No | "now" to run immediately on create | | projectInput | Yes | Input data (see input methods below) | | apiParams | No | Standard ScraperAPI parameters | | webhookOutput | No | Webhook delivery config | | notificationConfig | No | Email notification settings |

Input Methods

Direct list (up to 500 items)

{
  "projectInput": {
    "type": "list",
    "list": ["query one", "query two", "B09V3KXJPB"]
  }
}

CSV file (up to 100,000 items)

Upload a CSV with one URL/query/ASIN per line — no header rows, no commas. Do this through the dashboard when creating a project; the API accepts list inputs only.

Webhook input (dynamic polling)

{
  "projectInput": {
    "type": "webhook",
    "webhookUrl": "https://yourapp.com/input-items"
  }
}

ScraperAPI polls your webhook URL for the item list when the job starts. One item per line; no commas. Useful for dynamically generated lists (e.g., new ASINs added since the last run).

Scheduling Options

| scrapingInterval | Description | |-------------------|-------------| | "once" | Run a single job immediately | | "hourly" | Every hour | | "daily" | Once per day | | "weekly" | Once per week | | "monthly" | Once per month | | "cron" | Custom cron expression (use cron field instead of interval) |

Recurring schedules (hourly, daily, weekly, monthly, cron) require a paid plan.

Set "scheduledAt": "now" to trigger the first run immediately when the project is created.

Output / Delivery

Webhook delivery

Results are POSTed to your webhook URL as they complete. The webhookEncoding field controls the format:

{
  "webhookOutput": {
    "url":             "https://yourapp.com/results",
    "webhookEncoding": "multipart_form_data_encoding"
  }
}

Dashboard download

Omit webhookOutput and results are saved for download in the DataPipeline dashboard. Results are retained for 30 days then automatically deleted.

Output formats by project type:

• urls / urls_with_js → HTML wrapped in JSONL
• Structured types (Amazon, Google, Walmart, eBay, Redfin) → JSON or CSV

Managing Projects

# List all projects
projects = requests.get(f"{BASE}/projects", params={"api_key": API_KEY}).json()

# Get a single project
project = requests.get(f"{BASE}/projects/525", params={"api_key": API_KEY}).json()

# Update (partial update — only include fields to change)
requests.patch(
    f"{BASE}/projects/525",
    params={"api_key": API_KEY},
    json={
        "scrapingInterval": "daily",
        "apiParams":        {"premium": True},
        "notificationConfig": {"notifyOnSuccess": "never"}
    }
)

# Delete / archive (irreversible without support)
requests.delete(f"{BASE}/projects/525", params={"api_key": API_KEY})

Updatable fields: scrapingInterval, scheduledAt, outputFormat, apiParams, notificationConfig.

Managing Jobs

# List jobs for a project
jobs = requests.get(
    f"{BASE}/projects/525/jobs",
    params={"api_key": API_KEY}
).json()

# Cancel a running job
requests.delete(
    f"{BASE}/projects/525/jobs/{job_id}",
    params={"api_key": API_KEY}
)
# Running requests within the job finish first; final status becomes "Cancelled"

A new job can only start if no other job for that project is currently running.

Notification Config

{
  "notificationConfig": {
    "notifyOnSuccess": "with_every_run",
    "notifyOnFailure": "with_every_run"
  }
}

Options for both fields: "never", "with_every_run", "daily", "weekly".

apiParams Reference

All standard ScraperAPI parameters are supported inside apiParams:

| Parameter | Purpose | |-----------|---------| | country_code | Geotarget (e.g. "us", "gb") | | render | JavaScript rendering | | premium | Premium residential proxies | | ultra_premium | Ultra-premium proxies (mutually exclusive with premium) | | device_type | "desktop" or "mobile" | | output_format | "text" or "markdown" for LLM pipelines | | autoparse | Structured JSON extraction for supported sites | | keep_headers | Forward custom headers | | follow_redirect | Control redirect handling | | wait_for_selector | Wait for CSS selector (requires render: true) | | screenshot | Capture screenshot (auto-enables rendering) | | retry_404 | Retry 404 responses |

Credit Costs

DataPipeline uses the same underlying credit rates as the Standard API. Cost is the sum of all requests in a job run. Preview the estimated cost before launching a project from the dashboard.

Only successful 200 and 404 responses are charged; failed requests are not.

Limits

| Limit | Value | |-------|-------| | Max input items | 100,000 per job | | Direct list input | 500 items | | Data retention | 30 days | | Free plan concurrency | 5 connections | | Free plan scheduling | One-time runs only |

Documentation

• DataPipeline overview
• Dashboard — manage projects
• API reference