RonanCodes/apify

Apify

Run any Apify actor (a hosted scraper / automation) from your own backend and pull its results into your app. Apify actors take a JSON input, run on Apify's infrastructure, and write rows to a dataset. Your job is three calls: start the actor with input, wait for it to finish, read the dataset.

This skill teaches the two run patterns (sync for short jobs, async-poll for long ones), dataset pagination, cost-awareness, and a real worked example against easyapi/funda-nl-scraper. Everything here uses the plain REST API over fetch / curl so it works identically in Node, Next.js route handlers, and Cloudflare Workers, with no SDK dependency. (The apify-client npm package wraps the same endpoints if you prefer; the raw API is shown so the connector has zero runtime deps.)

Glossary

• Actor — a hosted program on Apify. Identified by an actorId (e.g. 69aVxdpQm6bIIJyNb) or a URL-safe slug username~actor-name (e.g. easyapi~funda-nl-scraper). In a path the / in username/name is written as ~.
• Run — one execution of an actor. Has an id, a status (READY, RUNNING, SUCCEEDED, FAILED, TIMED-OUT, ABORTED), and a defaultDatasetId.
• Dataset — the table of result rows the run produced. Read it by datasetId, paginated with offset / limit.
• PPE (pay-per-event) — newer pricing where you are charged per result row (and a tiny per-start fee). Funda is PPE: ~$0.00499/result + $0.00005/start.
• PPR (pay-per-result) — older flat per-1000-results pricing (~$4.99/1000 ≈ same $0.00499/result).

Auth

Every call needs your API token. Read it from the environment, never hardcode it.

• Env var: APIFY_TOKEN (looks like apify_api_...). Stored in ~/.claude/.env locally; set as a Worker/Vercel secret in prod.
• Pass it as the token query param or as a header Authorization: Bearer $APIFY_TOKEN. Header is preferred server-side so the token never lands in a logged URL.

# header form (preferred)
curl -s -H "Authorization: Bearer $APIFY_TOKEN" \
  "https://api.apify.com/v2/acts/easyapi~funda-nl-scraper"

The two run patterns

flowchart TD
  A["Your backend<br/>(input JSON)"] --> B{"Run length?"}
  B -->|"short, < ~5 min"| C["POST run-sync-get-dataset-items<br/>blocks, returns rows directly"]
  B -->|"long / many results"| D["POST runs<br/>returns run id immediately"]
  D --> E["poll GET runs/{id}<br/>until status SUCCEEDED"]
  E --> F["GET datasets/{datasetId}/items<br/>paginate offset+limit"]
  C --> G["Store rows<br/>(dedup by external id)"]
  F --> G

Rule of thumb: if the actor finishes in a few minutes and the result set is small (a few hundred rows), use sync. Anything that can run long, return thousands of rows, or risk an HTTP timeout (Workers cap a fetch subrequest; Vercel functions have a wall-clock limit), use async + poll + paginate.

Pattern 1 — sync (short runs)

One call. Blocks until the run finishes, then returns the dataset items as the response body. Good for "scrape 30 listings and hand them back".

curl -s -X POST \
  "https://api.apify.com/v2/acts/easyapi~funda-nl-scraper/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"searchUrls":["https://www.funda.nl/en/zoeken/koop?selected_area=%5B%22amsterdam%22%5D"],"maxItems":2}'
# -> returns a JSON array of result objects

Caveats: the connection is held open for the whole run. Apify keeps it alive but proxies / serverless platforms may cut a request that runs longer than their timeout. Don't use sync for runs you expect to take more than a couple of minutes.

Pattern 2 — async + poll + fetch (long runs)

# 1. start the run (returns immediately with a run id)
RUN=$(curl -s -X POST \
  "https://api.apify.com/v2/acts/easyapi~funda-nl-scraper/runs" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"searchUrls":["..."],"maxItems":500}')
RUN_ID=$(echo "$RUN" | python3 -c "import sys,json;print(json.load(sys.stdin)['data']['id'])")
DATASET_ID=$(echo "$RUN" | python3 -c "import sys,json;print(json.load(sys.stdin)['data']['defaultDatasetId'])")

# 2. poll until SUCCEEDED (or use the ?waitForFinish=60 param to long-poll up to 60s per call)
curl -s -H "Authorization: Bearer $APIFY_TOKEN" \
  "https://api.apify.com/v2/actor-runs/$RUN_ID?waitForFinish=60" \
  | python3 -c "import sys,json;print(json.load(sys.stdin)['data']['status'])"

# 3. fetch dataset items, paginated
curl -s -H "Authorization: Bearer $APIFY_TOKEN" \
  "https://api.apify.com/v2/datasets/$DATASET_ID/items?clean=true&offset=0&limit=1000"

?waitForFinish=N (max 60) makes the GET block up to N seconds waiting for a terminal status, so a poll loop is just "call with waitForFinish=60 until status is terminal" rather than a tight sleep loop.

Dataset pagination

/datasets/{id}/items returns rows. Page with offset and limit (max limit is 1000 per call); loop until you get fewer rows than limit. Useful params:

• clean=true — drop empty/hidden internal fields.
• format=json (default) — also supports csv, xlsx, jsonl.
• fields=id,price,address — project only the columns you store (smaller payloads).
• The response header X-Apify-Pagination-Total gives the total row count if you want a progress bar.

Copy-pasteable server-side helper

Drop this into a Node / Next.js route handler or a Worker. No dependencies, just fetch. It does sync for small jobs and falls back to async + poll + paginate for large ones.

// apify.ts — zero-dependency Apify client (Node 18+, Next.js, Cloudflare Workers)
const APIFY_BASE = "https://api.apify.com/v2";

type RunActorOpts = {
  /** actor slug "username~name" or actorId */
  actor: string;
  /** actor input JSON */
  input: Record<string, unknown>;
  /** APIFY_TOKEN */
  token: string;
  /** if true (default) and maxItems is small, use the blocking sync endpoint */
  sync?: boolean;
};

const auth = (token: string) => ({ Authorization: `Bearer ${token}` });

/** Short run: one blocking call, returns rows directly. */
export async function runActorSync<T = unknown>(o: RunActorOpts): Promise<T[]> {
  const res = await fetch(
    `${APIFY_BASE}/acts/${o.actor}/run-sync-get-dataset-items?clean=true`,
    {
      method: "POST",
      headers: { ...auth(o.token), "Content-Type": "application/json" },
      body: JSON.stringify(o.input),
    },
  );
  if (!res.ok) throw new Error(`Apify sync run failed: ${res.status} ${await res.text()}`);
  return (await res.json()) as T[];
}

/** Long run: start, poll to completion, then page the whole dataset. */
export async function runActorAsync<T = unknown>(o: RunActorOpts): Promise<T[]> {
  // 1. start
  const startRes = await fetch(`${APIFY_BASE}/acts/${o.actor}/runs`, {
    method: "POST",
    headers: { ...auth(o.token), "Content-Type": "application/json" },
    body: JSON.stringify(o.input),
  });
  if (!startRes.ok) throw new Error(`Apify start failed: ${startRes.status} ${await startRes.text()}`);
  const { data: run } = (await startRes.json()) as {
    data: { id: string; defaultDatasetId: string; status: string };
  };

  // 2. poll (long-poll 60s per call) until terminal
  let status = run.status;
  while (status === "READY" || status === "RUNNING") {
    const r = await fetch(
      `${APIFY_BASE}/actor-runs/${run.id}?waitForFinish=60`,
      { headers: auth(o.token) },
    );
    status = ((await r.json()) as { data: { status: string } }).data.status;
  }
  if (status !== "SUCCEEDED") throw new Error(`Apify run ended ${status}`);

  // 3. paginate the dataset
  const out: T[] = [];
  const limit = 1000;
  for (let offset = 0; ; offset += limit) {
    const r = await fetch(
      `${APIFY_BASE}/datasets/${run.defaultDatasetId}/items?clean=true&offset=${offset}&limit=${limit}`,
      { headers: auth(o.token) },
    );
    const page = (await r.json()) as T[];
    out.push(...page);
    if (page.length < limit) break;
  }
  return out;
}

/** Convenience: pick sync vs async from expected size. */
export async function runActor<T = unknown>(o: RunActorOpts): Promise<T[]> {
  const maxItems = Number((o.input as { maxItems?: number }).maxItems ?? 0);
  return o.sync !== false && maxItems > 0 && maxItems <= 100
    ? runActorSync<T>(o)
    : runActorAsync<T>(o);
}

Usage:

import { runActor } from "./apify";

const listings = await runActor<FundaListing>({
  actor: "easyapi~funda-nl-scraper",
  token: process.env.APIFY_TOKEN!, // Worker: env.APIFY_TOKEN
  input: {
    searchUrls: ["https://www.funda.nl/en/zoeken/koop?selected_area=%5B%22amsterdam%22%5D"],
    maxItems: 50,
  },
});

Cost-awareness

Check pricing before a big run. GET /v2/acts/{actor} returns pricingInfos with the current model.

• Funda actor (easyapi~funda-nl-scraper) is pay-per-event: $0.00499 per result row + $0.00005 per actor start. So maxItems: 1000 ≈ $5.00. A 2-item probe is ~$0.01.
• maxItems is your cost cap. Always set it. The actor stops at that count.
• Don't re-run a full scrape to get fresh rows. Do a delta: small daily run, dedup against what you already stored (see below).
• A run that fails part-way still charges for the rows it produced before failing.

Storing results + daily delta pattern

Persist rows keyed by a stable external id so re-runs are idempotent.

flowchart LR
  CRON["daily cron"] --> RUN["run actor<br/>(small maxItems)"]
  RUN --> ROWS["result rows"]
  ROWS --> UP["upsert by external id<br/>(insert new, update changed)"]
  UP --> DB[("your DB")]
  ROWS --> SEEN["mark which ids seen<br/>this run"]
  SEEN --> STALE["optionally flag<br/>not-seen as withdrawn"]

• Dedup key: use the actor's stable id field (for Funda that's the integer id). Upsert (INSERT ... ON CONFLICT (external_id) DO UPDATE) so price/status changes refresh in place.
• Daily cron: run with a modest maxItems against the same searchUrls; new listings appear, existing ones update. The portal sorts newest-first so a small cap catches the day's new stock.
• Withdrawn detection (optional): track last_seen_at per row; rows not seen across several consecutive runs for an area can be flagged sold/withdrawn. Don't hard-delete (you lose history).

Worked example: easyapi/funda-nl-scraper (Dutch property listings)

Real input + output shape, captured from a live run on 2026-06-28 (actor build 0.0.16).

Input schema (exact)

The actor id is 69aVxdpQm6bIIJyNb, slug easyapi~funda-nl-scraper. Input properties:

| Property | Type | Required | Default | Notes | |---|---|---|---|---| | searchUrls | string[] | yes | — | Plain Funda search-result URLs as strings, NOT {url} objects. e.g. https://www.funda.nl/en/zoeken/koop?selected_area=%5B%22amsterdam%22%5D. koop = buy, huur = rent. | | maxItems | integer | no | 30 | 1–10000. Your cost cap. | | proxyConfiguration | object | no | { "useApifyProxy": false } | Usually leave default; set { "useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"] } if you hit blocks. |

Minimal input:

{ "searchUrls": ["https://www.funda.nl/en/zoeken/koop?selected_area=%5B%22amsterdam%22%5D"], "maxItems": 50 }

Watch the key name: it is searchUrls (camelCase, plural, plain strings). It is not startUrls and not an array of { "url": "..." } objects.

Output shape (one full result item, abridged photo list)

{
  "id": 8063595,
  "object_type": "apartment",
  "construction_type": "resale",
  "zoning": "residential",
  "status": "none",
  "type": "single",
  "energy_label": "A",
  "number_of_bedrooms": 2,
  "number_of_rooms": 3,
  "offering_type": ["buy"],
  "publish_date": "2026-06-26T18:00:09.6895978+02:00",
  "object_detail_page_relative_url": "/detail/koop/amsterdam/appartement-rinus-michelslaan-243/44418958/",
  "floor_area": [62],
  "floor_area_range": { "gte": 62, "lte": 62 },
  "price": {
    "selling_price": [450000],
    "selling_price_range": { "gte": 450000, "lte": 450000 },
    "selling_price_type": "regular",
    "selling_price_condition": "kosten_koper"
  },
  "address": {
    "country": "NL",
    "province": "Noord-Holland",
    "municipality": "Amsterdam",
    "city": "Amsterdam",
    "wijk": "De Kolenkit",
    "neighbourhood": "Laan van Spartaan",
    "street_name": "Rinus Michelslaan",
    "house_number": "243",
    "postal_code": "1061MB",
    "is_bag_address": true,
    "identifiers": ["nl", "amsterdam", "1061mb", "1061", "..."]
  },
  "photo_image_id": [
    "tiara-media/0e929a8b-a4c1-43cc-b338-854918b539a1/4a3dd278-0440-44bc-8820-d0c5342574a0",
    "tiara-media/0e929a8b-a4c1-43cc-b338-854918b539a1/770d4423-8212-435a-b85f-916cbb2ccfe5"
  ],
  "available_media_types": ["floor_plan", "photo_360", "video"],
  "agent": [
    { "id": 24770, "name": "MAKELAAR BERT", "association": "NVM", "relative_url": "/makelaar/24770-makelaar-bert/", "is_primary": true, "office_id": "f3c83f82-...", "logo_id": 161000413 }
  ],
  "agent_personal_contact": [
    { "full_name": "Cornelie Witteveen", "office_name": "MAKELAAR BERT", "image_url": "https://cloud.funda.nl/valentina_media/212/167/901.jpg", "office_association": "NVM" }
  ],
  "product_entitlement": ["...", "..."]
}

Field mapping (CRM fields → exact JSON paths)

| CRM field | JSON path | Type | Notes | |---|---|---|---| | external id / funda id | id | integer | Stable dedup key. e.g. 8063595. | | address (street + number) | address.street_name + address.house_number | string | Build display: "Rinus Michelslaan 243". | | asking price (number) | price.selling_price[0] | number | It's an array; take [0]. e.g. 450000. Also price.selling_price_range.gte/lte. | | price condition | price.selling_price_condition | string | e.g. kosten_koper (buyer pays costs) vs vrij_op_naam. | | photos (image ids) | photo_image_id[] | string[] | NOT full URLs — they're CDN ids like tiara-media/<album>/<photo>. Build a URL (see below). | | living area m2 | floor_area[0] | number | Array; take [0]. e.g. 62. Also floor_area_range.gte/lte. | | bedrooms | number_of_bedrooms | integer | e.g. 2. | | rooms | number_of_rooms | integer | Total rooms (≥ bedrooms). e.g. 3. | | energy label | energy_label | string | e.g. "A". May be empty string if unknown. | | neighbourhood | address.neighbourhood | string | Finer than wijk. Also address.wijk (district). | | postcode | address.postal_code | string | e.g. "1061MB". | | city | address.city | string | e.g. "Amsterdam". Also address.municipality, address.province. | | offering type (buy/sell) | offering_type[0] | string | Array; "buy" or "rent". | | listing / publish date | publish_date | ISO 8601 string | e.g. "2026-06-26T18:00:09.689+02:00". Parse with new Date(...). | | funda detail URL | object_detail_page_relative_url | string (relative) | Prefix with host: https://www.funda.nl + value. | | listing agent | agent[0].name / agent[0].association | string | e.g. "MAKELAAR BERT", "NVM". | | object type | object_type | string | e.g. "apartment", "house". |

Two gotchas worth a helper:

// 1. detail URL is relative
const detailUrl = `https://www.funda.nl${item.object_detail_page_relative_url}`;

// 2. photos are CDN ids, not URLs. Funda serves them off cloud.funda.nl.
//    Build a sized image URL from the id (1080px wide shown):
const photoUrls = item.photo_image_id.map(
  (id) => `https://cloud.funda.nl/valentina_media/${id}/groot.jpg`,
);
// If a path 404s, inspect a live funda detail page's <img src> in the
// network tab and adjust the size segment (e.g. "1080x720.jpg"); the id
// itself is stable, only the size/format suffix convention may shift.

The floor_area, selling_price, and offering_type fields are arrays even though they hold a single value — always index [0]. energy_label and status can be empty strings.

TypeScript type for the result

type FundaListing = {
  id: number;
  object_type: string;
  construction_type: string;
  energy_label: string;
  number_of_bedrooms: number;
  number_of_rooms: number;
  offering_type: string[];
  publish_date: string;
  object_detail_page_relative_url: string;
  floor_area: number[];
  floor_area_range: { gte: number; lte: number };
  price: {
    selling_price: number[];
    selling_price_range: { gte: number; lte: number };
    selling_price_type: string;
    selling_price_condition: string;
  };
  address: {
    country: string;
    province: string;
    municipality: string;
    city: string;
    wijk: string;
    neighbourhood: string;
    street_name: string;
    house_number: string;
    postal_code: string;
  };
  photo_image_id: string[];
  available_media_types: string[];
  agent: { id: number; name: string; association: string; relative_url: string }[];
};

ToS caveat (read before shipping)

Scraping a portal like Funda is fine for internal tooling, a demo, or a CRM you operate privately. It is not a licensed data feed. For anything customer-facing or commercial in the Dutch market, move to an official feed: the NVM / brainbay data (the same NVM association you see in agent[].association) or a licensed MLS-style provider. Treat the Apify path as the fast way to prototype and validate the connector shape, then swap the data source behind the same internal model before going to production. Keep scrape volume modest and respect the portal's robots.txt and rate limits.

Further reading

• Apify API reference — runs: https://docs.apify.com/api/v2/act-runs-post
• Run-sync-get-dataset-items: https://docs.apify.com/api/v2/act-run-sync-get-dataset-items-post
• Dataset items (pagination): https://docs.apify.com/api/v2/dataset-items-get
• Funda actor page: https://apify.com/easyapi/funda-nl-scraper
• brainbay / NVM (official NL property data): https://www.brainbay.nl/

Open questions

• Funda photo URL suffix convention (groot.jpg vs <w>x<h>.jpg) is inferred from the public CDN, not documented by the actor. Confirm against a live detail page's network tab before relying on a specific size in production.
• product_entitlement[] (28 values in the sample) is undocumented; appears to be Funda's internal feature flags. Not needed for a CRM.
• Rent (huur) listings were not probed; price.selling_price likely becomes a rent field. Probe a huur search URL with maxItems: 2 before wiring rent.