nimbleway/nimble-crawl-reference

nimble crawl — reference

Async bulk crawling — starts a crawl job, returns a crawl_id, then you poll for status and retrieve per-page content via individual task_ids.

Tip: For LLM use, prefer nimble map + nimble extract --format markdown (faster, cleaner). Use crawl when you need raw HTML archives of many pages at once.

Table of Contents

• 1. Create crawl (async)
• 2. Get crawl status
• 3. List crawls
• 4. Cancel crawl
• Polling guidelines
• Retrieving page content
• Crawl vs Map

---

1. Create crawl (async)

Parameters:

| Parameter | CLI flag | Type | Default | Description | | ------------------------- | --------------------------- | -------- | --------- | -------------------------------------------- | | url | --url | string | required | Starting URL | | limit | --limit | int | 5000 | Max pages (1–10,000) — always set this | | name | --name | string | — | Label for tracking | | sitemap | --sitemap | string | include | include | only | skip | | include_path | --include-path | string[] | — | URL path regex to include (repeatable) | | exclude_path | --exclude-path | string[] | — | URL path regex to exclude (repeatable) | | max_discovery_depth | --max-discovery-depth | int | 5 | Max link depth from start (1–20) | | crawl_entire_domain | --crawl-entire-domain | bool | false | Follow sibling/parent paths | | allow_subdomains | --allow-subdomains | bool | false | Follow links to subdomains | | allow_external_links | --allow-external-links | bool | false | Follow links to external domains | | ignore_query_parameters | --ignore-query-parameters | bool | false | Deduplicate query-param variants | | callback | --callback | string | — | Webhook URL for real-time page notifications | | extract_options | --extract-options | object | — | Extraction config applied to each page |

CLI:

nimble crawl run --url "https://docs.example.com" --limit 50 --name "docs-crawl"

Python SDK:

from nimble_python import Nimble
nimble = Nimble(api_key=os.environ["NIMBLE_API_KEY"])

resp = nimble.crawl.run(url="https://docs.example.com", limit=50, name="docs-crawl")
crawl_id = resp.crawl_id

Response fields: crawl_id, status (initial: queued), url, name, crawl_options, created_at

Crawl status values: queued → running → succeeded / failed / canceled

---

2. Get crawl status

Parameters:

| Parameter | CLI flag | Type | Description | | --------- | -------- | ------ | --------------------- | | id | --id | string | crawl_id (required) |

CLI:

nimble crawl status --id "abc-123"

Python SDK:

crawl = nimble.crawl.status(id=crawl_id)
print(crawl.status, crawl.completed, crawl.total)

Response adds: total, pending, completed, failed, tasks[]

| Field | Description | | ----------------- | ----------------------------------------------------- | | tasks[].task_id | Use with nimble tasks results to fetch page content | | tasks[].status | pending / processing / completed / failed |

CRITICAL: Use tasks[].task_id — NOT crawl_id — to fetch page content. Using crawl_id with tasks returns 404.

---

3. List crawls

Parameters:

| Parameter | CLI flag | Type | Description | | --------- | ---------- | ------ | ---------------------------------------------------------------------- | | status | --status | string | Filter: queued | running | completed | failed | canceled | | limit | --limit | int | Results per page | | cursor | --cursor | string | Pagination cursor |

CLI:

nimble crawl list --status running

Python SDK:

result = nimble.crawl.list()
# result.data = list of crawl objects
# result.pagination = { has_next, next_cursor, total }

---

4. Cancel crawl

Parameters:

| Parameter | CLI flag | Type | Description | | --------- | -------- | ------ | --------------------- | | id | --id | string | crawl_id (required) |

CLI:

nimble crawl terminate --id "abc-123"

Python SDK:

nimble.crawl.terminate(id=crawl_id)
# Returns: {"status": "canceled"}

---

Polling guidelines

CRAWL_ID=$(nimble crawl run --url "https://example.com" --limit 100 --name "my-crawl" \
  | python3 -c "import json,sys; print(json.load(sys.stdin)['crawl_id'])")

while true; do
  STATUS=$(nimble crawl status --id "$CRAWL_ID" \
    | python3 -c "import json,sys; print(json.load(sys.stdin).get('status',''))")
  echo "Status: $STATUS"
  [ "$STATUS" = "succeeded" ] || [ "$STATUS" = "failed" ] || [ "$STATUS" = "canceled" ] && break
  sleep 30
done

| Crawl size | Poll interval | | ------------ | ------------- | | < 50 pages | every 15–30s | | 50–500 pages | every 30–60s | | 500+ pages | every 60–120s |

---

Retrieving page content

After crawl succeeded, fetch each completed task — see nimble-tasks reference:

nimble tasks results --task-id "task-456"

---

Crawl vs Map

| | map | crawl | | -------- | ------------------------------------------- | ------------------------- | | Speed | Seconds (sync) | Minutes (async) | | Output | URL list with metadata | Full page content per URL | | Best for | Discover URLs, then selectively extract | Archive all pages at once | | LLM use | ✅ Combine with extract --format markdown | ⚠️ Returns raw HTML |