giggle-official/tavily-crawl

Tavily Crawl Skill

Traverse entire websites and extract content from multiple pages in one API call. Ideal for ingesting documentation sites, knowledge bases, or any multi-page resource into context.

Authentication

Set TAVILY_API_KEY in ~/.openclaw/.env.local:

TAVILY_API_KEY=tvly-your-api-key-here

When to Use This Skill

• You need to read an entire documentation site (e.g. a library's API docs)
• Building a knowledge base from a website
• Exploring an unknown site structure before targeted extraction
• When tavily-extract would require too many individual URLs

Use tavily-extract when you already have specific URLs. Use this when you need to discover and read multiple pages from a starting URL.

Warning: max_depth ≥ 2 can take minutes and return large amounts of data. Always start with max_depth=1 and limit=20.

API Reference

Endpoint: POST https://api.tavily.com/crawl

Request Fields

| Field | Type | Default | Description | | ------------------- | ------- | ------------ | ------------------------------------------------------- | | url | string | Required | Root URL to begin crawling | | max_depth | integer | 1 | Levels deep to follow links (1–5) | | max_breadth | integer | 20 | Max links to follow per page | | limit | integer | 50 | Total pages cap | | instructions | string | null | Natural language focus guidance | | chunks_per_source | integer | 3 | Relevant chunks per page (1–5, requires instructions) | | extract_depth | string | "basic" | basic or advanced (for JS-rendered sites) | | format | string | "markdown" | markdown or text | | select_paths | array | null | Regex patterns to include (e.g. ["/docs/.*"]) | | exclude_paths | array | null | Regex patterns to exclude (e.g. ["/blog/.*"]) | | allow_external | boolean | true | Follow links to external domains | | timeout | float | 150 | Max wait in seconds (10–150) |

Response Format

{
  "base_url": "https://docs.example.com",
  "results": [
    {
      "url": "https://docs.example.com/getting-started",
      "raw_content": "# Getting Started\n\n..."
    }
  ],
  "response_time": 8.3
}

Examples

Basic crawl (start conservative)

curl -s -X POST https://api.tavily.com/crawl \
  -H "Authorization: Bearer $TAVILY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://docs.example.com",
    "max_depth": 1,
    "limit": 20
  }'

Focused crawl for agentic use (recommended pattern)

Always use instructions + chunks_per_source when feeding results into LLM context — returns only relevant chunks instead of full pages, preventing context overflow:

curl -s -X POST https://api.tavily.com/crawl \
  -H "Authorization: Bearer $TAVILY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://docs.example.com",
    "max_depth": 2,
    "instructions": "Find authentication guides and API reference",
    "chunks_per_source": 3,
    "select_paths": ["/docs/.*", "/api/.*"],
    "exclude_paths": ["/changelog/.*", "/blog/.*"]
  }'

Crawl only specific sections

curl -s -X POST https://api.tavily.com/crawl \
  -H "Authorization: Bearer $TAVILY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "max_depth": 2,
    "select_paths": ["/docs/.*"],
    "exclude_paths": ["/docs/legacy/.*"],
    "limit": 30
  }'

Map API (Discover URLs Without Content)

Use map when you only need to discover what pages exist before deciding which to crawl or extract:

curl -s -X POST https://api.tavily.com/map \
  -H "Authorization: Bearer $TAVILY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://docs.example.com",
    "max_depth": 2,
    "instructions": "Find all API reference and authentication pages"
  }'

Response returns a list of URLs only — much faster than crawl.

Depth vs Performance

| max_depth | Typical Pages | Time | | --------- | ------------- | ------------ | | 1 | 10–50 | Seconds | | 2 | 50–500 | Minutes | | 3+ | 500–5000+ | Many minutes |

Tips

• Always start with max_depth=1 and limit=20, then increase if needed
• Always set limit to prevent runaway crawls
• Use instructions + chunks_per_source for agentic workflows — omit only when saving full page data
• Use select_paths to focus on relevant site sections (docs, API, guides)
• Use map first to understand the site structure, then crawl or extract specific sections
• allow_external: false prevents leaving the target site