davila7/brightdata-local-search

Bright Data Local Search

Run powerful web searches locally using Bright Data's SERP API. This skill sets up the unfancy-search pipeline — a local search engine with query expansion, multi-engine SERP retrieval, RRF reranking, deduplication, and domain clustering.

Important: This skill uses the LOCAL version only. Do not use the hosted endpoint.

Prerequisites

• Bright Data account with SERP API access
• Anthropic API key (for query expansion, optional)
• Docker (recommended) or Node.js 18+ with pnpm

Setup

Step 1: Clone and Configure

git clone https://github.com/yaronbeen/unfancy-search.git
cd unfancy-search
cp .env.example .env

Step 2: Set Environment Variables

Edit .env with your credentials:

BRIGHT_DATA_API_TOKEN=your_brightdata_token
BRIGHT_DATA_SERP_ZONE=serp_api1
ANTHROPIC_API_KEY=your_anthropic_key   # Optional: enables AI query expansion

Get your Bright Data token from: https://brightdata.com (SERP API section)

Step 3: Start the Local Server

Docker (recommended):

docker compose up -d
# Server runs at http://localhost:3000

Node.js:

pnpm install
pnpm dev
# Server runs at http://localhost:3000

API Endpoints

All requests go to http://localhost:3000:

| Endpoint | Method | Description | |----------|--------|-------------| | /api/search | POST | Start a search job | | /api/search-status/{jobId} | GET | Poll for results | | /api/baseline | POST | Trigger baseline collection | | /api/baseline-status/{id} | GET | Poll baseline progress |

Running a Search

Step 1: Submit Search

curl -X POST http://localhost:3000/api/search \
  -H "Content-Type: application/json" \
  -d '{"query": "your search term"}'

The response returns a jobId.

Step 2: Poll for Results

curl http://localhost:3000/api/search-status/{jobId}

Poll every 3 seconds until status is "done".

Search Parameters

| Parameter | Type | Default | Description | |-----------|------|---------|-------------| | query | string | required | Search query | | expand | boolean | false | Enable AI query expansion via Claude | | research | boolean | false | Research mode (12 sub-queries for max coverage) | | engines | string[] | all | SERP engines to use | | geo | string | — | Geographic region filter | | count | number | 10 | Max results (up to 10) | | includeDomains | string[] | — | Only include results from these domains | | excludeDomains | string[] | — | Exclude results from these domains |

Search Modes

• Basic (expand: false): Single query, fastest, no AI cost
• Expanded (expand: true): Claude Haiku generates 3 sub-queries for broader coverage
• Research (research: true): 12 sub-queries for maximum coverage

Usage Examples

Basic Search from an Agent

# Start search
JOB_ID=$(curl -s -X POST http://localhost:3000/api/search \
  -H "Content-Type: application/json" \
  -d '{"query": "best practices for API rate limiting"}' | jq -r '.jobId')

# Poll until done
while true; do
  RESULT=$(curl -s http://localhost:3000/api/search-status/$JOB_ID)
  STATUS=$(echo $RESULT | jq -r '.status')
  if [ "$STATUS" = "done" ]; then
    echo $RESULT | jq '.results'
    break
  fi
  sleep 3
done

Research Mode with Domain Filtering

curl -X POST http://localhost:3000/api/search \
  -H "Content-Type: application/json" \
  -d '{
    "query": "kubernetes scaling strategies",
    "research": true,
    "excludeDomains": ["pinterest.com", "quora.com"]
  }'

Adding Search to an Existing Agent

To give your Claude Code agent search capabilities:

1. Ensure the local server is running (docker compose up -d in the unfancy-search directory)
2. Your agent can use curl or fetch to query http://localhost:3000/api/search
3. Parse the ranked results to ground responses with real web data

Response Format

Results include:

• Ranked URLs with RRF scores
• Domain clustering (grouped by source)
• Cost transparency (per-search expense breakdown)
• Raw and unique result counts
• Search duration

Troubleshooting

| Issue | Solution | |-------|----------| | Server won't start | Verify Docker is running or Node.js 18+ installed | | No results returned | Check BRIGHT_DATA_API_TOKEN is valid and SERP API zone is active | | Query expansion not working | Verify ANTHROPIC_API_KEY is set and valid | | Slow responses | Disable expand mode for faster single-query searches | | Port 3000 in use | Stop other services or modify the port in docker-compose.yml |