robbyczgw-cla/web-search-plus

Web Search Plus

Stop choosing search providers. Let the skill do it for you.

This skill now connects you to 11 search providers and adds a companion extraction flow for pulling content from URLs. Broad web query? → Brave or Serper. Research question? → Tavily or Exa. Need citations and grounding? → Linkup. Want scrape-ready content? → Firecrawl. Prefer privacy? → SearXNG. Need low-cost Google SERP with prepaid credits? → SerpBase (explicit/fallback-only).

---

✨ What Makes This Different?

• Just search — no need to think about which provider to use
• Smart routing — query analysis picks the best provider automatically
• 11 providers, 1 interface — general web, research, semantic discovery, direct answers, privacy-first, prepaid-credits, and extraction-capable providers together
• URL extraction included — pull markdown/HTML content with fallback across five providers
• Works with just 1 credential — start with any single provider, add more later
• Free/self-hosted options available — SearXNG can run at $0 API cost

---

🚀 Quick Start

# Interactive setup (recommended for first run)
python3 scripts/setup.py

# Or manually
cp .env.example .env
python3 scripts/search.py -q "latest OpenClaw release"
python3 scripts/extract.py --url https://example.com

The wizard explains providers, collects keys, and sets defaults.

---

🔑 Providers

Search providers

• Serper — shopping, prices, local, and general Google-style results; fast broad fallback
• Brave — independent web index and generic current-web queries; strong non-Google complement
• Tavily — research, explanations, and synthesis; strong research routing
• Querit — multilingual and international updates; good for cross-language recency
• Linkup — source-grounded/citation-heavy search; evidence-first queries
• Exa — semantic discovery, similar sites, and deep research; supports deep + deep-reasoning
• Firecrawl — search with scrape-ready metadata; also strong extraction provider
• Perplexity — direct answers with citations; via PERPLEXITY_API_KEY or KILOCODE_API_KEY
• You.com — current-web / RAG-friendly snippets; also supports extraction
• SearXNG — private/self-hosted search; no API key, just instance URL
• SerpBase — low-cost Google SERP with prepaid credits; explicit/fallback-only (opt-in via --provider serpbase or add to provider_priority in config.json)

Extraction providers

scripts/extract.py auto-falls back across:

1. Firecrawl
2. Linkup
3. Tavily
4. Exa
5. You.com

---

🧠 Routing at a Glance

Default priority (SerpBase excluded by design — opt-in only):

tavily → linkup → querit → exa → firecrawl → perplexity → brave → serper → you → searxng

Examples:

python3 scripts/search.py -q "weather in Vienna today"
# generic current-web intent → Brave or Serper

python3 scripts/search.py -q "find credible sources for AI tutoring outcomes"
# citation/evidence intent → Linkup

python3 scripts/search.py -q "latest AI policy updates in Germany"
# multilingual + recency → Querit or Tavily

python3 scripts/search.py -p exa --exa-depth deep -q "LLM scaling laws research"
python3 scripts/search.py -p firecrawl -q "YC startups web scraping"
python3 scripts/search.py -p serpbase -q "best laptop 2026"   # explicit SerpBase

Debug routing:

python3 scripts/search.py --explain-routing -q "your query"

---

📖 Extraction Examples

python3 scripts/extract.py --url https://example.com
python3 scripts/extract.py --url https://docs.linkup.so --provider linkup
python3 scripts/extract.py --url https://example.com --url https://example.org --include-images
python3 scripts/extract.py --url https://example.com --format html --include-raw-html

---

⚙️ Configuration Notes

• .env.example documents supported env vars
• config.example.json includes provider priority and provider-specific defaults
• config.json is your local runtime config
• SearXNG still supports explicit URL config and docker-aware auto-detection
• SerpBase is explicit/fallback-only by default; to include in auto-routing, append "serpbase" to auto_routing.provider_priority in config.json

---

🔒 Security

SearXNG SSRF protection:

• Enforces http / https only
• Blocks common cloud metadata endpoints
• Blocks private/internal IP resolution unless SEARXNG_ALLOW_PRIVATE=1
• Uses operator-controlled config/env only for the instance URL

---

✅ Verification

python3 -m unittest discover -s tests -p 'test_*.py'
python3 scripts/search.py --explain-routing -q "find credible sources for climate change impacts"
python3 scripts/extract.py --url https://example.com --provider auto --compact