acaprino/web-search-techniques

Web Search Techniques

Shared knowledge base for research:quick-searcher and research:deep-researcher. Scope: web-only. Covers query formulation, source authority, tool usage, bot-block fallback, and anti-loop rules.

Query Formulation

Extract core concepts from the question before querying:

• Identify synonyms and domain terminology (e.g. "authentication": auth, login, signin, session, token, jwt, oauth, credentials)
• Account for abbreviations and full forms
• Add the year ("2026") when the query has temporal dependency
• Add "official" or "documentation" to push toward authoritative sources
• Quote exact phrases for precise matches
• Use site: to restrict to known-good domains (e.g. site:developer.mozilla.org)

Start broad, narrow progressively. Overly specific first queries miss adjacent information. Each refinement round incorporates terms surfaced in prior results.

Source Authority Ranking

Rank every source before citing:

1. Official documentation sites and API references -- highest authority
2. RFC and specification documents -- canonical for standards
3. GitHub issues, discussions, and source code -- authoritative for specific libraries
4. Peer-reviewed or community-validated content -- Stack Overflow with high votes, maintainers' blogs
5. General blog posts and tutorials -- use only when nothing better exists
6. Deprioritize -- SEO content farms, AI-generated summaries, scraped aggregators

Currency checks:

• Last modified date on the page
• Version numbers cited vs latest release
• Deprecation warnings

WebSearch Techniques

Query operators (standard search-engine conventions, usually respected by WebSearch):

• site: -- restrict to a domain
• "exact phrase" -- match the phrase verbatim
• Year token -- add "2026" for recency
• "official" or "documentation" -- bias toward authoritative
• Version numbers when relevant (e.g. react 19)

WebFetch Guidance

• Prefer documentation pages and API references over blog posts
• Evaluate fetched content -- low-authority source means discard and re-search
• Large pages may be truncated -- target specific sections (anchor URLs) when possible
• Track the accessed URL with date for citation

webfetch.py Fallback

When WebFetch returns a bot-block (403, 429, Cloudflare challenge) or thin content (under ~200 chars of useful text), fall back to the plugin's stealth fetcher:

python3 ${CLAUDE_PLUGIN_ROOT}/scripts/webfetch.py <url>

Behavior:

• Impersonates Chrome TLS fingerprint via curl_cffi
• Returns clean extracted text on stdout
• Exits 0 on success, 1 on timeout or error
• On failure, proceed without the result (do not retry in a loop)

Invocation options:

• --timeout SECONDS (default: 30)
• --max-chars CHARS (truncate output)
• --raw (return raw HTML instead of extracted text)

Requires Bash tool in the agent's tools: frontmatter.

Anti-Loop Rules

• Never repeat the exact same query or search parameters
• If a search returns nothing, change terminology, broaden the regex, or switch tool/target
• Maximum 2 failed attempts per sub-topic before pivoting or escalating
• After 2 failed attempts on the same angle, document the gap and proceed with what you have