ranbot-ai/efficient-web-research

Efficient Web Research Skill

A protocol for accessing web content in the most token-efficient, accurate, and structured way — using the right tool at the right depth, and stopping as soon as the question is answerable.

---

When to Use

• Use this skill when the task matches this description: Protocol for token-efficient web research. Use when accessing URLs, GitHub repos, or running search queries. Prevents full-page fetching waste.

Core Principle

Fetch the minimum needed to answer. Skim before you dive. Stop when you can answer.

Every unnecessary fetch wastes tokens and adds noise. This skill enforces a layered approach where you escalate fetch depth only when shallower layers fail.

---

Step 1 — Classify the Input

Before fetching anything, identify what kind of input you received:

| Input Type | Example | Go To | |---|---|---| | GitHub repo URL | github.com/user/repo | GitHub Protocol | | Specific page URL | docs.python.org/3/library/os | URL Protocol | | Topic / query (no URL) | "how does RAFT consensus work" | Search Protocol | | Multiple URLs | List of links | Multi-URL Protocol | | PDF / file link | .pdf, .txt, .md URL | File Protocol |

---

GitHub Protocol

Use when input is a GitHub URL (repo, file, PR, issue, etc.)

Step 1 — Parse the URL

github.com/{owner}/{repo}                → Repo root
github.com/{owner}/{repo}/tree/{branch}  → Directory
github.com/{owner}/{repo}/blob/{branch}/{path} → Single file
github.com/{owner}/{repo}/issues/{n}     → Issue
github.com/{owner}/{repo}/pull/{n}       → Pull request

Step 2 — Use GitHub API (preferred over scraping)

Always prefer the GitHub API. It returns clean JSON — no HTML parsing needed.

# Repo metadata (name, description, language, stars, topics)
GET https://api.github.com/repos/{owner}/{repo}

# File tree (see what files exist — very cheap)
GET https://api.github.com/repos/{owner}/{repo}/git/trees/{ref}?recursive=1

# Single file content (base64 encoded)
GET https://api.github.com/repos/{owner}/{repo}/contents/{path}?ref={ref}

# README only (usually enough to understand the repo)
GET https://api.github.com/repos/{owner}/{repo}/readme

Step 3 — Layered Fetch for Repos

Layer 1 (always do first):
  → Fetch repo metadata + README only
  → Can you answer the user's question now? YES → STOP. NO → continue.

Layer 2 (only if needed):
  → Fetch file tree to understand structure
  → Identify the 1-3 most relevant files based on the question
  → Can you answer now? YES → STOP. NO → continue.

Layer 3 (last resort):
  → Fetch specific relevant files only (never fetch all files)
  → Prioritize: main entry point, config files, key modules

Token Rules for GitHub

• README alone answers ~70% of "what does this repo do" questions — always try it first
• Never fetch more than 3 files in a single research turn
• If a file exceeds ~300 lines, read only the top (imports + class/function signatures)
• Decode base64 content from API before passing to context

---

URL Protocol

Use when the user gives a specific non-GitHub URL (docs, articles, blogs, etc.)

Step 1 — Assess the URL type

| Site type | Likely works with | Notes | |---|---|---| | Static docs / MDN / ReadTheDocs | read_url_content | Fast, clean, cheap | | News articles / blogs | read_url_content | Usually fine | | SPAs / React/Next.js apps | browser_subagent | JS-rendered | | Auth-gated pages | browser_subagent | Needs login | | Raw GitHub files (raw.githubusercontent) | read_url_content | Direct text |

Step 2 — Layered Fetch

Layer 1 — Skim
  → Fetch the URL with read_url_content
  → Read only headings (H1, H2, H3) and first paragraph
  → Does this page contain what the user needs? NO → try a different URL or search. YES → continue.

Layer 2 — Targeted Extract
  → If the page has anchor links (e.g. /docs/page#section), fetch with the anchor
  → Extract only the relevant section (200–500 tokens max)
  → Can you answer? YES → STOP.

Layer 3 — Full Fetch
  → Fetch full page, strip boilerplate (nav, footer, ads, cookie banners, sidebars)
  → Cap at 2000 tokens. Summarize before passing to answer.

Layer 4 — Browser Subagent (last resort only)
  → Use ONLY if read_url_content returns empty, garbled, or JS-placeholder content
  → Instruct subagent: "Navigate to [URL], wait for content to load, extract [specific section]"
  → Do NOT use browser_subagent for static pages — it's expensive

What to Strip from Fetched Pages

Always remove before using fetched content:

• Navigation menus and breadcrumbs
• Cookie banners and GDPR notices
• "Related articles" / "You might also like" blocks
• Footer content (copyright, links)
• Social share buttons
• Ads and sponsored content

Extract and keep:

• Main article / documentation body
• Code blocks
• Tables with data
• Numbered steps or procedures

---

Search Pr