oborchers/source-evaluation

Source Evaluation

Source quality is the primary bottleneck in research agent pipelines. Research on deep research agent trajectories found that over 57% of source errors occur in early retrieval stages, where initial fabrication acts as the primary catalyst for cascading downstream errors (arXiv 2601.22984). A single bad source in the first retrieval round contaminates the entire research trajectory.

Source Credibility Tiers

Every source encountered during research falls into one of six tiers. Always prefer higher-tier sources and cite the tier when reporting findings.

| Tier | Source Type | Examples | Trust Level | |------|-----------|----------|-------------| | T1 — Primary | Peer-reviewed journals, official specs, primary datasets | Nature, Science, IEEE, IETF RFCs, W3C specs | Highest | | T2 — Institutional | Government agencies, established research institutions | NIH, WHO, NIST, ACM Digital Library | High | | T3 — Expert | Named expert blogs, conference proceedings, major tech engineering blogs | Anthropic blog, Google Research, NeurIPS/ICML papers | Moderate-High | | T4 — Quality Editorial | Major publications with editorial review | MIT Technology Review, Ars Technica, The Verge | Moderate | | T5 — Community | Well-moderated forums, high-reputation answers | Stack Overflow (high-score), GitHub discussions | Low-Moderate | | T6 — Unverified | Content farms, SEO-optimized articles, anonymous posts, AI-generated content | Medium listicles, affiliate blogs, uncredited tutorials | Do not cite |

Rule: Never cite T6 sources. Prefer T1-T3 for factual claims. Use T4-T5 for context and community consensus only.

The CRAAP Framework — Automated Signals

Adapted from the CRAAP framework (CSU Chico), five dimensions for evaluating sources:

| Dimension | What to Check | Red Flags | |-----------|--------------|-----------| | Currency | Publication date, last-modified headers | No date visible, information predates major changes in the field | | Relevance | Does it address the specific research question? | Tangential coverage, keyword-stuffed but shallow | | Authority | Who published it? Credentials? | Anonymous author, no institutional affiliation, no citations | | Accuracy | Are claims sourced? Can they be verified? | No inline citations, contradicts known facts, round numbers without source | | Purpose | Is it informing, selling, or persuading? | High ad density, affiliate links, promotional language |

Note: CRAAP evaluates surface features. Use it as an initial filter, not the sole credibility signal (Stanford research found reliance on CRAAP alone makes researchers susceptible to misinformation).

Multi-Provider Search Strategy

Different search providers excel in different domains. Route queries to the appropriate provider:

| Provider | Best For | Limitations | |----------|---------|-------------| | WebSearch (general) | Broad topics, recent events, technical documentation | May surface SEO-optimized content | | arXiv / Semantic Scholar | Academic ML/AI research, preprints | Not peer-reviewed, may be superseded | | PubMed | Medical, biomedical, clinical research | Limited to biomedical domain | | Official documentation | API specs, library usage, framework guides | May lag behind actual behavior | | GitHub | Code examples, implementation patterns, issue discussions | Quality varies widely |

Strategy: Start with domain-appropriate providers. Use general web search to fill gaps. Cross-reference findings across multiple providers when possible.

SEO Spam Detection

Red flags that indicate low-quality, SEO-optimized content:

• Listicle format with no depth ("Top 10 ways to...")
• Keyword stuffing — the search term appears unnaturally often
• No author attribution or author has no verifiable expertise
• High ad-to-content ratio
• Recycled/syndicated content appearing verbatim across multiple domains
• AI-generated markers — generic phrasing, lack of specific examples, overly smooth prose
• Affiliate links embedded throughout

When a source triggers 2+ red flags, discard it and search for a higher-quality alternative.

Artifact Evaluation

Research often involves evaluating non-content artifacts — packages, tools, technologies, standards, organizations. These require different signals than content sources. Every artifact has three signal dimensions:

| Dimension | What It Measures | Key Question | |-----------|-----------------|--------------| | Health | Is it alive and maintained? | When was the last meaningful activity? | | Adoption | Does anyone actually use it? | What are the real usage numbers? | | Authority | Who's behind it and are they credible? | Is this backed by a credible entity? |

Artifact Types and Key Signals

| Artifact Type | Health Signals | Adoption Signals | Authority Signals | |---------------|---------------|-----------------|-------------------| | Software packages | Last commit, release frequency, open issue response time | Downloads (npm weekly, PyPI monthly), dependents count | Maintainer reputation, organizational backing, license | | GitHub repos | Commit frequency, PR merge time, stale issue ratio | Stars, forks, contributor count | Bus factor (>1 critical), corporate sponsor, notable users | | APIs/Services | Uptime history, changelog frequency, deprecation notices | Customer logos, integration count, community size | Company funding, revenue stability, enterprise adoption | | Standards/Specs | Last revision date, errata activity | Implementation count, conformance test suites | Standards body status (draft/proposed/standard), industry backing | | Technologies | Release cadence, roadmap activity, CVE response time | Stack Overflow survey ranking, job postings, TIOBE/RedMonk index | Backing organization, governance model, ecosystem size | | Architectural patterns | Recent case studies, active community discussion | Industry adoption breadth, conference talk frequency | Documented at-scale deployments, known failure case studies | | People/Authors | Recent publication activity | Citation count, h-index, follower count | Institutional affiliation, industry role, peer recognition | | Companies/Orgs | Recent funding, hiring activity, product releases | Revenue, customer count, market share | Investor quality, leadership track record, industry awards | | Communities | Messages per week, new member rate | Member count, active member ratio | Moderation quality, notable members, signal-to-noise ratio | | Datasets/Benchmarks | Last update, known issues addressed | Citation count, leaderboard participation | Creator credentials, methodology transparency, peer review | | Claims/Statistics | Date of study, methodology recency | Citation count, replication status | Funding source, sample size, peer review, original source |

Programmatic Stat Verification

When evaluating artifacts, use APIs for exact stats instead of search snippets:

| Ecosystem | API Command | Returns | |-----------|-------------|---------| | GitHub | gh api repos/{owner}/{name} | Stars, forks, license, language, last update, open issues | | GitHub releases | gh api repos/{owner}/{name}/releases/latest | Latest version tag, release date | | npm | curl api.npmjs.org/downloads/point/last-week/{pkg} | Exact weekly downloads | | PyPI | curl pypistats.org/api/packages/{pkg}/recent | Recent download counts | | crates.io | curl crates.io/api/v1/crates/{crate} | Downloads, version, recent downloads | | RubyGems | curl rubygems.org/api/v1/gems/{gem}.json | Downloads, latest version | | Maven | Search site:mvnrepository.com {artifact} | Usage stats page |

These APIs return ground truth. Search snippets for these stats are unreliable.

Red Flags by Artifact Type

Software packages and repos:

• Last commit >6 months ago with open issues unanswered
• Fewer than 50 stars with no organizational backing
• Single maintainer (bus factor = 1) for critical dependency
• No tests, no CI, no changelog
• License incompatible with intended use

Technologies and standards:

• No major release in 12+ months
• Declining Stack Overflow activity trend
• Abandoned by original backing organization
• No conformance test suite (for standards)

Claims and statistics:

• No original source cited (circular citation)
• Study funded by party with commercial interest in the outcome
• Sample size <100 for quantitative claims
• No methodology description

General rule: When an artifact triggers 2+ red flags, flag it explicitly in the research output. Do not recommend it without noting the risks.

For detailed per-artifact-type evaluation guides and how to check each signal programmatically, consult references/artifact-signals.md.

Retrieval Best Practices

1. Front-load quality — The first retrieval round disproportionately determines research quality due to the saturation bottleneck (agents fixate on early results). Start with high-authority sources.
2. Search iteratively — Refine search queries based on initial results. First search identifies terminology; second search uses domain-specific terms.
3. Diversify sources — Do not rely on a single provider or a single source for any claim. Cross-reference across independent sources.
4. Verify fetched content — After WebFetch, scan the content for authority signals (author credentials, citations, publication venue) before incorporating.
5. Track provenance — Record which source produced which claim. This metadata is essential for citation and for tracing errors.

Retrieval Anti-Patterns

| Anti-Pattern | Problem | Fix | |-------------|---------|-----| | Single-provider dependency | All searches go through one provider | Route by domain; use multiple providers | | First-result trust | Accepting the top search result without evaluation | Evaluate credibility tier before incorporating | | Equal credibility | Treating a blog post the same as a journal paper | Apply tier system; weight higher-tier sources | | Ignoring retrieval failures | Silent fallback when search returns nothing useful | Log the gap; try alternative queries or providers | | Breadth without depth | Fetching 20 URLs but reading none carefully | Fetch fewer sources; read each thoroughly |

Reference Files

For detailed provider comparison, domain-specific source guides, and artifact evaluation:

• references/provider-comparison.md — Detailed comparison of search providers with API specifics, rate limits, and optimal use cases
• references/artifact-signals.md — Per-artifact-type evaluation guides with health/adoption/authority thresholds, how to check each signal, and the quick evaluation checklist