rickoslyder/claim-extraction

Claim Extraction Skill

Extract all substantive claims from AI research content. A claim is any assertion that:

• States something as true about AI capabilities, limitations, or progress
• Predicts future developments
• Hints at unreleased work
• Expresses a positioned opinion on the field's direction
• Critiques others' claims or work

Extraction Schema

For each claim, extract:

1. claimText

The claim in clear, standalone form. Paraphrase if needed for clarity.

2. claimType

• fact: Assertion about current state ("GPT-4 can do X")
• prediction: Forward-looking ("By 2026, we'll have...")
• hint: Implies unreleased work ("We've been seeing interesting results with...")
• opinion: Positioned take ("I think scaling is/isn't sufficient")
• critique: Challenges others ("Marcus is wrong because...")
• question: Genuine uncertainty expressed ("I'm not sure if...")

3. topic

Primary topic category:

• scaling: Scaling laws, compute, training efficiency
• reasoning: LLM reasoning, chain-of-thought, planning
• agents: AI agents, tool use, autonomy
• safety: AI safety, alignment, control
• interpretability: Mechanistic interpretability
• multimodal: Vision, audio, video models
• rlhf: RLHF, preference learning, Constitutional AI
• benchmarks: Evals, benchmarks, capability measurement
• infrastructure: Training infra, chips, hardware
• policy: AI policy, regulation, governance
• general: General AI commentary

4. stance

• bullish: Optimistic about AI progress/capabilities
• bearish: Skeptical/pessimistic about AI progress
• neutral: Balanced or factual without clear stance

5. bullishness

Float from 0.0 (maximally bearish) to 1.0 (maximally bullish)

6. confidence

How confident does the author seem? (0.0-1.0)

• Hedging language: "might", "could", "I think", "possibly" → lower
• Certainty language: "will", "definitely", "it's clear that" → higher

7. timeframe (for predictions)

• near-term: < 1 year
• medium-term: 1-3 years
• long-term: 3-10 years
• unspecified: No clear timeframe
• null: Not a prediction

8. evidenceProvided

• strong: Cites data, papers, or detailed reasoning
• moderate: Some reasoning but not rigorous
• weak: Assertion without support
• appeal-to-authority: "Trust me, I work on this"

9. quoteworthiness

Is this claim notable enough to quote in a digest? (0.0-1.0)

Output Format

Return JSON:

{
  "claims": [
    {
      "claimText": "The claim in clear form",
      "claimType": "prediction",
      "topic": "reasoning",
      "stance": "bullish",
      "bullishness": 0.8,
      "confidence": 0.7,
      "timeframe": "medium-term",
      "evidenceProvided": "moderate",
      "quoteworthiness": 0.6,
      "relatedTo": ["o1", "chain-of-thought"],
      "originalQuote": "Brief relevant quote if notable"
    }
  ]
}

Guidelines

• Extract MULTIPLE claims from a single piece of content if present
• Don't over-extract - only substantive, meaningful claims
• A tweet saying "Interesting paper" is NOT a claim
• Look for IMPLICIT claims ("We've made a lot of progress" implies capability gains)
• Pay attention to WHO is speaking - lab researchers hinting at their own work is high signal
• Critics often make claims by contradiction ("X is wrong, therefore Y")

Author Context Matters

Consider the author's affiliation when assessing:

• Lab researchers (Anthropic, OpenAI, DeepMind): May hint at unreleased work
• Critics (Marcus, Chollet, Mitchell): Often make claims through critique
• Independent (Simon Willison, Jim Fan): Provide practitioner perspectives