a-organvm/research-synthesis-workflow

Research Synthesis Workflow

This skill provides a systematic methodology for conducting research, synthesizing findings from multiple sources, and producing actionable knowledge artifacts.

Core Competencies

• Source Evaluation: Assessing credibility, relevance, and bias
• Information Extraction: Systematic note-taking and annotation
• Synthesis Methods: Thematic analysis, meta-analysis, framework building
• Knowledge Artifacts: Reports, literature reviews, decision frameworks

Research Workflow Overview

┌──────────────────────────────────────────────────────────────┐
│                    Research Synthesis Workflow                │
├──────────────────────────────────────────────────────────────┤
│                                                              │
│  1. SCOPE         2. GATHER         3. EXTRACT              │
│  ┌─────────┐      ┌─────────┐      ┌─────────┐              │
│  │ Define  │─────▶│ Find    │─────▶│ Capture │              │
│  │ Question│      │ Sources │      │ Insights│              │
│  └─────────┘      └─────────┘      └─────────┘              │
│       │                                  │                   │
│       │          5. PRODUCE         4. SYNTHESIZE           │
│       │          ┌─────────┐      ┌─────────┐              │
│       └─────────▶│ Create  │◀─────│ Connect │              │
│                  │ Artifact│      │ Themes  │              │
│                  └─────────┘      └─────────┘              │
│                                                              │
└──────────────────────────────────────────────────────────────┘

Phase 1: Scope Definition

Research Question Framework

Transform vague topics into answerable questions:

| Type | Pattern | Example | |------|---------|---------| | Exploratory | What is X? How does X work? | What is vector search? | | Comparative | How does X compare to Y? | PostgreSQL vs. Neo4j for graphs? | | Evaluative | Is X effective for Y? | Is RAG effective for technical docs? | | Causal | What causes X? What are effects of X? | What causes LLM hallucinations? | | Prescriptive | How should we implement X? | How to design a RAG pipeline? |

Scope Boundaries

Define explicitly:

• In scope: Topics to cover
• Out of scope: Adjacent topics to exclude
• Depth: Survey (broad) vs. deep-dive (narrow)
• Time bounds: Cut-off dates for sources
• Source types: Academic, industry, primary data

Example Scope Document

## Research Scope: Vector Database Selection

### Research Question
Which vector database best fits our production RAG system
requiring <50ms latency at 10M+ vectors?

### In Scope
- Pinecone, Weaviate, Milvus, Qdrant, pgvector
- Latency benchmarks at scale
- Cost analysis (cloud vs self-hosted)
- Operational complexity

### Out of Scope
- General-purpose databases with vector extensions
- Sub-million vector use cases
- Academic/research-only systems

### Success Criteria
Recommendation with supporting evidence for 2-3 top candidates

Phase 2: Source Gathering

Source Quality Assessment

Evaluate each source on:

| Criterion | High Quality | Low Quality | |-----------|--------------|-------------| | Authority | Expert author, peer-reviewed | Anonymous, no credentials | | Currency | Recent, updated | Outdated, no dates | | Accuracy | Citations, verifiable | Unsupported claims | | Purpose | Inform, educate | Sell, persuade | | Coverage | Comprehensive | Superficial |

Source Types and Uses

Primary Sources (original)
├── Research papers
├── Official documentation
├── Benchmark data
└── Expert interviews

Secondary Sources (analysis)
├── Review articles
├── Technical blogs
├── Industry reports
└── Book chapters

Tertiary Sources (summaries)
├── Wikipedia
├── Textbooks
└── Encyclopedias

Search Strategies

Keyword expansion:

• Start: "vector database performance"
• Expand: "approximate nearest neighbor", "HNSW benchmark", "embedding search latency"

Citation chaining:

• Forward: Who cites this paper?
• Backward: What does this paper cite?

Author tracking:

• Find key researchers, follow their work

Source Documentation

For each source, capture:

## Source: [Title]
- **URL/DOI**:
- **Author(s)**:
- **Date**:
- **Type**: [paper/blog/docs/report]
- **Quality Score**: [1-5]
- **Relevance**: [high/medium/low]
- **Key Topics**:
- **Notes**:

Phase 3: Information Extraction

Structured Note-Taking

Use consistent templates for extraction:

## Claim: [Specific assertion]
- **Source**: [reference]
- **Evidence**: [supporting data/reasoning]
- **Strength**: [strong/moderate/weak]
- **My Assessment**: [agree/disagree/uncertain]
- **Related Claims**: [links to other notes]

Evidence Classification

| Type | Description | Weight | |------|-------------|--------| | Empirical | Measured data, experiments | High | | Analytical | Logical derivation | Medium-High | | Anecdotal | Case studies, examples | Medium | | Expert Opinion | Authority statements | Medium | | Theoretical | Model predictions | Medium-Low |

Contradiction Tracking

When sources disagree:

## Conflict: [Topic]

### Position A: [Claim]
- Sources: [list]
- Evidence: [summary]

### Position B: [Claim]
- Sources: [list]
- Evidence: [summary]

### Analysis
- Methodological differences:
- Context differences:
- Possible resolution:
- My conclusion:

Phase 4: Synthesis

Thematic Analysis

1. Code individual insights with tags
2. Cluster related codes into themes
3. Review themes for coherence
4. Define each theme clearly
5. Relate themes to research question

Codes                    Themes                 Findings
├─ fast queries     ─┐
├─ low latency      ─┼── Performance      ─┬── Theme 1: Performance
├─ high throughput  ─┘                     │   varies significantly
├─ managed service  ─┐                     │   by workload type
├─ self-hosted      ─┼── Deployment      ─┼── Theme 2: Cloud vs
├─ kubernetes       ─┘                     │   self-hosted tradeoff
├─ pricing tiers    ─┐                     │
├─ compute costs    ─┼── Economics       ─┴── Theme 3: Total cost
├─ hidden costs     ─┘                         drives final choice

Framework Building

Create decision frameworks from synthesis:

## Vector Database Selection Framework

### Decision Tree
1. Scale requirement?
   - <1M vectors → pgvector (simplicity)
   - 1M-100M vectors → Continue to 2
   - >100M vectors → Milvus/Weaviate (distributed)

2. Operational capacity?
   - Limited DevOps → Pinecone (managed)
   - Strong DevOps → Continue to 3

3. Cost sensitivity?
   - Budget constrained → Qdrant (open source)
   - Budget flexible → Evaluate all options

### Comparison Matrix
| Criterion      | Weight | Pinecone | Milvus | Qdrant |
|----------------|--------|----------|--------|--------|
| Latency        | 30%    | 4        | 5      | 4      |
| Scalability    | 25%    | 5        | 5      | 4      |
| Operations     | 20%    | 5        | 3      | 4      |
| Cost           | 15%    | 2        | 4      | 5      |
| Features       | 10%    | 4        | 5      | 4      |
| **Weighted**   |        | **4.0**  | **4.4**| **4.2**|

Phase 5: Knowledge Artifact Production

Artifact Types

| Format | Purpose | Audience | |--------|---------|----------| | Executive Summary | Quick decision support | Leadership | | Technical Report | Detailed analysis | Engineers | | Literature Review | Academic synthesis | Researchers | | Decision Framework | Structured evaluation | Decision makers | | Reference Guide | Quick lookup | Practitioners |

Structure Templates

Executive Summary (1-2 pages):

1. Context and question
2. Key findings (3-5 bullets)
3. Recommendation
4. Risks and considerations

Technical Report (5-20 pages):

1. Executive summary
2. Background and scope
3. Methodology
4. Findings by theme
5. Analysis and discussion
6. Recommendations
7. Appendices (data, sources)

Quality Checklist

Before finalizing:

• [ ] Research question answered?
• [ ] All claims supported by evidence?
• [ ] Contradictions addressed?
• [ ] Limitations acknowledged?
• [ ] Actionable recommendations?
• [ ] Sources properly cited?
• [ ] Appropriate for audience?

Best Practices

Avoiding Bias

• Seek disconfirming evidence actively
• Include multiple perspectives
• Note your priors and update them
• Separate observation from interpretation
• Document methodology for transparency

Managing Scope Creep

• Return to research question frequently
• Park interesting tangents in "Future Research"
• Time-box each phase
• Define "good enough" criteria upfront

Iteration

Research is rarely linear:

• New sources may require scope adjustment
• Synthesis may reveal gaps requiring more gathering
• Artifacts may need multiple drafts

References

• references/evaluation-rubrics.md - Source quality scoring guides
• references/synthesis-methods.md - Detailed synthesis techniques
• references/artifact-templates.md - Document templates and examples