wanshuiyin/research-review

Research Review via External Reviewer Backend (ultra reasoning)

🔒 Do not wrap this skill in /loop, /schedule, or CronCreate. It is verdict-bearing — it produces a cross-model review verdict, multi-round with reviewer thread continuity. An external timer re-fires the verdict on wall-clock time and breaks the reviewer's round-to-round memory: zero new signal, full token cost. Schedule the external wait that precedes it (work ready → then review once), not the verdict. See shared-references/external-cadence.md.

Get a multi-round critical review of research work from the selected external reviewer backend with maximum reasoning depth.

Constants

• REVIEWER_MODEL = gpt-5.6-sol — Default model for the Codex backend, reasoning effort ultra (deep-audit tier). Must be an OpenAI model (e.g., gpt-5.6-sol, gpt-5.5, o3). Manual backend uses whatever model the user chooses.
• REVIEWER_BACKEND = codex — Default: Codex MCP (ultra). Override with — reviewer: oracle-pro for Oracle MCP, or — reviewer: manual for Manual Review MCP. If manual-review MCP is unavailable, stop and print the install command; do not fall back to Codex. See shared-references/reviewer-routing.md.

Reviewer Calling Convention

When calling the reviewer, branch on REVIEWER_BACKEND:

If REVIEWER_BACKEND = codex: Use mcp__codex__codex for new review threads. Use mcp__codex__codex-reply for follow-up rounds (reuse threadId).

If REVIEWER_BACKEND = manual: Use mcp__manual_review__review for new review threads with: prompt: [exact same prompt that would go to Codex] config: {"model_reasoning_effort": "xhigh"} Save the returned threadId. Use mcp__manual_review__review_reply for follow-up rounds with: threadId: [saved manual-review threadId] prompt: [follow-up prompt] config: {"model_reasoning_effort": "xhigh"}

Content fidelity: the manual reviewer should see the same substantive review brief Codex would read. If the manual UI supports file upload / attachment, reuse the same brief file; otherwise paste the brief contents inline because remote web UIs cannot read your local filesystem paths. Review tracing applies equally to both backends.

Context: $ARGUMENTS

Prerequisites

• Codex MCP Server configured in Claude Code:

  claude mcp add codex -s user -- codex mcp-server
  

• This gives Claude Code access to mcp__codex__codex and mcp__codex__codex-reply tools

Workflow

Step 1: Gather Research Context

Before calling the external reviewer, compile a comprehensive briefing:

1. Read project narrative documents (e.g., STORY.md, README.md, paper drafts)
2. Read any memory/notes files for key findings and experiment history
3. Identify: core claims, methodology, key results, known weaknesses

Step 2: Initial Review (Round 1)

Send a detailed prompt with ultra reasoning, using the selected backend. For the codex backend, keep the MCP payload short: write the full briefing to RESEARCH_REVIEW_REQUEST.md, then point Codex at that file.

For codex backend:

mcp__codex__codex:
  model: gpt-5.6-sol
  config: {"model_reasoning_effort": "ultra"}
  prompt: |
    Read the review brief at <absolute path to RESEARCH_REVIEW_REQUEST.md>.
    Executor notes are not evidence beyond the files they cite, so verify the
    referenced artifacts before judging.
    Please act as a senior ML reviewer (NeurIPS/ICML level). Start from the
    assumption that the work is broken somewhere — your job is to find where.
    Be adversarial. Trust nothing the author tells you — verify everything
    yourself. Identify:
    1. Logical gaps or unjustified claims
    2. Missing experiments that would strengthen the story
    3. Narrative weaknesses
    4. Whether the contribution is sufficient for a top venue
    Please be brutally honest.

The review brief should contain the full research context, the specific questions, and the primary artifact / raw-result paths the reviewer should inspect.

For manual backend: use mcp__manual_review__review with the same brief contents. If the manual-review UI supports attachments, attach RESEARCH_REVIEW_REQUEST.md; otherwise paste the brief inline. Save the returned threadId.

Step 3: Iterative Dialogue (Rounds 2-N)

For codex backend: use mcp__codex__codex-reply with the returned threadId. For manual backend: use mcp__manual_review__review_reply with the same threadId. Use the appropriate tool to continue the conversation. For Codex follow-up rounds, write an updated brief such as RESEARCH_REVIEW_ROUND_2.md and send only the path:

mcp__codex__codex-reply:
  threadId: [saved reviewer threadId from Step 2]
  # replies inherit the thread's model/effort (gpt-5.6-sol ultra)
  prompt: |
    Read the updated review brief at <absolute path to
    RESEARCH_REVIEW_ROUND_2.md>.
    Focus on unresolved weaknesses and whether the revision actually fixed them.

For manual follow-up rounds, attach that same updated brief if possible; otherwise paste it inline.

For each round:

1. Respond to criticisms with evidence/counterarguments
2. Ask targeted follow-ups on the most actionable points
3. Request specific deliverables: experiment designs, paper outlines, claims matrices

Key follow-up patterns:

• "If we reframe X as Y, does that change your assessment?"
• "What's the minimum experiment to satisfy concern Z?"
• "Please design the minimal additional experiment package (highest acceptance lift per GPU week)"
• "Please write a mock NeurIPS/ICML review with scores"
• "Give me a results-to-claims matrix for possible experimental outcomes"

Step 4: Convergence

Stop iterating when:

• Both sides agree on the core claims and their evidence requirements
• A concrete experiment plan is established
• The narrative structure is settled

Step 5: Document Everything

Save the full interaction and conclusions to a review document in the project root:

• Round-by-round summary of criticisms and responses
• Final consensus on claims, narrative, and experiments
• Claims matrix (what claims are allowed under each possible outcome)
• Prioritized TODO list with estimated compute costs
• Paper outline if discussed

Update project memory/notes with key review conclusions.

Composed mode — if invoked with — composed: <canonical-report-path> (an orchestrator like /idea-discovery passes this), do not write a standalone review .md in the project root. The raw conversation is already persisted to .aris/traces/… (see Review Tracing below — that audit copy is kept in every mode); fold the review conclusions (consensus, claims matrix, prioritized TODOs) into the orchestrator's canonical report and cite the trace path there. Default (no — composed: directive): behave exactly as above — write the standalone review document. Never infer composed mode from a report file merely existing. Full rules: shared-references/output-composition.md.

Key Rules

• ALWAYS pin model: gpt-5.6-sol + config: {"model_reasoning_effort": "ultra"} for reviews (deep-audit tier; capability fallback per reviewer-routing.md, never below xhigh)
• Put comprehensive context in the review brief. Codex can read local files when you pass an absolute path; manual reviewers usually cannot, so attach or paste the same brief there.
• Be honest about weaknesses — hiding them leads to worse feedback
• Push back on criticisms you disagree with, but accept valid ones
• Focus on ACTIONABLE feedback — "what experiment would fix this?"
• Document the threadId for potential future resumption
• The review document should be self-contained (readable without the conversation)

Prompt Templates

For initial review:

"I'm going to present a complete ML research project for your critical review. Please act as a senior ML reviewer (NeurIPS/ICML level)..."

For experiment design:

"Please design the minimal additional experiment package that gives the highest acceptance lift per GPU week. Our compute: [describe]. Be very specific about configurations."

For paper structure:

"Please turn this into a concrete paper outline with section-by-section claims and figure plan."

For claims matrix:

"Please give me a results-to-claims matrix: what claim is allowed under each possible outcome of experiments X and Y?"

For mock review:

"Please write a mock NeurIPS review with: Summary, Strengths, Weaknesses, Questions for Authors, Score, Confidence, and What Would Move Toward Accept."

Review Tracing

After each reviewer call (mcp__codex__codex, mcp__codex__codex-reply, mcp__manual_review__review, or mcp__manual_review__review_reply), save the trace following shared-references/review-tracing.md (Policy C — forensic; never silently skip). Use save_trace.sh (resolved per the chain in shared-references/integration-contract.md §2) or write files directly to .aris/traces/<skill>/<date>_run<NN>/. Respect the --- trace: parameter (default: full).