davepoon/onboard

Onboard CIAgent into this repo

You are setting up CIAgent (pip install ciagent) so this repo's AI agent has recorded golden baselines and a runnable regression suite. The end state: the user can run ciagent test --runs 3 and see a stability report for their agent.

Work through the steps in order. Do not skip the cost gate in step 4.

1. Find the agent and install CIAgent

• Locate the agent: search for LLM SDK usage (openai, anthropic, langgraph, langchain) and for the function or endpoint that takes a user message and returns the agent's answer.
• Install with the matching extra so trace capture hooks the SDK: pip install "ciagent[openai]", [anthropic], [langgraph], or [all].
• Sanity check: ciagent --version then ciagent doctor (it reports what is missing; a missing spec is expected at this point).

2. Write the runner

Create agentci_runner.py at the repo root (or inside the package if the repo has one clear package):

def run_for_agentci(query: str) -> str:
    """CIAgent entry point: one query in, final answer text out."""
    # import the user's agent and invoke it ONCE, no chat history
    ...
    return final_answer_text

Rules:

• Return the final answer string. CIAgent wraps the call in its own trace capture, so LLM calls and tool calls are recorded automatically — do not build Trace objects unless the repo already produces them.
• Fresh context per call: no shared history between queries.
• Reuse the repo's own config/env loading so the runner works from the repo root.
• Verify it imports and answers before going further: python -c "from agentci_runner import run_for_agentci; print(run_for_agentci('hello'))".

3. Choose queries

Write agentci_queries.txt, one query per line — 8 to 15 queries:

• Cover the agent's main jobs (mine the README, docs, knowledge base, prompts, and existing tests for what it is supposed to handle).
• Include at least 2 out-of-scope queries the agent should refuse or deflect.
• Prefer queries whose correct answers contain hard facts (prices, dates, limits, names) — those become deterministic checks in step 6.

4. Cost gate — ask before running live

Recording baselines runs the real agent once per query, on the user's API keys. State the query count and a cost ballpark, and ask the user to confirm before step 5. If there are no API keys or the user declines: write agentci_spec.yaml by hand instead (same queries, runner: set), validate with ciagent test --mock, and tell the user which step to resume later.

5. Record golden baselines

ciagent bootstrap --runner agentci_runner:run_for_agentci \
  --queries agentci_queries.txt --agent <agent-name> --yes

This runs every query, saves each trace as a golden baseline under ./baselines/<agent-name>/, and writes agentci_spec.yaml with path and cost budgets derived from the recorded traces. Read the printed answers as they stream by — if an answer is visibly wrong, that query should not be golden: fix the agent or the query, delete that baseline file, and rerun.

6. Add correctness checks

The generated spec has path and cost budgets but no correctness checks. Add a correctness: block per query, derived from the recorded baseline answers and the repo's docs/KB — never from what you wish the agent said:

correctness:
  expected_in_answer: ["30 days"]          # hard facts, AND
  any_expected_in_answer: ["$9.95", "9.95"] # phrasing variants, OR
  not_in_answer: ["I don't know"]           # forbidden content

Check facts, not phrasing. If the repo has a knowledge-base directory, run ciagent generate-checks --kb <dir> --dry-run and review its candidates — every surviving candidate was already validated against the recorded goldens.

7. Verify

ciagent test --mock                      # structure check, zero API calls
ciagent test --yes --format json         # live run (covered by step 4 approval)
ciagent test --runs 3 --yes              # stability report

Exit codes: 0 = pass (flaky-but-passing is 0), 1 = correctness failure in every run, 2 = infra/config error. In the stability report, flips labeled agent-variance mean the agent's answer changed (an agent problem); flips labeled judge-flake mean the eval itself is unstable (a check/judge problem).

If a check fails, fix the agent or fix a factually wrong check. Do not loosen a correct check to make the run green — report the failure to the user instead.

8. Wire CI and hand off

• ciagent init scaffolds a GitHub Actions workflow (add --hook for a pre-push hook if the user wants it).
• Commit: runner, agentci_queries.txt, agentci_spec.yaml, baselines/, and the workflow.
• Tell the user: how many goldens were recorded, the suite score, anything flaky (with its flip source), and that ciagent test --runs 3 is the command to watch after future agent changes.