orcaqubits/nlweb-prompts-customization

NLWeb Prompts Customization

Before writing code

Fetch live docs:

1. Fetch https://github.com/nlweb-ai/NLWeb/blob/main/docs/nlweb-prompts.md for the prompts framework reference.
2. Fetch https://github.com/nlweb-ai/NLWeb/blob/main/config/prompts.xml for the canonical prompt templates currently shipped — these change between releases.
3. Fetch https://github.com/nlweb-ai/NLWeb/blob/main/config/site_types.xml for per-type prompt inheritance.
4. Inspect AskAgent/python/core/prompts.py to see how prompts are loaded and parameterized.
5. Cross-reference with handlers in methods/ to see which prompt each LLM call site uses.

Conceptual Architecture

Two XML Files Drive Prompt Behavior

| File | Purpose | |------|---------| | prompts.xml | Defines <promptString> templates and their <returnStruc> JSON output contracts | | site_types.xml | Maps Schema.org @type → tool list + per-type prompt overrides, with inheritance |

The <promptString> Template

A typical prompt entry:

<promptString name="decontextualize_query">
  <description>Rewrite the user's query to be standalone, given prior conversation.</description>
  <input>
    <field name="prev_queries" type="array" />
    <field name="current_query" type="string" />
  </input>
  <text>
Given the prior queries: {prev_queries}
And the current query: {current_query}
Return a standalone version of the current query.
  </text>
  <returnStruc>
{
  "decontextualized": "the standalone query as a single string"
}
  </returnStruc>
</promptString>

The exact tag names may differ — verify the live prompts.xml. The pattern: human-readable description, input schema, prompt text with {placeholders}, JSON output contract.

Per-Type Prompt Inheritance

site_types.xml uses extends to inherit prompts up a type hierarchy:

<site_type name="Recipe" extends="CreativeWork">
  <prompt name="rank_results" override="rank_results_recipe" />
  <tool>recipe_substitution</tool>
</site_type>

When the handler asks for the rank_results prompt at runtime, it walks the type chain: Recipe → CreativeWork → default, taking the first override.

Prompt Categories

Common prompt types (verify against live file):

| Category | Used For | |----------|----------| | decontextualize_query | Rewrite "what about the second one?" → "tell me about <item X>" | | detect_item_type | Identify Schema.org type the user is asking about | | select_tool | Router decides which methods/ handler to invoke | | rank_results | LLM scores retrieved items by relevance | | summarize_results | mode=summarize condensation | | generate_answer | mode=generate RAG synthesis | | extract_key_fields | Pull specific Schema.org properties for the answer |

The <returnStruc> Contract

Every LLM call has a strict JSON output schema. NLWeb parses the response as JSON and crashes if it doesn't validate. This is intentional — mixed-mode programming requires deterministic parsing.

Tips for writing <returnStruc>:

• Keep the JSON minimal — every field is more tokens.
• Use enums for categorical outputs to reduce hallucinations.
• Always include a reasoning field — invaluable for debugging.
• Don't nest arbitrarily — flat JSON parses more reliably.

Localization

Prompts are plain text — translate them to localize. NLWeb doesn't have first-class i18n; site operators fork the prompt file per locale (or load locale-specific files). Verify whether the runtime supports per-request locale selection in the current release.

Custom Domain Prompts

To add prompts for a new Schema.org type (or a custom type):

1. Add <site_type name="YourType" extends="..."/> in site_types.xml.
2. Add <promptString> entries with names like rank_results_yourtype, generate_answer_yourtype in prompts.xml.
3. Reference them via <prompt name="rank_results" override="rank_results_yourtype"/> in the site_type.
4. Restart; verify by query.

Implementation Guidance

Editing a Prompt Safely

Prompts are upgrade-fragile. Strategy:

1. Fork prompts.xml to a site-specific file (e.g., prompts_mysite.xml).
2. Load it via the config option (verify whether NLWeb supports a prompts_file: setting per site — varies by release).
3. If no per-site option exists, accept that you'll re-merge on upgrade.

Tuning Answer Quality

In order of cost-effectiveness:

1. Fix the data (better Schema.org → better answers). Most "bad answer" complaints are upstream of the prompt.
2. Increase the model tier for the failing call site (low → high in config_llm.yaml).
3. Tighten the <returnStruc> so the model can't ramble.
4. Edit the prompt text to give clearer instructions or examples.
5. Add per-type overrides so each domain gets a tailored prompt.

Debugging a Prompt

• Set tool_selection_enabled: false to bypass routing and isolate the prompt under test.
• Hit /ask with mode=list to skip generate-mode prompts entirely.
• Add a reasoning field to the <returnStruc> and log it.
• Run the prompt manually in the LLM provider's playground with realistic inputs.
• Use a debugger / log breakpoint in core/prompts.py to see the fully-interpolated prompt.

Common Prompt Failures

• Model returns invalid JSON — <returnStruc> is too complex, model tier too low, or prompt text is ambiguous. Bump to high tier and simplify.
• Model returns extra fields — make the schema stricter; explicitly forbid extras in the prompt text.
• Inheritance doesn't apply — typo in extends or override attribute; check exact XML attribute names against the live file.
• Localized prompts break parsing — <returnStruc> field names must stay English; only the human-facing prompt text translates.
• Long prompts cost too much — every /ask triggers multiple LLM calls; even a small prompt-length increase multiplies.

Prompts Are Per-Call-Site, Not Per-Query

A single /ask invocation may use 3-5 different prompts (decontextualize, select tool, rank, summarize). Editing one only affects that call site. Use call-site logs to confirm which prompt fired.

Versioning Prompt Changes

Prompts are config — version them in git alongside your other config files. When upgrading NLWeb, diff the upstream prompts.xml against yours to catch new prompts you should adopt.

Always re-fetch prompts.xml and site_types.xml from the live repo before customizing — XML attribute names and prompt template syntax evolve.