orcaqubits/nlweb-dev-patterns

NLWeb Development Patterns

Before writing code

Fetch live docs:

1. Fetch https://github.com/nlweb-ai/NLWeb/blob/main/docs/nlweb-systemmap.md for module layout.
2. Fetch https://github.com/nlweb-ai/NLWeb/blob/main/docs/nlweb-control-flow.md for the request lifecycle.
3. Fetch https://github.com/nlweb-ai/NLWeb/blob/main/docs/life-of-a-chat-query.md for an end-to-end trace.
4. Fetch https://github.com/nlweb-ai/NLWeb/blob/main/docs/nlweb-configs-files.md for the config precedence rules.
5. Inspect core/baseHandler.py, core/router.py, core/retriever.py, core/ranking.py for current code paths.

Pattern: Mixed-Mode Programming

NLWeb's defining design choice. Rather than one big LLM call per query, NLWeb makes many small calls, each with a strict JSON output schema (<returnStruc>), feeding Python control flow.

Implications:

• Cost and latency scale with the number of call sites, not the size of any one call.
• Failures are localized — one bad call doesn't poison the response.
• Steerability is high — you can tune any single prompt without touching the rest.
• Debugging is harder — you must trace which of N calls misbehaved.

When designing extensions, follow the same pattern: small, schema-constrained LLM calls, deterministic Python glue.

Pattern: FastTrack vs Analysis (Parallel Paths)

NLWebHandler runs two paths in parallel:

| Path | What it does | When it wins | |------|--------------|--------------| | FastTrack | Immediate vector search → stream early results | Common queries with obvious retrieval matches | | Analysis | Decontextualize → detect type → route via ToolSelector to a specific handler | Ambiguous queries, complex flows (compare, recipe substitution) |

Both paths stream into the same response. FastTrack results appear quickly; Analysis results appear when ready. The agent decides whether to render incrementally or wait.

Implications for handlers you write: if you write a slow, expensive handler, FastTrack will still beat you to first byte for simple queries. That's fine — it's the design.

Pattern: Config File Precedence

8 YAML config files in config/. Precedence (highest first):

1. Environment variables (always win)
2. Query-string params — but only when mode: development in config_webserver.yaml
3. YAML defaults

The mode: development override is a foot-gun in production. A query like ?write_endpoint=other_qdrant would silently switch the write target. Always set mode: production before deploying.

Pattern: "Headers" Are In-Stream Messages, Not HTTP Headers

NLWeb's "NLWS headers" mechanism is JSON message objects on the SSE channel, not HTTP response headers. Each carries a message_type:

| message_type | Carries | |--------------|---------| | license | Content license terms | | data_retention | How long the agent may cache | | cache_policy | Caching directives | | usage_terms | Acceptable use | | rate_limits | Calls/sec, daily quota | | data_freshness | Last index time | | api_version | NLWeb release identifier | | ui_component | Optional rendering hint |

Client parsing rule: buffer message objects until you see a results chunk or terminal marker. Don't assume the first chunk is data.

Pattern: Embedding/Ingest Determinism

The most common NLWeb bug: changing the embedding provider after ingest, getting empty or garbage results.

Rule: pick the embedding provider FIRST, configure the retrieval backend's vector dimension to match, ingest with that provider, query with that provider. Never change mid-stream without re-ingesting.

If you need to migrate embedding providers:

1. Choose a maintenance window
2. Configure the new provider as the preferred_provider
3. db_load.py --only-delete delete-site <site> for each site
4. Re-ingest with the new provider
5. Restart and verify

Pattern: Debugging the LLM Call Chain

When /ask returns a bad answer, the bug is in one of these call sites:

| Call site | Symptom | Fix | |-----------|---------|-----| | Decontextualize | Query rewritten wrong; off-topic results | Pre-compute decontextualized_query, log the prompt's output | | Type detection | Wrong handler invoked | Pass itemType explicitly, or check site_types.xml | | Tool selection | Right type, wrong tool | Adjust tool descriptions; set tool_selection_enabled: false to bypass | | Ranking | Top results are off | Check embedding alignment first; then try scorer=nlwebscorer | | Summarize / generate | Final answer is poor | Improve Schema.org source data; bump model tier |

Isolate by mode: mode=list skips summarize/generate. If list is bad, the issue is retrieval or ranking, not synthesis.

Pattern: NLWebScorer (Optional Neural Reranker)

The NLWebScorer/ subsystem provides a ModernBERT + GAM neural reranker as an alternative to LLM-based ranking. Activate via ?scorer=nlwebscorer on /ask. Configure checkpoints in config_*.yaml:

scorers:
  nlwebscorer:
    bert_checkpoint: ./checkpoints/modernbert.pt
    gam_checkpoint: ./checkpoints/gam.pt

Use cases:

• Cost reduction (LLM-ranking is expensive at scale)
• Latency reduction (BERT is faster than even small LLMs)
• Reproducible ranking (no LLM stochasticity)

Tradeoff: it's domain-specific — you may need to fine-tune on your data. See docs/training-recipe-modernbert-gam.md.

Pattern: The Five Subsystems

NLWeb's repo isn't just one server. Five top-level folders are conceptually distinct:

| Subsystem | Purpose | When relevant | |-----------|---------|---------------| | AskAgent/ | The core /ask and /mcp server | Always | | AgentFinder/ | Cross-site NLWeb discovery (federated /who) | Multi-site federations | | DataFinder/ | NL→SQL for enterprise sources (HubSpot, Dynamics, Jira) | Enterprise data, not vector-backed | | ModelRouter/ | Cost/quality routing across LLM providers | Cost optimization at scale | | NLWebScorer/ | Neural reranker (ModernBERT + GAM) | High-volume retrieval |

Most deployments use only AskAgent. The rest are opt-in.

Pattern: A2A and MCP as Co-Equal Bindings

NLWeb supports three transport bindings in parallel:

| Binding | Path | Audience | |---------|------|----------| | REST /ask | port 8000 | Browsers, custom clients | | MCP /mcp | port 8000 | AI agents (Claude, Gemini, native MCP) | | A2A | webserver/a2a_wrapper.py, route a2a.py | Google Agent-to-Agent protocol | | AppSDK adapter | port 8100 | ChatGPT specifically |

All share the same backend pipeline. No data duplication. Choose by audience, not by feature.

Pattern: Conversation Memory Hooks

core/conversation_history.py persists exchanges per authenticated user. methods/conversation_search.py queries the persisted history.

Long-term memory (cross-conversation user preferences) is NOT shipped. Hook points to add it:

• After response generation in NLWebHandler.respond() — extract durable facts, write to user profile
• Before query in the same handler — load user profile, inject into the decontextualize prompt

This is intentional: NLWeb leaves opinionated personalization to the integrator.

Pattern: Idempotency and Retries

NLWeb doesn't define idempotency keys — /ask calls are read-side; replays are safe. /mcp follows JSON-RPC 2.0 semantics: include id in every request, retry with the same id if the connection drops mid-request (server may dedup if implemented).

For db_load.py, idempotency is upsert by URL. Re-running on the same source updates existing records rather than duplicating.

Pattern: Schema.org as the Common Currency

Every result carries a schema_object. Agents pattern-match on @type to render appropriately. Design rule: any new tool or handler you write should preserve the schema_object in its output. Don't strip it down to text — that defeats the whole point of NLWeb.

Pattern: Versioning

NLWeb releases as dated markdown files in docs/release_notes/, not semver tags. When pinning a deployment:

• Pin the git commit, not a tag
• Read the release_notes entries from your pinned commit to the latest before upgrading
• The MCP wrapper docstring explicitly warns "Backwards compatibility is not guaranteed" — re-test agent integrations on every upgrade

Pattern: Don't Modify Core Files

Most extensibility goes via:

• config/*.yaml and XML files (preferred)
• New files in methods/ (custom handlers)
• New providers in llm_providers/, embedding_providers/, retrieval_providers/
• aiohttp middleware in webserver/middleware/

Avoid editing core/baseHandler.py, core/router.py, etc. — they change frequently and your fork rots.

Pattern: Disable Defaults Aggressively

The default config enables three retrieval backends (qdrant_local, nlweb_west, shopify_mcp), the federated /who endpoint, and mode: development. For any non-demo deployment, set these:

# config_webserver.yaml
mode: production

# config_nlweb.yaml
who_endpoint_enabled: false

# config_retrieval.yaml
endpoints:
  nlweb_west: { enabled: false }
  shopify_mcp: { enabled: false }

These defaults make sense for hello-world demos. They are anti-patterns for production.

---

Always cross-reference with the latest docs/release_notes/ and the live core/ modules — patterns evolve and the code is the source of truth.