oaustegard/creating-kb

creating-kb

Turn a pile of files into a portable, deployable knowledgebase. The output is a .skill bundle — an ordinary zip — containing a BM25 inverted index, the chunk text, a pure-Node searcher, and a query protocol. It has no embedding model and no semantic search: retrieval is lexical, and the consuming agent supplies the semantic layer by expanding the query at search time. That is what makes the bundle portable — any agent that can run node can query it with no npm install, no model download, and no network.

The whole toolchain is JavaScript so one implementation serves both this builder and the in-browser packer. Build with the bundled script; do not hand-roll the index.

SCRIPTS=/mnt/skills/user/creating-kb/scripts
node $SCRIPTS/build_lexkb.js CORPUS_DIR --out /tmp/kb --name my-kb --zip

Workflow

1. Gather the sources

Collect the files into one directory. In a Claude.ai chat, uploads land in /mnt/user-data/uploads/ — point the builder there. Otherwise use any path the user names. Supported extensions default to txt,md,html,htm; pass --ext to change them.

This MVP interface is bounded by how many files a chat can accept. For a large corpus, stage the files in a directory first, or use the browser packer (built from the same scripts) that runs entirely client-side.

2. Build the bundle

SCRIPTS=/mnt/skills/user/creating-kb/scripts
node $SCRIPTS/build_lexkb.js /mnt/user-data/uploads \
  --out /tmp/kb --name my-kb --zip \
  --source "human description of the corpus"

The script chunks each file, builds the BM25 index, writes the bundle dir (SKILL.md + search.js + index.json + chunks.jsonl), and — with --zip — emits my-kb.skill next to --out.

3. Deliver

Move the .skill to the outputs directory and give the user a download link:

cp /tmp/my-kb.skill /mnt/user-data/outputs/

[Download my-kb.skill](computer:///mnt/user-data/outputs/my-kb.skill)

Tell the user how to deploy it: unzip into an agent's skill directory (or upload it as a skill). The bundle's own SKILL.md then drives querying — the consuming agent reads it, expands each question into search terms, and runs the bundled search.js. No further setup.

Choosing chunk size

The retrieval unit and the reasoning unit are decoupled, which makes chunk size a low-stakes choice. search.py/search.js rank on the whole chunk (best recall) but return only the query-densest passage of it by default (--snippet, ~1200 chars), so a big chunk does not flood the consuming agent's context with surrounding noise. Index for recall; the searcher handles signal.

--target-chars controls chunk size (whole paragraphs are packed up to the target; --target-chars 0 makes each file one chunk). Lexical BM25 tolerates — and on a real-corpus sweep slightly preferred — larger chunks than embedding-based retrieval, because there is no vector to dilute: BM25 scores individual term presence with length normalization, so a big chunk still ranks on the exact terms it contains.

• Default: --target-chars 0 (whole document). Best recall, fewest chunks; the snippet return keeps reasoning context focused.
• Long, multi-topic files where you want tighter citation units: 1500–4000.
• 500 only if you need very fine-grained chunk ids and accept more chunks.

Verifying the bundle

Test before delivering. Run a query against the freshly built bundle and confirm it returns sensible hits:

node /tmp/kb/search.js --query "a representative question" \
  --core "key term" --expand "synonym" --k 3

Each hit's text is the query-focused passage by default; add --snippet 0 to inspect a full chunk.

search.js prints JSON {"hits": [...]}. Confirm the right chunks surface.

What ships in the bundle

| File | Role | |---|---| | SKILL.md | the query protocol the consuming agent follows (expand → search → cite) | | search.js / search.py | equivalent BM25 + RM3 + metadata-filter searchers; return query-focused passages (matched sentences kept in neighbour context, merged); the agent runs whichever runtime it has | | index.json | precomputed inverted index (postings, df, doc lengths, BM25 params) | | chunks.jsonl | chunk text + structured metadata |

Both searchers are thin readers of the same neutral JSON index, so the bundle runs in a Node-only or a Python-only consumer. Metadata stays structured (not folded into the indexed text), which lets the consuming agent filter on it (--filter section=blog, --filter date>=2025).

Scripts

• scripts/build_lexkb.js — chunker + BM25 index builder + .skill writer.
• scripts/search.js — the JS runtime searcher, copied verbatim into every bundle. It owns the tokenizer; the builder imports it so index and queries tokenize identically.
• scripts/search.py — the Python runtime searcher, copied verbatim into every bundle; a thin reader of the same neutral JSON index, parity-pinned to search.js (identical results on a shared index).
• scripts/zipstore.js — pure-JS ZIP-STORED writer (used by the builder; shared with the in-browser packer).
• scripts/bundle_SKILL.md — the query-side SKILL.md template written into each bundle.