wenmin-wu/nlp-whoosh-fulltext-search-index

Whoosh Full-Text Search Index

Overview

Whoosh is a pure-Python full-text search library (no external dependencies) that supports boolean queries, field-scoped search, wildcards, and proximity operators. It's ideal for Kaggle competitions and prototyping where Elasticsearch is unavailable. Build an index over document fields (title, abstract, classification codes), then query with structured boolean expressions. Supports BM25 scoring out of the box.

Quick Start

import os
from whoosh import index
from whoosh.fields import Schema, TEXT, ID
from whoosh.qparser import MultifieldParser, OrGroup

# Define schema
schema = Schema(
    doc_id=ID(stored=True, unique=True),
    title=TEXT(stored=True),
    abstract=TEXT,
    cpc=TEXT,
)

# Build index
ix_dir = "my_index"
os.makedirs(ix_dir, exist_ok=True)
ix = index.create_in(ix_dir, schema)
writer = ix.writer()
for doc in documents:
    writer.add_document(
        doc_id=doc['id'], title=doc['title'],
        abstract=doc['abstract'], cpc=doc['cpc_codes']
    )
writer.commit()

# Query with field scoping and boolean operators
ix = index.open_dir(ix_dir)
searcher = ix.searcher()
qp = MultifieldParser(["title", "abstract", "cpc"], schema, group=OrGroup)

query = qp.parse("title:neural OR abstract:classification OR cpc:G06F")
results = searcher.search(query, limit=50)
hits = [r['doc_id'] for r in results]

Workflow

1. Define a schema with field types (TEXT for searchable, ID for stored keys)
2. Create index and add documents via writer
3. Open index and create a searcher
4. Build queries with field scoping (field:term), boolean ops (OR, AND), wildcards (term*)
5. Retrieve ranked results by BM25 score

Key Decisions

• Schema design: Use TEXT for searchable fields, ID for unique keys, KEYWORD for tags
• Field scoping: title:word searches only the title field — reduces false positives
• Proximity: ADJ5 (within 5 positions) for phrase-like matching, but doesn't work with wildcards
• Performance: Whoosh is single-threaded — for 100k+ docs, consider batching queries

References

• Basic Whoosh Search Demo
• USPTO - Annealing [LB = 0.31]