jmsktm/Vector Search Designer

Vector Search Designer

The Vector Search Designer skill helps you architect and implement vector similarity search systems that power semantic search, recommendation engines, and AI applications. It guides you through selecting the right vector database, designing index structures, optimizing query performance, and scaling to millions or billions of vectors.

Vector search has become foundational to modern AI systems, from RAG pipelines to product recommendations. This skill covers the full stack: understanding approximate nearest neighbor (ANN) algorithms, choosing between database options, tuning recall vs latency tradeoffs, and implementing production-ready search infrastructure.

Whether you are building on Pinecone, Weaviate, Qdrant, pgvector, or implementing your own solution, this skill ensures your vector search system meets your performance and accuracy requirements.

Core Workflows

Workflow 1: Select Vector Database

1. Gather requirements:
   • Scale: How many vectors?
   • Query patterns: Single vs batch, filters needed?
   • Latency requirements: Real-time vs batch?
   • Update frequency: Static vs dynamic?
   • Infrastructure: Managed vs self-hosted?
2. Compare options: | Database | Scale | Managed | Features | Best For | |----------|-------|---------|----------|----------| | Pinecone | Billion+ | Yes | Hybrid, namespaces | Production, zero-ops | | Weaviate | 100M+ | Both | GraphQL, modules | Multi-modal, complex queries | | Qdrant | 100M+ | Both | Rust perf, filtering | High performance, self-hosted | | Milvus | Billion+ | Both | GPU support, clustering | Large scale, ML teams | | pgvector | 10M | No | PostgreSQL native | Existing Postgres, small scale | | Chroma | 1M | No | Simple API | Prototyping, embedded |
3. Evaluate with your data
4. Document decision rationale

Workflow 2: Design Index Architecture

1. Choose ANN algorithm:
   • HNSW: Best recall/speed tradeoff, memory intensive
   • IVF: Good for very large datasets, requires training
   • PQ: Compression for scale, some accuracy loss
   • Flat: Exact search, small datasets only
2. Configure index parameters:

   # HNSW example configuration
   hnsw_config = {
       "M": 16,  # Connections per node (higher = better recall, more memory)
       "efConstruction": 200,  # Build-time search depth
       "efSearch": 100,  # Query-time search depth
   }
   
   # IVF example configuration
   ivf_config = {
       "nlist": 1024,  # Number of clusters
       "nprobe": 32,  # Clusters to search at query time
   }
   

3. Plan sharding strategy for scale
4. Design metadata schema for filtering

Workflow 3: Optimize Search Performance

1. Benchmark baseline performance:
   • Queries per second (QPS)
   • Latency (p50, p95, p99)
   • Recall@k accuracy
2. Identify bottlenecks:
   • Index loading time
   • Search latency
   • Filter computation
   • Network overhead
3. Apply optimizations:
   • Tune index parameters (ef, nprobe)
   • Implement query caching
   • Optimize filter expressions
   • Consider quantization for memory
4. Validate recall vs latency tradeoff
5. Document optimal configuration

Quick Reference

| Action | Command/Trigger | |--------|-----------------| | Choose database | "Which vector database for [use case]" | | Design index | "Design vector index for [scale]" | | Optimize search | "Speed up vector search" | | Add filtering | "Add metadata filters to vector search" | | Scale vectors | "Scale to [N] million vectors" | | Benchmark search | "Benchmark vector search performance" |

Best Practices

• Right-Size Your Database: Don't over-engineer for scale you don't need

  • <1M vectors: pgvector or Chroma often sufficient
  • 1-100M vectors: Qdrant, Weaviate, or Pinecone
  • 100M+ vectors: Milvus or Pinecone with careful design

• Understand Recall vs Speed Tradeoff: ANN is approximate by design

  • Higher recall = slower queries
  • Tune based on your accuracy requirements
  • Measure actual recall, don't assume

• Use Hybrid Search When Needed: Combine vector and keyword search

  • Vector: semantic similarity
  • Keyword (BM25): exact terms, names, codes
  • Typically improves results for mixed queries

• Design Metadata for Filtering: Plan your filter strategy upfront

  • Indexed fields for frequent filters
  • Avoid filtering on high-cardinality fields
  • Pre-filter vs post-filter tradeoffs

• Batch Operations When Possible: Reduce network overhead

  • Batch upserts for ingestion
  • Batch queries when latency allows
  • Use async operations

• Monitor and Alert: Production search needs observability

  • Query latency percentiles
  • Index size and memory usage
  • Recall degradation over time

Advanced Techniques

Multi-Vector Search

Handle documents with multiple representations:

class MultiVectorIndex:
    def __init__(self):
        self.title_index = VectorIndex(dim=768)
        self.content_index = VectorIndex(dim=768)
        self.summary_index = VectorIndex(dim=768)

    def search(self, query_embedding, weights=None):
        weights = weights or {"title": 0.3, "content": 0.5, "summary": 0.2}

        results = {}
        for field, weight in weights.items():
            index = getattr(self, f"{field}_index")
            field_results = index.search(query_embedding, k=20)
            for doc_id, score in field_results:
                results[doc_id] = results.get(doc_id, 0) + score * weight

        return sorted(results.items(), key=lambda x: x[1], reverse=True)[:10]

Filtered Vector Search Strategies

Optimize search with filters:

def filtered_search(query_embedding, filters, k=10):
    # Strategy 1: Pre-filter (for selective filters)
    if estimate_selectivity(filters) < 0.1:
        candidate_ids = apply_filters(filters)
        return vector_search_subset(query_embedding, candidate_ids, k)

    # Strategy 2: Post-filter (for non-selective filters)
    elif estimate_selectivity(filters) > 0.5:
        results = vector_search(query_embedding, k * 3)
        filtered = [r for r in results if matches_filters(r, filters)]
        return filtered[:k]

    # Strategy 3: Hybrid (general case)
    else:
        return vector_search_with_filters(query_embedding, filters, k)

Quantization for Scale

Reduce memory with acceptable accuracy loss:

# Product Quantization configuration
pq_config = {
    "nbits": 8,  # Bits per sub-quantizer
    "m": 16,  # Number of sub-quantizers
    # 768-dim * 4 bytes = 3KB/vector -> 16 * 1 byte = 16 bytes/vector
}

# Binary quantization (extreme compression)
binary_config = {
    "threshold": 0,  # Values > 0 -> 1, else -> 0
    # 768-dim * 4 bytes = 3KB/vector -> 768 bits = 96 bytes/vector
}

Incremental Index Updates

Handle dynamic data efficiently:

class DynamicVectorIndex:
    def __init__(self, rebuild_threshold=10000):
        self.main_index = build_optimized_index()
        self.delta_index = []  # Recent additions
        self.rebuild_threshold = rebuild_threshold

    def add(self, vector, metadata):
        self.delta_index.append((vector, metadata))
        if len(self.delta_index) >= self.rebuild_threshold:
            self.rebuild()

    def search(self, query, k):
        main_results = self.main_index.search(query, k)
        delta_results = brute_force_search(self.delta_index, query, k)
        return merge_results(main_results, delta_results, k)

    def rebuild(self):
        all_data = self.main_index.get_all() + self.delta_index
        self.main_index = build_optimized_index(all_data)
        self.delta_index = []

Common Pitfalls to Avoid

• Using exact (flat) search at scale instead of ANN
• Not measuring actual recall, assuming ANN is "good enough"
• Over-indexing metadata fields, slowing down updates
• Ignoring the memory requirements of HNSW indexes
• Not planning for index rebuilds and maintenance windows
• Assuming vector databases handle all use cases (sometimes Elasticsearch is better)
• Forgetting to normalize vectors for cosine similarity
• Mixing embeddings from different models in the same index