Building Production-Grade RAG Systems: From Vector Indexing to Hybrid Search

Retrieval-Augmented Generation (RAG) is often introduced as a simple two-step process: chunk your document, stick it into a vector database, and query an LLM. In real production workloads, naive vector search breaks down fast when dealing with domain terminology, part numbers, or exact keyword queries.

Here is the battle-tested architecture we use to deliver fast, highly accurate context to LLM models.

The Limits of Pure Dense Vector Search

Dense embeddings (e.g., OpenAI text-embedding-3-small, BGE-large) capture semantic intent extremely well. However, they struggle with exact matching:

Code identifiers and variable names
Serial numbers and exact product codes
Acronyms specific to your organization

When users search for error_code_9021, vector similarity might pull general error handling documents instead of the specific bug fix.

Hybrid Search: Combining Dense Vector + BM25 Sparse Search

To solve this, we combine dense semantic search with sparse keyword search (BM25 or SPLADE) and rerank the combined results using a cross-encoder:

PYTHON

# Example Hybrid Search pipeline using Qdrant & Reciprocal Rank Fusion (RRF)
def hybrid_search(query, top_k=10):
    # 1. Fetch dense vector results
    dense_results = qdrant_client.search(
        collection_name="docs",
        query_vector=get_embedding(query),
        limit=top_k * 2
    )
    
    # 2. Fetch sparse BM25 results
    sparse_results = bm25_index.search(query, top_k=top_k * 2)
    
    # 3. Combine scores using RRF
    combined_scores = reciprocal_rank_fusion(dense_results, sparse_results)
    
    # 4. Final reranking using Cohere or BGE Cross-Encoder
    final_docs = rerank(query, combined_scores[:top_k])
    return final_docs

Key Takeaways for Production Deployment

Chunking Strategy: Avoid fixed-size character chunking. Use semantic chunking based on header boundaries or document sections.
Metadata Filtering: Always attach payload metadata (e.g. tenant_id, category, created_at) to narrow vector search spaces before executing dense comparisons.
Cross-Encoder Reranking: Re-ranking top candidates with a model like bge-reranker-large improves top-3 precision by over 30%.