Why Most RAG Pipelines Fail Before the LLM Even Sees the Query

Most RAG pipelines look correct on paper — a retriever, a reranker, a generator — and still return garbage in production. The failure isn't usually in the LLM. It's upstream: the wrong chunks reaching the model, unresolved vocabulary mismatches, unconstrained search across noisy vector spaces, and analytics queries being fed to a probabilistic similarity engine designed for text.

The gap between a working prototype and a system you can trust with enterprise data is wider than it looks. What follows is an architectural walkthrough of where that gap actually lives.

Production RAG Pipeline Architecture Overview

---

The Ingestion Problem Nobody Takes Seriously

A flat character-split chunker is the most common source of silent precision loss in RAG. Splitting a PDF into 512-character windows without respecting document structure produces fragments that are syntactically complete but semantically orphaned — a sentence that references "the table above" with no table in its embedding context, or a clause that ends mid-argument because the split landed at 511 characters.

The established standard uses recursive character splitting with a target window of 512–1,024 tokens and a 10–15% sliding overlap. [1] The overlap is doing real work: it prevents key propositions from being truncated across adjacent vectors by ensuring every sentence boundary appears intact in at least one chunk. Structural parsing runs before chunking, extracting section headings, subheadings, and tables as explicit metadata rather than flattening them into the text stream.

Each chunk is stored alongside source lineage metadata — document ID, section path, page range — in a resilient table format like Delta Lake. When embedding models change or chunk boundaries are updated (which they will be, more often than expected), this lineage enables targeted re-indexing rather than a full pipeline rebuild.

---

Hybrid Embeddings: Why Dense-Only Is a Production Liability

Dense embeddings do one thing well: they map conceptually synonymous text close together in latent space regardless of lexical overlap. A query about "vehicle accidents" retrieves chunks about "automotive collisions" correctly. This works until the query contains an exact serial number, a product SKU, a regulatory code, or a piece of domain jargon the embedding model never saw during training — all of which are common in enterprise documents.

The fix is dual-vector representation. Each chunk is encoded by both a dense model (e.g., bge-large-en, text-embedding-3-small) and a sparse model. Modern sparse models like SPLADE don't just do BM25-style term frequency — they assign learned importance weights to query terms while expanding the representation with semantically related lexical variants. [1] The result is a high-dimensional, highly sparse vector where dimensions map directly to tokens, making exact lexical matches reliable without sacrificing semantic coverage.

ONNX-optimized engines like Qdrant FastEmbed generate both vector types within the same data processing pipeline at low latency, so the dual representation doesn't add a separate processing stage — it runs in parallel with the dense encoding step.

---

Qdrant, HNSW, and Why Rank Fusion Replaces Score Normalization

Qdrant stores dense vectors in Hierarchical Navigable Small World (HNSW) graphs and sparse vectors in inverted indexes, executing both retrievals concurrently against a query. [2] The immediate problem is that the outputs are incompatible: dense cosine similarity scores and sparse term scores occupy different numerical ranges. Normalizing them to a common scale requires assumptions about score distributions that don't hold reliably across query types.

Reciprocal Rank Fusion sidesteps this by operating on ranks rather than scores:

$RRF(d\in D)={\sum }_{m\in M}\frac{1}{k+r_m(d)}$

where $r_m(d)$ is the rank of document $d$ in retrieval result set $m$, and $k$ is a smoothing constant (conventionally 60). A document that ranks 3rd in dense retrieval and 5th in sparse retrieval scores higher than one that ranks 1st in dense but absent from sparse entirely. RRF rewards consistent relevance across both channels — the documents that belong at the top tend to surface there regardless of which modality found them first.

The initial hybrid retrieval targets 20–50 candidates. That pool then passes to a cross-encoder reranker.

---

The Reranking Tier: What Bi-Encoders Miss

Bi-encoders — the dense models used for retrieval — encode queries and documents independently and compare them via cosine distance. The advantage is pre-computation: document vectors are indexed offline, so retrieval is fast. The cost is that the model never sees the query and document together, which means it can't evaluate fine-grained token-level interactions.

A cross-encoder like BAAI/bge-reranker-base or jina-reranker-v2-base-multilingual processes the query and candidate chunk simultaneously through full self-attention layers. [1] At inference time, both texts pass through the model jointly, producing a relevance score that captures interactions bi-encoders structurally cannot. The cross-encoder doesn't scale to full index retrieval — it's too slow for that — but applied to 20–50 candidates from hybrid retrieval, it filters down to the 3–5 most authoritative chunks before they reach the generator.

That constraint matters more than it looks. Restricting context to under 3,500–4,000 tokens mitigates the "lost in the middle" effect — the empirically documented phenomenon where LLMs underweight information positioned in the middle of long contexts — while also reducing inference token cost and the probability of context rot from irrelevant surrounding material.

---

Structured Data Is a Different Problem Entirely

Vector similarity search is the wrong tool for analytics. [3] The architecture that works for policy documents fails completely when the question is "What is our customer churn rate in the EU for Q3?"

The three failure modes for structured data are well-documented. Probabilistic vector RAG retrieves static text chunks that contain historical numbers — non-deterministic, unauditable, and incapable of computing a fresh aggregate. Raw Text-to-SQL passes unadorned CREATE TABLE schemas to an LLM and asks it to generate SQL directly, achieving roughly 40% accuracy due to invalid join selections and hallucinated metric definitions. Neither is acceptable for enterprise reporting.

The approach that works is a compilation-first semantic layer positioned between the consumption client and the data warehouse. Frameworks like dbt Semantic Layer (MetricFlow), LookML, Cube, or Databricks Unity Catalog Metric Views act as the compilation engine. The semantic layer defines metrics, dimensions, entities, and join paths in version-controlled YAML or LookML — a declarative graph abstracted away from physical table schemas.

When an analytical query arrives, the semantic layer compiles the natural language intent against its certified metric graph, validates join paths, injects row-level security at compile time, and emits dialect-perfect SQL directly to BigQuery or Databricks. Testing confirms this approach boosts Text-to-SQL accuracy to 83–100% — compared to the ~40% baseline from raw schema prompting. [3]

The comparison across approaches is stark:

Enterprise warehouses often contain thousands of tables — too many to include in an LLM prompt. The solution is RAG for metadata rather than raw data. A dedicated Schema Vector Store indexes enriched semantic descriptions of tables, certified metrics, dimension attributes, and human-verified "Golden Queries" — verified (question, SQL) pairs. When a query arrives, it retrieves only the narrow sub-graph of tables and measures needed to answer that specific question, assembles a targeted prompt with golden query few-shot examples, and feeds the result to a semantic layer compiler rather than generating SQL directly.

---

Hierarchical Indexing for Large Document Corpora

Flat vector search across a 50,000-chunk corpus of technical manuals produces two failure modes: semantic noise from irrelevant sections competing with the right ones, and high retrieval latency from searching an unconstrained space. Both are solvable with hierarchical indexing and payload pre-filtering.

Parent-child chunking addresses the precision-vs-context tradeoff directly. Small child chunks (150–300 tokens) are indexed for dense vector matching — they're precise enough to capture specific semantic concepts. Larger parent blocks (1,000–2,000 tokens) represent the complete surrounding section. During retrieval, similarity matching runs against child chunk vectors, but when a child chunk matches, the engine returns its parent context block to the generator. The LLM receives complete surrounding context instead of an isolated fragment.

For multi-chapter literature, RAPTOR constructs a multi-tier abstraction tree bottom-up: leaf-level chunks are embedded and clustered, LLMs summarize each cluster into abstract summaries, and those summaries are embedded and recursively clustered at higher tiers. [4] Broad thematic queries match near the root; specific technical queries bypass root summaries and retrieve leaf-level chunks directly. The hierarchy routes queries to the right abstraction level rather than competing across the entire flat space.

Qdrant's server-side payload pre-filtering restricts HNSW graph traversal to nodes satisfying payload boolean constraints — book ID, chapter number, section title, access level — before computing any vector distances. [2] This eliminates irrelevant content from the search space entirely, rather than computing distances and discarding results afterward. Standard post-filtering fails when constraints are restrictive: if 99% of the index is filtered out, k-NN search often returns zero valid results because the right entries were excluded during nearest-neighbor selection.

Strict pre-filtering causes its own problem: isolated nodes in the HNSW graph create dead ends during traversal. Qdrant's ACORN algorithm addresses this by evaluating second-hop and multi-hop neighbor connections whenever direct neighbors are filtered out by payload rules, [2] maintaining high recall across specific chapter-scoped searches without requiring separate indexes per document section.

---

Routing Before Retrieval: The 30–85% Inference Cost Cut Nobody Implements

The most expensive mistake in production RAG is retrieving for queries that don't need retrieval. Greetings, out-of-scope questions, compliance queries with hardcoded answers, and analytics questions that belong to the semantic layer — all of these trigger embedding calls, HNSW traversals, and cross-encoder scoring that return nothing useful and add latency.

Semantic routing fixes this at the pre-retrieval layer using lightweight vector classifiers. Pre-embedded exemplar utterances define distinct execution routes — conversational, structured analytics, document search, operational handoff. When a query arrives, an ultra-fast encoder (MiniLM, FastEmbed) computes cosine distance against each route's exemplar cluster: [5]

$S(q,e_i)=\frac{q\cdot e_i}{\Vert q\Vert \Vert e_i\Vert },e_i\in E_r$

Route selection applies a confidence threshold $\tau$ (typically 0.75–0.82):

$\text{Selected Route}=\{\begin{array}{ll}\arg {\max }_r\left({\max }_{e_i\in E_r}S(q,e_i)\right)& \text{if }\max S\ge \tau \\ \text{Fallback (General RAG)}& \text{if }\max S<\tau \end{array}$

The whole classification runs in under 10 milliseconds, cuts downstream LLM inference expenses by 30–85% depending on query mix, and improves end-to-end system accuracy from ~58% to over 83% by ensuring each query reaches the execution engine actually built for it. [5]

Before the semantic router runs, a deterministic rule-based triage layer handles the cases that shouldn't reach ML models at all: regex pattern matching for sensitive account actions, safety screening via models like Llama-Guard or NeMo Guardrails, and hardcoded overrides for compliance questions in regulated domains where LLM non-determinism is unacceptable.

---

What Holds the Whole System Together

The full architecture is four sequential stages: a pre-retrieval control plane with rule-based triage and semantic routing, specialized retrieval branches (semantic layer for structured data, ACORN-filtered hybrid vector retrieval for documents), a constrained prompt assembly stage that caps context under 4,000 tokens, and a post-generation verification layer that checks faithfulness before the response leaves the system.

Continuous evaluation runs in the background using RAGAS metrics: [1]

$\text{Faithfulness}=\frac{\text{Claims Supported by Retrieved Context}}{\text{Total Claims Generated}}$

$\text{Context Recall}=\frac{\text{Ground Truth Statements in Retrieved Chunks}}{\text{Total Ground Truth Statements}}$

Queries and retrieved outputs log asynchronously to Delta inference tables. Embedding drift monitoring detects shifts in query distributions relative to the indexed corpus and triggers targeted re-indexing — the lineage metadata stored during ingestion makes this surgical rather than a full pipeline rebuild.

The architectural insight that connects all of this is that retrieval failures are almost always upstream failures. By the time a hallucinated answer reaches a user, the real error happened at ingestion (wrong chunk boundaries), at retrieval (vocabulary mismatch, unconstrained search space), at routing (analytics query sent to a vector engine), or at context assembly (too many tokens, wrong abstraction level). The LLM at the end is usually doing exactly what the architecture asked it to — which is the problem.

---

References

1. Agile Ventures — Building Production RAG Pipelines on Databricks: A Practical Guide
2. Towards AI — Beyond Vectors: A Deep Dive into Modern Search in Qdrant
3. Colrows — RAG vs. Semantic Layer (2026): Which One Do AI Agents Need?
4. Machine Learning Plus — RAPTOR RAG Explained: Building Hierarchical Retrieval for Smarter AI Answers
5. Guild.ai — Query Routing (AI)