Advanced RAG

Why this week is positioned where it is. Week 5 gave you the floor — the simplest pipeline that works at all. Week 6 is about everything between the floor and a production-grade system. The framing matters: these techniques are usually additive, not alternatives. A real production RAG system uses hybrid search and a reranker and query rewriting and contextual chunks. The wins compound. We'll build them up one at a time.

1. The advanced-RAG mindset

Naive RAG is the floor. Production RAG is multiple retrieval signals, query understanding, second-stage scoring, and continual evaluation. Each technique adds a few points of retrieval recall; in combination, they take a system from "60% of queries get the right context" to "92%."

The mindset shift from Week 5 is this: retrieval is a pipeline, not a function call. A request flows through query rewriting → multiple retrieval signals → fusion → reranking → metadata filtering → context assembly. Each stage has a budget (latency, cost, model calls), each has a failure mode, each can be turned off and measured independently.

The order to add things, roughly:

1. Metadata filtering. Always. Never optional in multi-tenant systems.
2. Hybrid search. Adds 5–10 points of recall over dense-only on technical content. Cheap.
3. Reranker. Adds another 5–15 points of precision. The single highest-leverage addition for most teams.
4. Query rewriting. Adds 5–10 points on hard queries (vague, multi-intent, very short). One LLM call upstream.
5. Contextual retrieval. Adds 10–30 points on docs where chunks lack standalone context. Index-time cost only.
6. Agentic / multi-hop retrieval. Adds capability for questions baseline RAG simply can't answer. Significant cost.

You don't need all of them on day one. You do need to know they exist and roughly when each becomes worth it.

2. Hybrid search — lexical + semantic together

Dense embeddings are great at semantics and bad at exact matches. Ask "what does error code 401 mean?" and the embedding for "401" is just a number-shaped vector that doesn't strongly attract chunks containing the literal token "401." Ask "rollback after health_check_path failure" and the dense model ignores the underscore-laden identifier; it sees "rollback failure" and matches on that, missing the chunk where health_check_path is the exact answer.

The fix: run two retrieval systems in parallel. BM25 (the classical information-retrieval algorithm — token frequencies with IDF and length normalization) handles exact-match needs. Dense embedding search handles semantic generalization. Fuse the scores.

The fusion math is simple. After running both retrievers, you have two ranked lists with two score scales. Two patterns dominate:

Score normalization + linear fusion:

score_hybrid = α · normalize(dense_score) + (1 - α) · normalize(bm25_score)

α ≈ 0.6 is a common default — slightly favoring dense because it generalizes. Tune per corpus.

Reciprocal rank fusion (RRF):

score_hybrid(d) = sum over retrievers of 1 / (k + rank(d))

where k is a constant (60 is the canonical default). RRF is rank-based, score-scale-independent, and the safer choice when you're not confident in your normalization.

The 2026 production pattern: hybrid search by default, with α tuned on a held-out eval set. The alternative — pure dense — leaves real points on the table on technical content, and pure BM25 fails on paraphrases. Hybrid is dominant in part because the gap is small enough that "do both" is cheap insurance.

When hybrid doesn't help: corpora that are very pure prose (literary, conversational), where lexical signal adds noise more than it helps. For technical docs, codebases, support content — always hybrid.

3. Rerankers — second-stage precision

Vector search returns the top-K candidates. A reranker scores those candidates as a pair with the query and produces a sharper top-K.

The architecture: - First-stage retrieval (cheap, fast): embedding search returns top-50. - Second-stage reranking (expensive, precise): cross-encoder scores all 50 with the query, returns top-5.

The math is the difference between embeddings and cross-encoders: - An embedding model encodes query and chunk separately, then compares vectors. Fast (vectors are precomputed). Accurate-ish. - A cross-encoder encodes query + chunk together through one transformer pass. Slow (must run for every query × chunk pair). Much more accurate — it sees both texts in context and can model how the chunk specifically answers the query.

You don't run a cross-encoder over your whole corpus (too expensive). You run it over the top-50 from first-stage retrieval. The reranker turns 50 noisy candidates into 5 precise ones.

The 2026 frontier of rerankers: - Cohere rerank-v3 — strong baseline, multilingual. Hosted only. - Voyage rerank-2 — paired with voyage-3 embeddings, strong on technical content. - BGE rerank-large — open-weight, runs locally, surprisingly competitive. - Jina rerank-v2 — open-weight, multilingual, good price/performance.

The cost model: - Embedding: ~$0.10 per million tokens for first-stage indexing. - Reranker: ~$0.50–$2 per 1k searches (typical 50 candidates × ~500 tokens each). - Latency: 100–500 ms added to query time.

For most production systems, the reranker is the single highest-leverage upgrade — and the most underused. Teams obsess over embedding model choice and skip the reranker entirely. That's backwards. Pick a reasonable embedding model, then add a reranker.

The classic reranker win: the between-clusters failure from Week 5. Embedding retrieval pulls a mixed top-20 (chunks from two adjacent topics). The reranker reads each chunk with the query and discards the ones that don't actually answer it, surfacing the truly relevant chunks even if they're geometrically further from the query embedding.

4. Query rewriting and expansion

User queries are noisy. They're short, ambiguous, sometimes typo-laden, sometimes phrased in a way that doesn't match how documents are written. Embeddings are surprisingly robust to this — but only up to a point. The fix is to clean and expand the query before retrieval.

Query rewriting patterns that work:

• Expansion to specifics. A vague "help me deploy" gets rewritten into 3–5 concrete sub-queries: "deploy command CLI," "deployment configuration," "deployment failures," etc. Run retrieval against each, merge results.
• HyDE (Hypothetical Document Embeddings). Have the LLM hallucinate what an answer document would look like for the query, then embed that hallucination instead of the query. The hallucinated answer is closer in embedding space to the real answer than the query is. Counterintuitively effective.
• Step-back prompting. For specific questions, generate a more general version too. "How do I configure HNSW index parameters in pgvector?" → also retrieve for "vector index configuration." The general version pulls in context the specific version misses.
• Multi-query retrieval. Generate three semantic variants ("rephrase this query in three different ways"). Retrieve for each. Union and dedupe.

The cost: one extra LLM call before retrieval, ~50–200 tokens. Latency adds 200–800 ms. For chat applications, the rewriting can run in parallel with displaying the user's typing; users barely notice.

When query rewriting is most valuable: - Short or vague queries ("explain this," "how does X work"). - Conversational systems where queries reference earlier context ("what about the second one?"). - Multi-intent queries that should pull from multiple parts of the corpus. - Domain-mismatched queries (user phrasing differs from doc terminology — e.g., "how do I make a chart" vs. docs that say "visualization API").

When it's overkill: - Single-shot lookup queries with clear intent and corpus-matching terminology. - Latency-critical paths where 500 ms is a budget you don't have.

5. Contextual retrieval

A 2024 Anthropic post named a fundamental problem: chunks lose context when separated from their parent document. A paragraph that says "the price increases by 20% annually" is meaningless without knowing which product, which tier, which contract type. The chunk's embedding represents the words present, not the implicit context.

The fix: before embedding, prepend a short context summary to each chunk — generated by an LLM at index time, describing where the chunk lives in the larger document.

Original chunk:
  "The price increases by 20% annually."

Contextualized chunk (prepended at index time):
  "[This excerpt is from the Enterprise Plan section of the
  2026 pricing guide.] The price increases by 20% annually."

The contextualization happens once, at index time. Each chunk gets one extra LLM call (~$0.001 with current models). The embedding now captures the chunk plus its context, dramatically improving retrieval on docs where chunks would otherwise be ambiguous.

Anthropic's published numbers showed retrieval failure rate dropping ~50% on technical documentation with this technique. Not every corpus benefits equally — heavily structured docs (where each chunk is already self-contained) see smaller wins. Long-form prose and contract-style documents see big ones.

The cousin pattern: contextual BM25. Same idea, applied to the BM25 side of hybrid search. The added context gives BM25 more terms to match on. Often paired with contextual embedding for compounded gains.

The implementation gotcha: contextualization must be done with the LLM seeing the whole document for each chunk. That's expensive in tokens — for a long document, you'll re-send the full doc with each chunk for context. Use prompt caching (Week 2) to make this affordable. Without caching, the cost is prohibitive.

6. Multi-hop and agentic retrieval

Some questions can't be answered from a single retrieval. "Which customers signed contracts in Q3 that included the enterprise SLA but didn't include single sign-on?" requires retrieving customer lists, contract details, and SLA terms — and combining them. One vector search isn't going to do it.

The 2026 pattern: agentic retrieval, where the LLM orchestrates retrieval as a tool.

The flow: 1. LLM receives the user query. 2. LLM decides: do I have enough info? (Almost certainly no on first turn.) 3. LLM issues a search query — which can be a rewritten/decomposed version of the user's query. 4. Receives top-K chunks. Decides: is this enough? 5. If not, issues another search with a refined query informed by what it just learned. 6. Loop until the LLM has enough context, or hits a step budget. 7. Generate the final answer.

The same architecture that powers tool-using agents (Week 7) underlies multi-hop RAG. The "retrieval" step is just one tool the agent can call.

Cost: 3–10× a single-shot RAG call, in latency and tokens. Capability gain: answers questions a single retrieval simply can't. Use selectively, behind a quality classifier: simple queries go through one-shot RAG; complex queries trigger the agentic path.

The 2026 frontier: interleaved retrieval — retrieving while generating, not just before. The model writes a partial answer, decides it needs to verify a claim, retrieves a chunk, continues. Used in some research systems and increasingly in production code-assistance tools. Powerful when it works, hard to make reliable.

7. Filtering and metadata — the security and relevance backbone

A retrieval system that doesn't filter by metadata is broken in two ways: it returns irrelevant chunks (a query about Q3 pricing pulls Q2 pricing chunks) and it leaks data (a tenant's query pulls another tenant's documents).

Every chunk in your vector DB needs metadata: tenant ID, document type, date, owner, language, source URL. Every query needs filters: only my workspace, only docs from the last six months, only English, only public knowledge.

The two architectural patterns:

Pre-filter then search. Filter the candidate set first (using SQL or a metadata index), then do vector search over the smaller filtered set. Best when filters are highly selective.

Post-filter. Search the whole vector index, then filter the top-K results by metadata. Cheaper when filters aren't selective, but risks the entire top-K being filtered out (you searched for nothing useful).

Most modern vector DBs (pgvector, Qdrant, Weaviate) support both natively. The choice is mostly about which is faster on your filter selectivity.

The non-negotiable rule for multi-tenant systems: tenant ID is always a pre-filter, applied at the database layer, never at the application layer. A bug in application code that forgets to filter by tenant is a data-exfiltration incident; a bug in database-layer enforcement is a much rarer kind of incident.

Permissions and RBAC. Document-level permissions get expressed as metadata: who can read this. Query filters constrain to "documents the user can read." The complexity grows with the permission model — keeping a queryable permissions index in sync with the source-of-truth permissions system is a real engineering problem most teams underestimate.

8. Putting it together — the production RAG pipeline

The 2026 reference architecture for a serious production RAG system:

Index time: 1. Ingest documents from source systems. 2. Chunk with structure-aware boundaries (Week 5), with parent-child for context preservation. 3. Contextualize each chunk with an LLM-generated context summary (section 5). 4. Embed contextualized chunks with a strong model (voyage-3, text-embedding-3-large). 5. Build BM25 index over chunk text (section 2). 6. Store with full metadata: tenant, doc type, dates, permissions (section 7).

Query time: 1. Receive user query. 2. Rewrite the query into expanded variants if needed (section 4). 3. Apply metadata filters (tenant, permissions, date) at the DB layer (section 7). 4. Hybrid search: dense + BM25 in parallel, fused (section 2). Top-50. 5. Rerank the top-50 to top-5 with a cross-encoder (section 3). 6. Assemble prompt with retrieved chunks + parent-doc context where helpful. 7. Generate with a strong LLM, with citation generation (Week 5). 8. Log: query, retrieved chunks, answer, citations. For eval and debugging.

Latency budget for a production system: - Query rewriting: 200 ms (skipped on simple queries) - Hybrid search: 50–150 ms - Reranking: 200–500 ms - Generation: 1–4 s - Total: 1.5–5 s end-to-end

Cost per query (rough, at 2026 prices): - Query rewriting: ~$0.0005 - Embedding the query: ~$0.00001 - Vector search: free at small scale, ~$0.0001 at scale - Reranking: ~$0.001 - Generation: $0.005–$0.05 - Total: ~$0.01–$0.05 per query

This adds up. A system serving 100 RPS pays $25k–$125k/month in inference costs alone. The cost optimization techniques from Week 2 (caching, smaller models for sub-tasks) become essential at production scale.

Build this week

Pick at least two:

1. Add hybrid search to your Week 5 RAG. Implement BM25 alongside your dense retrieval (most vector DBs have built-in support, e.g., pgvector + tsvector). Compare recall@5 on your held-out eval set. Tune α. Document the wins (and the queries where pure dense was actually better).

2. Add a reranker. Pick Cohere rerank-v3 or Voyage rerank-2 (if hosted is OK) or BGE rerank-large (if you want it local). Run first-stage retrieval at K=50, rerank to K=5. Compare to baseline on your eval set. The recall@5 jump should be obvious; report it.

3. Implement query rewriting. Use Claude Haiku or GPT-5-mini to expand each user query into 3 specific variants. Retrieve for each, union and dedupe. Compare on a subset of eval queries that are short or vague — this is where the wins concentrate.

4. Contextual retrieval. Pick a corpus where chunks are short and lose context (e.g., a long contract or a book chapter). Implement contextualization: for each chunk, generate a 1–2 sentence context summary using the parent doc. Reindex. Measure recall@5 vs. baseline. Anthropic's published wins are real but corpus-dependent — see what you get on yours.

5. Build the production pipeline. Combine three or four of the above into a single pipeline. Measure each stage's contribution to recall@5 by ablating each. Write up the cost/latency budget and where each stage's added cost is justified.

Read this

• "Contextual Retrieval" (Anthropic, 2024). The original write-up of the contextualization pattern. Required reading.
• Cohere's reranker documentation and benchmarks. Best public benchmarks for cross-encoder rerankers in 2026.
• "BM25 Reloaded" — Doug Turnbull's writeups on hybrid search. Practical IR perspective from someone who's deployed this for decades.
• "From Local to Global: A Graph RAG Approach to Query-Focused Summarization" (Microsoft, 2024). The graph-RAG pattern for synthesis questions; another category of advanced RAG worth knowing about.
• Voyage AI evaluation reports. Best comparative data on rerankers across domains.
• Anyscale's "Building RAG-based LLM applications for production" series. Long, opinionated, very practical.

Interview prompts

1. Walk through a production-grade RAG architecture with all the bells and whistles. Where does each technique live in the pipeline?
2. Explain hybrid search. When does BM25 beat dense retrieval, and vice versa?
3. What's a reranker, why is it expensive, and why is it usually worth the cost? Walk through the typical first-stage K / final K parameters.
4. Your team is debating: bigger embedding model vs. add a reranker. They have $X to spend. How do you decide?
5. Describe contextual retrieval. What problem does it solve and what does it cost?
6. Walk through query rewriting. When is it most valuable and when is it overkill?
7. Multi-hop / agentic retrieval — what kinds of questions need it, and what's the cost vs. capability tradeoff?
8. Metadata filtering in multi-tenant RAG. What's the difference between pre-filter and post-filter, and which do you choose when?
9. RRF (reciprocal rank fusion) vs. linear score fusion — when do you use which?
10. Your RAG recall@10 is 92% but recall@3 is 64%. What are you adding next, and why?

What "done" looks like

By the end of this week you should be able to:

• Diagnose which advanced-RAG technique a system is missing based on its failure pattern.
• Add hybrid search, reranking, query rewriting, and contextual retrieval to a baseline system, measuring each.
• Make principled cost/latency tradeoffs across the retrieval pipeline.
• Choose between agentic retrieval and one-shot retrieval based on query class.
• Design metadata filtering for a multi-tenant system without security gaps.
• Argue convincingly for what a team should add next, given their current eval numbers.

If you can do those, you're ready for Week 7 — agents — where the same orchestration pattern that powers multi-hop retrieval expands into general-purpose tool use, and we build the frontier of what models can do beyond passive Q&A.