Interactive · Week 5 · Section 4

Five chunking strategies, same document.

A real production decision: how do you split documents into retrievable chunks? The same source doc, the same query, but five different chunking strategies produce five different "best" retrieved chunks. Pick a strategy and a query, and watch the top-retrieved chunk shift in the document — and the similarity score change. The fragmentation and dilution failure modes are visible in the chunks themselves.

Chunk size 200 chars

Overlap 40 chars

Query

Chunks

—

total produced

Mean length

—

chars per chunk

Top-1 score

—

retrieval similarity

Answer captured

—

in the top-1 chunk

Source document · with chunk boundaries

retrieved chunk highlighted in accent

Chunks ranked by similarity

click any chunk to expand

What's happening here

Similarity is computed via stop-word-removed token overlap with length penalty: matches/q_len ÷ √(chunk_tokens/30). This rewards medium-length chunks where most query terms appear and penalizes very long chunks where the relevant content gets diluted by surrounding noise. Real systems use cosine similarity on dense embeddings, but the failure modes — fragmentation, dilution, boundary cuts — manifest the same way. Source-of-truth lives as <ChunkingStrategyComparator/> in the curriculum codebase.