Semantic chunking splits text on embedding-derived meaning boundaries instead of fixed character counts. It often improves retrieval on narrative and heterogeneous corpora, but it is no free win - here is when it helps, when it is a wash, and how to implement it.
Most RAG quality complaints get blamed on the model. The answers are vague, they miss the point, they hallucinate facts that were never in the source. Teams reach for a bigger model or a longer context window. Often the real problem sits one stage earlier, in the chunker.
A retriever never sees your documents. It sees chunks: the slices you cut a document into before embedding. If those slices fragment a single idea across two chunks, or staple two unrelated ideas into one, retrieval quality is capped before the LLM ever runs. Semantic chunking splits text on embedding-derived meaning boundaries rather than character or token counts, producing variable-length chunks that line up with topic shifts. It often improves retrieval on narrative and heterogeneous corpora. On short, well-structured documents it is frequently a wash, and sometimes the extra compute is not worth it.
Why Chunking Caps Retrieval Quality
A chunk is the atomic unit of context an LLM receives in a RAG pipeline. The embedding model turns each chunk into a single vector, and that vector has to summarize everything in the chunk. When a chunk holds one coherent idea, its embedding is a clean signal. When it spans two ideas, the embedding becomes a blurred average that matches neither query well.
Bad chunk boundaries produce three recurring failure modes:
- Topic drift. A chunk spans two unrelated subjects, so its embedding sits in the semantic gap between them and ranks poorly for both. The relevant passage is in your index, but the retriever cannot surface it.
- Context starvation. A chunk gets cut mid-thought. The retriever finds it, but the LLM lacks the surrounding sentences needed to answer faithfully, so it guesses.
- Redundant retrieval. Heavy overlap means several near-identical chunks claim multiple top-k slots, crowding out passages that carry the rest of the answer.
The classic symptom: a user asks something specific, the retriever returns five chunks that each contain part of the answer, and none contains the whole thing. A larger model helps synthesize fragments, but it cannot recover information that fragmentation pushed out of the top-k entirely. Faithfulness and context-recall scores degrade with poor boundaries regardless of how capable the generator is. The retrieval stage is the ceiling, which is why it pays to get chunking right before tuning anything downstream in the RAG pipeline.
From Fixed-Size to Semantic Splitting
Fixed-size splitting cuts every N characters or tokens with a fixed overlap. It is fast and trivial, and it ignores sentences, paragraphs, and topic shifts entirely. The defaults people copy around (LangChain's chunk_size=1000, chunk_overlap=200 characters, most often) get applied without evaluation and cut straight through the middle of arguments.
Recursive character splitting is the common upgrade. LangChain's RecursiveCharacterTextSplitter walks a hierarchy of separators (\n\n, then \n, then . , then ) and falls back to raw character count only when a segment is still too long. It respects paragraph and sentence boundaries when the formatting cooperates. What it cannot do is notice a topic shift inside a long, well-punctuated paragraph. The split is structural, not semantic, and that is the gap semantic chunking targets: embed each sentence or small group of sentences, measure the cosine distance between adjacent embeddings, and insert a chunk boundary wherever that distance spikes. A large jump signals a topic shift, so the resulting chunks vary in length and align with ideas rather than character budgets.
One point of terminology that trips people up: semantic chunking is not semantic search. Semantic chunking is an ingest-time splitting decision. Semantic search is query-time retrieval over dense vectors. They both lean on embeddings, but they operate at different stages and solve different problems. (For the retrieval side, see how dense and sparse vectors actually differ.)
The mental model worth borrowing here is Greg Kamradt's 5 Levels of Text Splitting: character count, recursive, document-specific, semantic (embedding-based), and agentic (LLM-driven). Each level trades compute for semantic fidelity. You climb it until the gains stop paying for the cost on your corpus, not until you reach the top.
When Semantic Chunking Is Worth It (and When It Is Not)
Here is the part the hype usually skips. Chroma's chunking evaluation study found that semantic chunking does not reliably beat simpler methods. A plain RecursiveCharacterTextSplitter at 200 tokens with zero overlap was one of the top performers (88.1% recall with text-embedding-3-large). Chroma's own reimplementation of the Kamradt semantic chunker, run at default settings, came in at 83.6%; adding a maximum chunk length lifted it to 87.1%. The gap between the best and worst strategy in their tables was roughly 8 points of recall. The paper Is Semantic Chunking Worth the Computational Cost? reaches a similar verdict: across document retrieval, evidence retrieval, and answer generation, the extra compute is not justified by consistent gains.
Semantic chunking often helps on long narrative documents, transcripts, and heterogeneous corpora where structural cues are unreliable. It is frequently a wash on short, well-structured content, and on a cost-adjusted basis it can lose on huge corpora with tight ingest budgets. Treat it as a tool to reach for when boundaries matter, and benchmark it against a recursive splitter on your own data before committing.
Benchmarking does not need to be elaborate. Build a golden set of 50-200 real questions with the passage that answers each one marked. Run every chunking configuration through the same embedding model and the same retriever, then measure context recall (did the answering passage land in the top-k?) and, as Chroma does, token-level IoU between retrieved and relevant text, which penalizes chunks that drag in a lot of unrelated material. Hold the embedding model constant across runs; changing it shifts every result and makes the chunking comparison meaningless.
RAG Chunking Strategies Compared at a Glance
No single strategy dominates. Pick based on document shape and your ingest budget.
| Strategy | Ingest cost | Effort | Best fit |
|---|---|---|---|
| Fixed-size | Negligible | Trivial | Short, uniform docs; throwaway prototypes |
| Recursive character | Negligible | Low | Markdown/HTML with clear structure; default starting point |
| Semantic (embedding breakpoints) | One embedding call per sentence group | Medium | Long narrative docs, transcripts, mixed-format corpora |
| LLM-based / agentic | Very high (LLM per segment) | High | High-value, low-volume: legal, medical, compliance |
| Late chunking | One long-context embed pass | Medium | Docs that fit the model's window (8K tokens for jina-embeddings-v2) |
| Hierarchical (small-to-big) | Adds a parent index | Medium | When small chunks retrieve well but answers need wider context |
A rough decision tree:
- Short structured docs (under ~500 tokens, clear headers): recursive or structure-aware splitting. Semantic chunking buys little.
- Long narrative docs with no headers: semantic chunking.
- Mixed-format docs (tables, prose, code): structure-aware pre-split, then semantic refinement on the prose.
- Real-time ingest under tight latency SLAs: recursive, since it makes no embedding calls.
On that last point: semantic chunking makes one embedding call per sentence or sentence group. Fixed-size and recursive splitting make none. On a 10-million-document corpus that is tens of millions of extra embedding calls at ingest, and in practice several times slower than recursive splitting. The cost is paid once and is invisible at query time, but on very large corpora with tight ingest windows it can tip the decision back to recursive.
How Embedding Breakpoints Work in Practice
The two dominant libraries expose the same core idea with slightly different knobs.
LangChain's SemanticChunker (in langchain_experimental.text_splitter) supports four breakpoint_threshold_type values: percentile (the default; break where the distance exceeds the Nth percentile), standard_deviation (break beyond mean plus k sigma), interquartile (a more outlier-robust variant), and gradient (break where the rate of change in distance spikes, useful for dense domain text where absolute distances stay compressed). Percentile is the safest place to start.
from langchain_experimental.text_splitter import SemanticChunker
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings
chunker = SemanticChunker(
OpenAIEmbeddings(),
breakpoint_threshold_type="percentile", # or standard_deviation, interquartile, gradient
breakpoint_threshold_amount=95,
)
docs = chunker.create_documents([long_text])
# SemanticChunker has no max size parameter, so enforce the ceiling afterwards
capper = RecursiveCharacterTextSplitter(chunk_size=2000, chunk_overlap=0)
docs = capper.split_documents(docs)
print(len(docs), sum(len(d.page_content) for d in docs) / len(docs))
LlamaIndex's SemanticSplitterNodeParser exposes buffer_size (how many sentences on each side are pooled into one embedding before comparison; larger smooths noise) and breakpoint_percentile_threshold (the percentile cutoff; 95 means only the top 5% of distances trigger a split). Those two parameters are your main lever on granularity.
Two things to know before you commit. First, the embedding model used for chunking does not have to match the one used for retrieval. A lightweight model like all-MiniLM-L6-v2 is plenty for boundary detection, and heavier models add ingest latency for marginal gains. What matters is consistency: use the same chunker model across re-indexes, because changing it shifts every boundary. Second, always set a hard max-chunk-length ceiling. A long stretch of uniform text can produce one runaway chunk that blows past your embedding model's token limit. LangChain's SemanticChunker only exposes min_chunk_size (source), which is why the snippet above runs the output through a RecursiveCharacterTextSplitter. Chroma's result, where a length cap was the difference between the semantic chunker underperforming and competing, is the same lesson from the other direction.
Beyond Splitting: Two Adjacent Techniques
Two other techniques attack the same fragmentation problem from different angles:
- Anthropic Contextual Retrieval prepends an LLM-generated, document-aware summary to each chunk before indexing it. It is enrichment, not splitting, and it directly fights context starvation. Anthropic reports failed retrievals dropping 35% with contextual embeddings, 49% when combined with BM25, and 67% with reranking added on top (Anthropic). The cost is one LLM call per chunk at ingest.
- Late chunking (Jina AI, arXiv 2409.04701) inverts the order: embed the whole document first with a long-context model, then segment the token-level embeddings into chunks. Each chunk vector keeps awareness of the full document. It fits documents that sit inside the model's window (jina-embeddings-v2 handles roughly 8,192 tokens).
If your failure mode is context starvation rather than topic drift, evaluate these before spending more time on breakpoint thresholds. Whatever you land on, version the chunking config (model, threshold, buffer, cap) alongside the index, and re-run the golden set whenever the corpus changes shape.
