When a RAG system gives a wrong or half-right answer, the model is usually not the reason. The chunking is. If the passage that holds the answer never makes it into the retrieved context, no model can answer from it, and no amount of prompt tuning changes that. Chunking decides what can be retrieved at all, which makes it the first thing to inspect and the last thing most teams look at.

Amazon Bedrock Knowledge Bases lets you pick a chunking strategy when you create a data source. That single choice quietly sets the ceiling on your retrieval quality. Get it wrong and you spend weeks blaming the embedding model, the reranker, or the LLM for a problem that lives in how you cut the documents.

Why chunking is the load-bearing decision

Retrieval works on chunks, not documents. Each chunk is embedded into a vector, and at query time you fetch the top-k chunks nearest to the question. Two failure shapes follow directly from chunk size.

Chunks that are too large dilute the embedding. A 2000-token chunk covering four subtopics produces one vector that is the average of all four, so it matches everything weakly and nothing strongly. The right chunk gets buried under near-misses. Chunks that are too small sever context. A definition split from its example, or a step split from its warning, retrieves cleanly but arrives without the surrounding text that made it useful. The answer is technically present and practically incomplete.

The job of a chunking strategy is to cut on meaningful boundaries so each chunk is one coherent idea, self-contained enough to answer from and specific enough to rank.

The three strategies Bedrock gives you

Fixed-size chunking

Split every N tokens with some overlap. It is the default and the cheapest, and it is fine for uniform, prose-heavy content where topic shifts are gradual. It is actively bad for structured documents, because it cuts wherever the token counter lands: mid-table, mid-list, between a heading and the paragraph it introduces. Overlap softens the damage but does not remove it. Reach for fixed-size when your corpus is homogeneous and you want a baseline, not because it is the default.

Semantic chunking

Split on meaning. Semantic chunking measures the embedding similarity between adjacent sentences and starts a new chunk where the topic shifts, so boundaries fall between ideas instead of at a fixed token count. This is the right default for mixed content: FAQs, knowledge articles, mixed prose where each answer or concept wants to stay whole. It costs more to build the index because it embeds while it chunks, but the retrieval quality difference on heterogeneous corpora is the reason to pay it.

Hierarchical chunking

Build parent and child chunks. Small child chunks are embedded and matched for precise retrieval, but the larger parent chunk is what gets returned to the model, so you rank on specificity and answer with context. This fits documents with real structure, technical manuals, legal contracts, anything with sections and subsections, where a query hits a narrow clause but the model needs the surrounding section to use it correctly. It is the most involved to reason about and the best fit when your documents have a genuine hierarchy to exploit.

Evaluate retrieval before you blame the model

The single most useful habit is to separate the retrieval question from the generation question. Before you touch the prompt or swap the model, ask one thing: for a set of real questions, did the chunk containing the answer show up in the retrieved context?

  • Build a small evaluation set: 30 to 50 real questions, each with the source passage that answers it.
  • Run retrieval only. For each question, check whether the correct passage appears in the top-k results. That hit rate is your retrieval ceiling.
  • If the correct passage is not being retrieved, generation quality is irrelevant. The fix is chunking, embeddings, or top-k, not the model.
  • Only once retrieval reliably surfaces the right passage does it make sense to work on how the model uses it.

Most teams skip this and jump straight to prompt engineering, which is why they spend so long on a problem that a retrieval hit-rate measurement would have located in an afternoon. If retrieval is missing the answer half the time, you have a chunking problem wearing a model costume.

A practical starting point

Start with semantic chunking for general knowledge content, move to hierarchical when your documents have clear section structure and answers depend on surrounding context, and keep fixed-size only for large, uniform prose where you want speed and simplicity. Then measure retrieval hit rate, change one variable, and measure again. Chunking is not a set-and-forget config value. It is the knob with the most influence over whether your RAG system is any good.

The takeaway

RAG quality is decided before the model ever runs, at the moment you cut your documents into chunks. Fixed-size is a baseline, semantic is the sensible default for mixed content, and hierarchical wins when structure matters. Measure retrieval hit rate on a real question set first, because if the answer never enters the context, the model was never the problem.

Read this next

For the storage and infrastructure side of running vector search at scale, the cloud field notes are at ercan.cloud, and the hub is at ercanermis.com.