A large context window is a capacity limit, not a retrieval strategy. The fact that a model accepts several hundred thousand tokens does not mean it reads them with even attention, and it certainly does not mean you should pay to send them. Every long-context benchmark that has bothered to measure position and length says the same thing: accuracy falls as input grows, and evidence buried in the middle of a prompt gets used least.

The reframing worth internalising: the context window is a budget you spend, not a database you query. Teams that treat it as storage build systems that are slower, more expensive, and less accurate than the retrieval pipeline they skipped.

The two failure modes are different

These get conflated, and they have different fixes.

  • Positional degradation. The lost-in-the-middle effect from Liu et al. (2023): accuracy is highest when the relevant span sits at the very start or the very end of the input and sags in the middle, producing a U-shaped curve. Move the same fact from the edge to the centre and answer quality drops even though nothing else changed.
  • Length degradation. Accuracy declines as the input grows even when the evidence is fixed and favourably placed. Follow-up work like RULER (Hsieh et al., 2024) tested long-context models on multi-needle and reasoning-over-context tasks and found effective context length is consistently shorter than the advertised number.

Positional degradation you can sometimes engineer around by reordering. Length degradation you cannot, because the problem is the size of the haystack itself. Fixing it means putting less in.

Why the benchmark number misleads

The needle-in-a-haystack test earned its popularity by being easy to run and easy to pass. Hide one out-of-place sentence at various depths in a long document, ask the model to repeat it, plot a green grid. A model can score near-perfect on that and still fail the thing you actually need, because real questions are not lexical lookups.

Your users ask things that require finding four related facts scattered across a corpus, noticing that two of them conflict, and reasoning over the rest. That is the task RULER measures, and it is where the advertised window and the usable window diverge. Treat single-needle recall as a smoke test, not as evidence the window works.

The cost side is not subtle

Retrieval degradation is arguable. The bill is not. Input tokens are billed per call, so a design that stuffs 200,000 tokens of context into every request pays for all 200,000 every time, whether the answer needed twelve of them or none.

# Stuff-the-window: whole handbook, every turn
200,000 input tokens x 50,000 calls/month = 10,000,000,000 tokens

# Retrieve-then-answer: system prompt + 6 chunks
  3,000 input tokens x 50,000 calls/month =    150,000,000 tokens

That is a factor of sixty-six on the input line, before you count the latency. Time to first token scales with how much the model has to read, so the stuffed prompt is also the slow one. You are paying more for a worse answer, delivered later. There is no axis on which that wins.

Prompt caching complicates the arithmetic in one specific case: a genuinely stable prefix that repeats across calls can be read at a steep discount. That is real, and it is worth using. But it rescues the cost of a static preamble, not the recall of a bloated haystack. A cached bad prompt is a cheaper bad prompt.

What to do instead

The discipline is unglamorous and it works.

  • Retrieve, then answer. Put a retrieval step in front of the model and pass the top handful of chunks. A Bedrock Knowledge Base does this without you owning an index. Six good chunks beat six hundred mediocre ones, every time.
  • Rank, and mean it. If you pass ten chunks, the order matters because position matters. Put the strongest evidence at the top and the next strongest at the bottom, where attention is best. Fill the middle with the marginal material, not the load-bearing fact.
  • Cap what any one step can see. A hard token ceiling per call forces the retrieval layer to be selective. Without a cap, context grows until something breaks, usually at the worst moment.
  • Cache the stable prefix, not the variable body. System prompt and tool schemas are the same every call and belong in the cache. Retrieved evidence changes per question and does not.
  • Measure recall on your own questions. Run your real queries against your real corpus and score the answers. Vendor context-length claims are a marketing number; your eval is the only one that describes your system.

When the big window is the right answer

There are cases where filling the window is correct, and refusing to on principle is its own mistake. Summarising a single document end to end needs the whole document; there is nothing to retrieve because everything is relevant. Reasoning that requires holding an entire codebase or contract in view at once genuinely needs the span. One-off analysis where engineering a retrieval pipeline costs more than the tokens is a fine place to be lazy on purpose.

The test is whether you can name the subset that matters. If you can, retrieve it. If you genuinely cannot, because the task is over the whole corpus rather than a part of it, then the window is doing real work and you should use it.

The takeaway

Bigger context windows are a real engineering achievement, and they keep getting bigger. They are also the most expensive way to avoid writing a retrieval step. Recall degrades with length and with position, the bill scales with every token you send, and latency follows the prompt size. Decide what the model needs to see, send that, and stop mistaking capacity for competence.

Read this next

For the infrastructure side of running retrieval at scale, the cloud and platform field notes live at ercan.cloud, and the hub is at ercanermis.com.