Agentic RAG replaces a single retrieval with a loop: the model retrieves, reads, decides it needs more, retrieves again, and repeats until it is satisfied. Each hop is a full model round trip plus a search, and the hops are sequential because each one depends on the last. For most questions, this buys a marginally better answer at several times the latency, when one well-constructed query would have returned the same context in a single pass. Multi-hop retrieval is a real tool for a narrow class of questions, and a default that quietly triples your response time everywhere else.

The reframing: agentic retrieval trades latency for the ability to decompose a question at runtime. That trade is worth it only when the question actually needs decomposing. Most do not, and paying the loop tax on questions that a single query answers is the most common way RAG systems get slow for no gain.

Where the latency comes from

Single-pass RAG is one embedding, one vector search, one model call. Agentic RAG is a loop, and the loop is serial by construction: the model cannot issue the second query until it has read the results of the first. So the wall-clock cost is not one retrieval plus one generation, it is the sum of every hop's search, every hop's model call to decide what to do next, and the final generation. Three hops is roughly three times the round trips, and because each model call also re-reads accumulated context, the per-hop cost tends to grow rather than stay flat.

None of this is hidden work the user asked for. From their side it is one question that now takes eight seconds instead of two. If those eight seconds did not buy a materially better answer, you spent your latency budget on nothing.

One good query beats a clever loop, usually

A surprising amount of agentic RAG exists to compensate for a weak first query. If the initial retrieval pulls the wrong chunks, the loop's job becomes "notice the context is bad and go get better context," which is expensive error recovery. Fix the query and the loop often has nothing left to do.

Before reaching for multi-hop, spend the effort on the single pass:

  • Rewrite the query. One cheap model call to turn a messy user question into a clean search query improves retrieval more than a second hop does, at a fraction of the latency.
  • Retrieve more, then rerank. Pull a wider candidate set in one search and rerank it down, instead of doing several narrow searches in sequence.
  • Fix the chunks. Most bad retrieval is a chunking problem. Better chunks make the first query land, and the loop becomes unnecessary.

These are all single-pass improvements. They raise answer quality without adding a serial round trip, which is the opposite of what the loop does.

When agentic retrieval actually wins

The loop earns its latency for questions that genuinely cannot be answered by one query, because the second query depends on the answer to the first. The clearest case is multi-hop reasoning over facts you have to chain: "which of our vendors in the region affected by the new regulation are up for renewal this quarter" is really three lookups where each narrows the next, and no single retrieval holds the joined answer.

The pattern to look for is a genuine dependency between retrievals: you cannot write the second query until you know the result of the first. When that dependency is real, the loop is doing necessary work and the latency is the price of an answer you could not otherwise get. When it is absent, and it usually is, the loop is elaborate machinery for a question a single query would have closed.

Decide per question, not per system

The failure is making agentic retrieval the default for every request. A better design routes: cheap questions take the single pass, and only questions that show a decomposition signal, multiple entities, a chained condition, an explicit "compare across," escalate to the loop. A small classifier or even a heuristic in front of retrieval can make that call. The goal is to pay the multi-hop tax exactly on the questions that repay it, and never on the ones that do not.

The takeaway

Agentic RAG is not wrong, it is over-applied. It turns retrieval into a serial loop of model round trips, and for the large majority of questions that a single good query answers, that loop is pure latency. Spend your effort first on the single pass: rewrite the query, retrieve wider and rerank, and fix the chunks. Reserve multi-hop for questions with a real dependency between retrievals, and route to it per question instead of defaulting the whole system into it. The fastest RAG system is the one that only loops when the question actually requires a second look.

Read this next

For the infrastructure behind low-latency retrieval at scale, the cloud field notes live at ercan.cloud, and the hub is at ercanermis.com.