Streaming does not make your model faster. It makes the wait feel shorter. The total time to generate a response is almost identical whether you stream it or not. What streaming changes is when the user sees the first token, and that single number, time to first token, drives the entire perception of speed. Treat streaming as a UX decision, because that is what it is, and you will make better calls about when to use it and when it actively hurts.

The reflex is to reach for InvokeModelWithResponseStream or ConverseStream because streaming feels like the performant choice. But the model produces the same tokens in the same total time either way. You are not optimizing throughput. You are deciding whether a human stares at a blank screen for six seconds or watches text appear after three hundred milliseconds.

Why time to first token is the number that matters

A non-streamed response has one latency the user feels: the whole generation, start to finish. Ask for three paragraphs and the user waits for all of it before anything appears. That wait reads as the system being slow, even when the total time is fine.

Streaming splits that into two numbers. Time to first token is how long until something appears. Then tokens arrive continuously, and because people read slower than a model generates, the text tends to stay ahead of the reader. The perceived experience is "instant and flowing" even though total generation time did not move. This is the same psychology as a progress bar: motion reads as progress, and a blank screen reads as a hang. For anything conversational, where a person is reading the output as prose, streaming is close to mandatory for that reason alone.

Where streaming quietly makes things worse

Streaming is a default, not a law. Several common cases are better served by waiting for the complete response.

Structured output the client has to parse

If the response is JSON your frontend deserializes, a partial stream is useless or harmful. You cannot parse half a JSON object, and showing the user a half-built structure is worse than showing them a spinner. When the consumer of the output is code, not a reader, wait for the whole thing and parse it once. Streaming buys you nothing and adds the risk of rendering malformed partial state.

Tool use inside an agent

When a model streams a decision to call a tool, there is nothing for the user to read: the "output" is a tool invocation, not prose. Streaming the tokens of a function call to the UI shows the user machinery they did not ask to see. The right pattern is to stream the final, user-facing answer and keep the intermediate tool-call reasoning off the screen, or represent it as a status like "checking your order," not as raw streamed tokens.

Short responses

If the answer is one sentence, streaming adds protocol complexity for no perceptible gain. The whole thing arrives in a few hundred milliseconds either way. Save the streaming plumbing for responses long enough that the wait would otherwise be felt.

The cost you take on when you stream

Streaming is not free engineering. You hold a connection open for the duration, which changes how you think about timeouts, retries, and load balancing. Error handling gets harder: a failure halfway through a stream leaves the user with a truncated answer you have to detect and recover from, versus a clean all-or-nothing failure on a buffered call. And you cannot run a Guardrails output check or a JSON validation on a response you have already shown the user token by token. If validation matters, either buffer, or validate as you stream and be ready to retract, which is real work. Weigh that against the UX gain rather than assuming streaming is strictly better.

Lower the number instead, sometimes

If the problem is genuinely that first token takes too long, streaming hides it but does not fix it. Prompt caching does: caching the stable prefix of a long prompt cuts the time before generation starts, which lowers time to first token directly. On a heavily-prompted agent, trimming and caching the prompt can do more for perceived speed than streaming does, and it helps the non-streamed and structured cases too.

The takeaway

Streaming does not change how fast your model runs. It changes time to first token, which changes how fast the system feels, so it is a UX decision. Stream conversational prose where a human is reading. Do not stream structured output a client must parse, raw tool-call reasoning, or one-line answers. And when first-token latency is the real complaint, cache the prompt to lower the number rather than streaming to disguise it.

Read this next

For the infrastructure behind long-lived connections and latency at the edge, the cloud field notes are at ercan.cloud, and the hub is at ercanermis.com.