Streaming
Filed under
2 posts
One Door to the Models, Part 3: The Provider Abstraction and Streaming
Set stream to true and token counting becomes estimation. Open a WebSocket and load balancing stops existing. Part 3 maps guarantees to transports.
Streaming Responses Are a UX Decision, Not a Performance One
Streaming model responses is a user-experience choice about time to first token, not a speed fix. Sometimes it makes structured output and tool use worse.