Streaming
Filed under
1 post
Streaming Responses Are a UX Decision, Not a Performance One
Streaming model responses is a user-experience choice about time to first token, not a speed fix. Sometimes it makes structured output and tool use worse.
Streaming model responses is a user-experience choice about time to first token, not a speed fix. Sometimes it makes structured output and tool use worse.