Processing partial input instead of waiting for complete input
The core idea behind streaming is simple: instead of waiting for a complete unit of input, a full utterance, a full sentence, a full audio clip, before doing anything with it, a streaming stage starts processing as soon as any usable piece arrives and keeps updating as more arrives. Applied to voice, this means the ASR stage emits partial transcripts as the user is still speaking, rather than staying silent until the user stops. Those partial transcripts update and correct themselves as more audio comes in, but downstream stages can start reacting to them long before the user has finished their sentence.
The payoff is that streaming turns a chain of sequential waits into a chain of overlapping work. If the LLM has to wait for a fully finalized transcript before it starts generating anything, and TTS has to wait for a fully finished LLM response before it starts synthesizing anything, the pipeline's total latency is close to the sum of each stage's full processing time. If each stage can instead start working on partial output from the stage before it, those processing times overlap in wall-clock time instead of stacking, which is often the single biggest lever available for cutting end-to-end latency without touching model quality at all.