Three models, one conversation, one clock
A real-time voice agent is not one model, it's a relay race. Audio comes in, a speech-to-text system turns it into text, an LLM decides what to say, and a text-to-speech system turns that decision back into audio the user can hear. Each of those three stages was likely designed and benchmarked in isolation, on its own task, with its own latency profile. The moment you chain them together for a live conversation, none of that isolated performance matters as much as the sum. The user doesn't experience 'the ASR was fast,' they experience the total elapsed time between the last word they spoke and the first word they hear back.
This chaining also means the stages aren't independent in practice, even though they're architecturally separate. The LLM can't start reasoning about a request until ASR has produced usable text. TTS can't start producing audio until the LLM has produced usable text. If any single stage in the middle stalls, buffers, or waits for a clean boundary before handing off, that stall is inherited by everything downstream. A text chat interface never has this problem because there's only one stage the user is waiting on: the model's generation. Voice adds two more stages on either side of it, and both of them cost real time.