A few hundred milliseconds, not a few seconds
Human conversation has its own timing built in from a lifetime of talking to other people. In natural spoken exchange, the gap between one person finishing a sentence and the other person starting to respond is typically small, often well under a second. When a response takes noticeably longer than that, listeners don't just passively wait, they read the delay as meaningful: confusion, disinterest, a dropped call, a broken system. A text chat interface doesn't carry this baggage. A user watching a 'thinking' indicator for a few seconds while a complex answer gets generated is a completely normal, even expected, experience. That same few seconds of silence in a voice call feels broken.
This difference in tolerance is the reason latency gets treated as THE headline constraint for voice AI in a way it simply isn't for text agents. A text agent can spend a couple of extra seconds doing careful multi-step reasoning, calling tools, or double-checking an answer, and most users will tolerate it, especially if there's a visible indicator that work is happening. A voice agent doing the same multi-step reasoning has to either finish it inside a much tighter window or find some other way to keep the conversation feeling alive while it works, because dead air reads as failure far faster than a spinning loader does.