Knowing when someone is actually done talking
A silent gap in someone's speech is ambiguous. It might mean they've finished their thought and are waiting for a reply, or it might mean they're pausing mid-sentence to think, breathe, or recall a word. Humans resolve this ambiguity constantly and mostly unconsciously, using cues like intonation, sentence structure, and context to guess when a turn has actually ended. A voice agent has to make the same judgment call automatically, using signals like a pause of a certain length combined with sentence-final acoustic cues, and it has no equivalent of the shared social context a human listener brings to the guess.
Voice activity detection (VAD) is the underlying mechanism that flags whether audio currently contains speech or silence, and it's the first, cheapest signal a system uses to decide a turn may be ending. But VAD alone is a blunt instrument: it can't tell 'I'm done' apart from 'um, let me think.' Getting turn-taking right generally means combining VAD with some notion of end-of-utterance detection that looks at longer context, sometimes using the ASR transcript itself, or a lightweight classifier, to decide whether what's been said so far sounds complete. Get this wrong in one direction and the agent talks over the user; get it wrong in the other and it leaves long, awkward silences waiting for a sentence that already ended.