Latency budgets and turn-taking in voice agents.

Voice is unforgiving

Voice is unforgiving. A text agent that takes three seconds to reply feels thoughtful; a voice agent that takes three seconds to reply feels broken.

The user has no visual feedback, no typing indicator, no progress bar. They have silence — and silence, past about eight hundred milliseconds, reads as a system that has failed.

Budget the turn end-to-end

Assign milliseconds to each stage of the turn and defend them like a performance budget on the web. Anything over the budget needs a spoken filler or a redesign — never a stall.

• Capture and VAD — 60 ms.
• ASR streaming to first token — 180 ms.
• Orchestration and routing — 40 ms.
• Tool calls (parallel) — 250 ms.
• TTS to first audio frame — 220 ms.
• Total wall-clock target — under 800 ms p50, under 1.4 s p95.

Barge-in is a requirement

Barge-in is not a feature, it is a requirement. Users will interrupt. Design the agent to yield the floor, cancel in-flight generation, and resume from a coherent state.

The failure mode to avoid: the agent hears the interruption, keeps talking anyway, and now two voices are competing. Users hang up.

Silence past eight hundred milliseconds reads as failure.

Tool calls during a turn

Tool calls during a turn are the hardest part. Stream a spoken acknowledgement while the tool runs. If the tool exceeds its budget, hand off to a human — do not stall.

The acknowledgement matters. "Give me a second while I pull that up" buys you two seconds of grace. Dead air buys you a hangup.

Measure every turn

Every turn. Ninety-fifth percentile latency, barge-in success rate, and handoff quality are the numbers that decide whether the agent is usable at all.

If you are not looking at those three numbers on a dashboard every morning, you are shipping a demo — not a service.