Engineering
Designing a latency budget for voice agents
Conversational AI lives or dies on turn-taking. Here is how we allocate a sub-second response budget across the stack, and what we cut when it does not close.
In text, a two-second wait reads as thinking. In speech, it reads as a dropped call. That difference drives nearly every architectural decision in a voice system.
The budget
We start from a target of roughly eight hundred milliseconds between the caller finishing a sentence and the first syllable of the reply, then allocate:
| Stage | Budget |
|---|---|
| Endpointing | 150 ms |
| Transcription (streaming) | 100 ms |
| Retrieval | 120 ms |
| First token from the model | 300 ms |
| Speech synthesis, first chunk | 130 ms |
Everything streams. Nothing waits for a complete result before starting the next stage — the reply begins synthesising while the model is still generating it.
What we cut first
When the budget does not close, the order of sacrifice is: retrieval breadth, then model size, then scope. We do not sacrifice the endpointing window, because cutting a caller off mid-sentence is worse than any answer quality gain it buys.
Measure at the ear, not at the server
Server-side timings will tell you a comfortable story. Instrument the round trip from the caller's perspective, over the actual telephony path, on the worst connection you support. That number is the only one that matters.