800 milliseconds is the point where latency can separate a conversational voice agent from one that feels like a survey with hold music. Our testing and research on human dialogue keep pointing to that figure. This article covers why it matters, where delay occurs in a voice AI pipeline, and what we changed to remain below it.
Why the 800ms Mark Matters
Human dialogue runs on a narrow timing budget. In natural conversation, the pause between one speaker ending and the next beginning is usually 200-500ms. Longer gaps can signal hesitation, calculation, or emotional processing. Once a pause reaches 700-900ms, listeners start hearing no response rather than thought. They may ask, "Hello? Are you there?" or hang up.
The 800ms figure is not a strict point where everything below works and everything above breaks. Think of it as a range. Under 600ms, pauses usually go unnoticed. From 600ms to 800ms, they are apparent but tolerable. Past 800ms, they begin hurting the caller experience, with the effect building over multiple turns. After waiting 900ms three or four times, a caller may feel the call is broken, regardless of response quality.
Outbound sales calls tend to fail through early disconnects. Inbound support calls tend to lose trust in the agent's ability. Either way, callers rarely think, "the AI is processing my request." They think, "something is wrong with this system."
Inside the Pipeline
A voice agent has four sequential sources of delay. Reaching sub-800ms means knowing each source and assigning it a realistic budget.
First comes end-of-speech detection. The pipeline must know the caller has stopped before replying. Acoustic silence alone is quick but unreliable, since mid-sentence pauses can trigger false replies. More advanced models combine silence, pitch movement, and semantic completion to identify a finished turn. They are more accurate, but add 50-150ms to detection.
Second is ASR transcription delay. Streaming ASR supplies partial text while the caller speaks, letting the pipeline begin work before end-of-speech detection fires. Final confirmation usually takes another 100-200ms after that signal, since the complete utterance helps resolve context-dependent words. With a constrained vocabulary, common in support, streaming ASR can bring this close to zero by detection time.
Third is LLM response generation, the biggest and least predictable share of total delay. Generating a full 50-100 word answer can take 800-1500ms, enough to consume the entire budget. Instead, TTS should begin before the LLM finishes. Its first word should play within 300-400ms of receiving the prompt. That calls for an architecture that streams tokens into a TTS queue rather than waiting for a complete answer.
Fourth is TTS synthesis and audio delivery. Current TTS systems produce initial phonemes at near-real-time speed, while audio travels through the call path in parallel. Callers perceive the reply when the first audio arrives, not when generation ends. For voice pipelines, TTS should therefore target time-to-first-audio, not time-to-full-audio.
Our Starting Point
Our first internal Ringg build measured about 1,400ms end-to-end on a typical turn. That result fit an early design: every stage ran synchronously, and LLM calls waited for a complete answer before sending it to TTS.
The approximate breakdown was end-of-speech detection 180ms, ASR final transcription 200ms, LLM generation 750ms, TTS startup 200ms, and audio delivery to the telephony path 70ms. Together, these total 1,400ms, matching our measurement.
Moving from 1,400ms to below 800ms required work at every stage. The largest gains came from LLM streaming and replacing sequential execution with a pipeline.
Changes We Made
First, we streamed LLM output into TTS instead of waiting for the full answer. The response module had to pass tokens onward as they appeared rather than store the entire response. The main issue was that TTS needs complete sentences for natural prosody, while individual tokens sound choppy. We added a lightweight sentence-boundary detector to group the stream into complete chunks before TTS. This cost 30-60ms in buffering, but produced acceptable speech.
Second, we ran ASR finalization and LLM prefill in parallel. When ASR returns its final text, the LLM may already have processed the streaming transcript and be ready to produce a response token. ASR delay remains in the pipeline, but it becomes a background task that usually finishes before turning into the bottleneck.
Third, we placed ASR and TTS inference in the same region as our telephony gateway. We had assumed model inference was the main delay and network travel between components barely mattered. Instead, geographic routing added 80-120ms to the audio round trip. Co-located infrastructure reduced that to 20-30ms.
Current Position and Challenges
Ringg calls now show median end-to-end latency of about 680ms in our Bengaluru-routed infrastructure. At the 90th percentile, latency is about 820ms, so roughly one in ten turns passes the 800ms threshold. The tail is driven mainly by LLM time on complex-intent turns that need more processing than typical replies.
Complex-intent turns ask the agent to reason beyond a simple reply, such as cross-referencing caller context, choosing among response branches, or producing more than the 30-50 words typical of most turns. We route these cases to a faster, lighter model variant, accepting some quality loss. The choice is intentional: a good fast reply is preferable to an excellent slow one.
Function-call delay remains only partly solved. A CRM lookup or backend API call can add 200-600ms, depending on the external system. If it occurs during LLM generation, it stalls the TTS stream. We now pre-load likely calls when a turn starts, using the partial ASR transcript, so results can arrive early. This suits predictable flows better than unexpected questions.
800ms is not the finish line. For high-volume call operations, it is a commercially important divide between usable and noticeably broken. Reaching it meant treating the voice pipeline as a hard real-time system, not a chain of API calls, which demands a different engineering approach. We continue tuning the balance among latency, accuracy, and response quality as infrastructure changes.