Voice Agent Latency: Where the Seconds Go and How to Cut Them

In short: When callers say a voice agent "feels slow", they are reacting to the gap between when they stop talking and when they hear the agent start. That gap is the sum of several stages: deciding the caller has finished, transcribing, thinking, calling tools, generating speech, and moving audio across networks. Measure each stage, fix the biggest one first, and design the conversation so slow work never happens in silence.

Key takeaways

• Track one number above all: the time from the caller's last word to the agent's first audible sound.
• End-of-turn detection and the LLM are usually the largest pieces. Text-to-speech is often the smallest.
• Slow tool calls are the most common cause of long silences we see in real builds.
• Streaming, faster models, nearby regions, and short first sentences all help. So does saying something useful while a tool runs.

Why a few hundred milliseconds matter

People are very sensitive to timing in conversation. A study of turn-taking across ten languages, published in PNAS, found that speakers everywhere avoid overlapping talk and keep the silence between turns short, with language averages falling within a narrow band of about 250 ms of the cross-language mean (Stivers et al., 2009).

Software voice agents cannot match that today, but every extra second makes the call feel less natural, invites the caller to talk over the agent, and makes the agent look unsure even when its answer is right.

Where the time goes

A typical cascaded voice agent runs this loop on every turn:

1. Audio in: the caller's voice travels from their phone through the carrier and telephony provider to the agent platform.
2. End-of-turn detection: the system decides the caller has finished speaking.
3. Speech-to-text (STT): the final words are transcribed.
4. LLM: the model reads the conversation and starts producing a reply.
5. Tools (sometimes): the model calls an API to check availability, look up an order, or book a slot.
6. Text-to-speech (TTS): the reply is turned into audio.
7. Audio out: the audio travels back to the caller and is buffered before playback.

Twilio published a useful breakdown of this path. In their illustrated example, STT takes about 350 ms, the LLM about 375 ms, and TTS about 100 ms, plus smaller amounts for media edges, buffering, decoding, and service hops. They list launch targets of about 1,115 ms median mouth-to-ear turn gap, with an upper limit of 1,400 ms (Twilio, Core latency in AI voice agents). Treat those figures as a reference, not a promise. Your own numbers depend on your stack and your callers' locations.

1. End-of-turn detection

Twilio's guide calls end-of-turn detection "the long pole in the tent", because systems pad it to avoid cutting people off. Most silence-based fallbacks default to around 500 ms, and lowering that risks treating a natural pause as the end of a turn.

In ElevenLabs Agents, the related control is turn eagerness, with Eager, Normal, and Patient modes (ElevenLabs, Conversation flow). Eager suits quick back-and-forth. Patient suits calls where people read out addresses, reference numbers, or long explanations. There is no universal right answer, so test with real callers.

2. Speech-to-text

Streaming STT transcribes while the caller is still speaking, so most of the work is done by the time they stop. If your stack waits for the full utterance before transcribing, that is an easy win.

3. The LLM

The model's time to first token is often the biggest single piece. It grows with:

• Larger or reasoning-heavy models
• Long system prompts and long conversation history
• Large knowledge base context added to every turn
• Busy or distant model endpoints

ElevenLabs lets you choose the LLM per agent (Models) and has a cascading fallback for when a model is slow or fails (LLM cascading). It also offers Realtime models, where ElevenLabs manages the LLM, TTS, and ASR settings together: Eleven Realtime v1 Mini is described as the lowest latency and cost option, while v1 Max reasons before responding for complex instructions and multi-step tool use (Realtime models).

ElevenLabs also notes that its agents start speaking once enough words and a comma arrive from the model, rather than waiting for complete sentences. That rewards prompts that make the agent lead with a short, direct first clause.

4. Tool calls

This is where we see the longest silences in real projects. A tool call adds a full round trip to your API, plus whatever that API does: database queries, CRM lookups, calendar checks, sometimes a chain of automations.

Common causes:

• A webhook into a no-code workflow that runs several steps before replying
• Calendar availability checked across many staff calendars one at a time
• A CRM search that is not indexed on the field you query
• Cold starts on serverless functions

Fixes, in order of impact:

1. Make the endpoint fast. Cache what you can, query only what you need, and return as soon as you have the answer.
2. Return a short answer the agent can read out. Less for the model to process, fewer follow-up tool calls.
3. Fill the gap honestly. ElevenLabs supports tool call sounds and a soft timeout that speaks a brief filler phrase when the LLM is slow (Conversation flow). A short "Let me check that for you" is better than three seconds of nothing.
4. Never fake completion to save time. If the tool is slow, the agent must not say "you're booked" before the tool confirms it. We covered this failure in The Webhook That Couldn't Answer Back.

5. Text-to-speech

Modern streaming TTS is fast. ElevenLabs states its Flash models reach about 75 ms model inference, while pointing out that this excludes network and application overhead, and that time to first audio is the number users actually feel (Understanding latency). Their latency guide adds that default, synthetic, and instant-clone voices are generally faster than professional voice clones, and higher-quality output formats can add delay (Latency optimization).

6. Network and geography

Every hop between regions costs time. ElevenLabs lists typical time-to-first-byte for Flash over websockets of 100 to 150 ms in North America, Europe, and South East Asia, and 150 to 200 ms in South Asia and North East Asia (Latency optimization). Twilio's guide makes the same point for multi-vendor stacks: crossing several external networks adds delay at each hop.

Practical moves:

• Run your tool endpoints in the same region as the agent platform you use.
• Avoid chaining several telephony intermediaries when one will do.
• Test from where your callers are, not only from your office.

How to measure it

You cannot fix what you only feel. For each test call, record:

Listen to recordings alongside the numbers. A 1.5 second gap after "Can I book Tuesday?" feels very different from the same gap after "Let me check the calendar."

A short fix list

1. Measure the turn gap on twenty real or realistic calls.
2. Find the biggest stage. It is usually turn detection, the LLM, or a tool.
3. Shorten the system prompt and move reference material into retrieval or tools.
4. Try a faster model or a realtime model, then rerun your test calls.
5. Speed up the slowest tool endpoint, and add a filler sound or phrase for anything that must stay slow.
6. Tune turn eagerness for the kind of call you handle.
7. Co-locate your services and test from your callers' regions.

Frequently asked questions

What is a good response time for an AI voice agent?

There is no single standard. Twilio publishes launch targets of roughly 1.1 seconds median mouth-to-ear turn gap with a 1.4 second upper limit for their stack. Use that as a reference, then measure your own calls and aim to remove long outliers first.

Why does my voice agent pause before answering?

Usually because of end-of-turn padding, a slow LLM response, or a slow tool call. Check tool durations first. A single slow webhook can add seconds.

Does a better voice model make the agent slower?

It can. ElevenLabs states that Flash models trade some quality for speed and that professional voice clones are generally slower than default or instant-clone voices.

Should I make the agent respond faster by cutting people off?

Only carefully. More eager turn-taking reduces gaps but increases interruptions. For calls where people read out details, a more patient setting often gives a better experience.

How Octacer can help

We build and repair voice agents on ElevenLabs, Retell, and custom stacks, including the webhook and CRM side where most latency hides. If your agent pauses, talks over callers, or claims bookings it has not made, our System Rescue work starts with measuring the turn gap and the tools behind it. For new builds, see AI Apps and Integrations, read The ElevenLabs Voice Agent Playbook, or check the AI cost and latency spike triage playbook. You can also book a call.

Sources

• Stivers, T. et al. (2009). Universals and cultural variation in turn-taking in conversation. PNAS: https://pubmed.ncbi.nlm.nih.gov/19553212/
• Twilio, Core latency in AI voice agents: https://www.twilio.com/en-us/blog/developers/best-practices/guide-core-latency-ai-voice-agents
• ElevenLabs Docs, Understanding latency: https://elevenlabs.io/docs/eleven-api/concepts/latency
• ElevenLabs Docs, Latency optimization: https://elevenlabs.io/docs/eleven-api/guides/how-to/best-practices/latency-optimization
• ElevenLabs Docs, Conversation flow: https://elevenlabs.io/docs/eleven-agents/customization/conversation-flow
• ElevenLabs Docs, Models: https://elevenlabs.io/docs/eleven-agents/customization/llm
• ElevenLabs Docs, LLM cascading: https://elevenlabs.io/docs/eleven-agents/customization/llm/llm-cascading
• ElevenLabs Docs, Realtime models: https://elevenlabs.io/docs/eleven-agents/customization/realtime-models
• ElevenLabs Docs, Tool call sounds: https://elevenlabs.io/docs/eleven-agents/customization/tools/tool-configuration/tool-call-sounds