Voice Agent Latency: Measure TTFT, TTFS and TTFA to Hit 700–800 ms Voice‑to‑Voice Targets

You hear the first token, but the user still waits

You push to talk. An inference API returns a token. The UI stays quiet for another half second. The user frowns. TTFT, the oft-cited “time to first token, ” marks when generation starts, not when your user actually hears anything. For voice agents, perceived responsiveness covers the whole chain: STT → turn detection → LLM inference → tokens/sec → TTS → network and player buffering.

Three latency metrics you must measure (and how)

  • Time to first token (TTFT), when the LLM emits the first token after your request. Measure as: timestamp(request sent) → timestamp(first token received). Useful for LLM vendor comparisons, but incomplete for voice UX.
  • Time-to-first-sentence (TTFS), LiveKit’s user-facing metric: when there is enough generated text for the TTS to begin producing audio. Measure as: timestamp(request sent) → timestamp(TTS emits first audio sample corresponding to a complete clause/sentence).
  • Time-to-first-audio (TTFA), for speech-to-speech models: when the first audio sample is played. Measure as: timestamp(request sent) → timestamp(playback starts on the client).

Capture p50/p95/p99 for each metric. Correlate TTFT with TTFS/TTFA to see whether a fast first token actually converts into audible output.

The real voice stack, realistic timings

Use these practical ranges as a starting baseline (LiveKit’s voice-turn breakdown is a helpful reference):

  • STT + endpointing: ~100 to 200 ms
  • LLM inference (streaming, warm): ~300 to 500 ms
  • TTS model inference: ~100 to 200 ms (model-only numbers often quoted by vendors)
  • Network/WebRTC RTT: ~50 to 150 ms typical between colocated cloud regions and client, cross‑continent RTTs are higher

Stacked together, LiveKit frames a pragmatic end-to-end target of roughly 700 ms to 1.2 s. Kwindla Hultman Kramer recommends an 800 ms median voice-to-voice target (with 1, 500 ms acceptable for a proof of concept). Daily’s testing echoes this UX baseline: human turn latency and comfort start slipping past about 800 ms, and a cascaded transcription→LLM→voice harness often leaves roughly a 700 ms TTFT budget to stay natural-feeling.

What leaderboards actually teach us (distilled)

Benchmarks are noisy and workload-dependent. Artificial Analysis (AA) changed its default workload in March 2026 to 10k input tokens and runs tests from a specific cloud region; AA reports that its TTFT numbers include network latency. That matters, because prompt size, host region, warm versus cold, and the definition of “first token” all shift results.

The headline patterns you should take away:

  • Low TTFT does not guarantee low TTFS or TTFA. Throughput, tokens per second, and TTS behavior determine when a clause is complete and audio begins.
  • Hosting and colocation matter. The same model can show different TTFT on different hosts.
  • Reasoning workload and decoding settings can multiply latency, sometimes dramatically.

Selected, provenance-linked examples (illustrative, not exhaustive):

  • Artificial Analysis reported Baseten serving gpt-oss-120b with a first-chunk TTFT of 0.23 s on a 10k workload (AA provider leaderboard, retrieved Aug 30, 2026).
  • Artificial Analysis example: Mercury 2 produced a first chunk at 3.07 s with throughput ~770 tok/s on the workload tested (AA).
  • Cerebras example showed throughput ~1, 697 tok/s with TTFT 0.49 s, illustrating a high-throughput, low-TTFT tradeoff (AA).
  • GPT-5.6 Luna measured 0.59 s on Amazon Bedrock and 0.74 s on the OpenAI API for the same model, demonstrating host differences (AA).
  • LiveKit reports warm-request TTFTs varying by model. For example, LiveKit measured Gemma 4 31B TTFT at 192 ms (warm), while other models ran slower; LiveKit also states a warm request “starts returning tokens in around 100 ms” under some conditions, these are different conditions (warm vs model vs measurement method) and not directly comparable without context (LiveKit documentation and measurements).
  • LiveKit’s TTFS across full conversations: Gemma 4 31B 354 ms; Gemini 2.5 Flash 1, 034 ms; GPT-4.1 1, 088 ms; Gemini 3.0 Flash 1, 267 ms; GPT-5.5 1, 404 ms (LiveKit).
  • Speech-to-speech: Grok Voice Think Fast 2.0 High showed TTFA 0.70 s with 97% speech reasoning and 94.7% task success on AA / Big Bench Audio (retrieved Aug 30, 2026).

Always read the fine print. Many vendor numbers are model-inference-only and exclude network, client buffering, and queuing. AA and LiveKit call out those caveats explicitly.

STT and TTS are the backbone, don’t ignore them

STT latency and endpointing often dominate user time-to-hear. AssemblyAI reported a 307 ms median word emission for Universal-Streaming versus Deepgram Nova-3 at 516 ms (AssemblyAI’s numbers). Deepgram’s Flux approach claims folding end-of-turn into recognition can cut agent response latency by 200 to 600 ms compared with an STT plus separate VAD pipeline. That exposes parameters like eot_threshold and an EagerEndOfTurn event, which are powerful but trade accuracy for speed in some cases.

TTS vendor numbers look attractive but are often model-only. ElevenLabs reports about 75 ms for Flash v2.5 inference. Cartesia has published sub-90 ms TTS claims for Sonic-3.6. Add network RTTs (commonly 20 to 200 ms depending on geography and colocation) and client jitter and playout buffers (many web players and SDKs buffer roughly 200 to 500 ms), and the real wall-clock until the user hears audio grows.

Two levers that change everything: reasoning effort and tokens/sec

Reasoning effort matters. Aggressive internal chain-of-thought, long tool chains, or high reasoning settings materially increase latency. Example: Gemini 3.1 Flash Live’s TTFA moved from 0.96 s to 2.99 s between Minimal and High reasoning settings. OpenAI’s gpt-realtime-2.1 moved from 0.97 s to 1.21 s for a similar policy change (benchmarks cited by vendors and AA). That’s a knob you can turn: cap reasoning for real-time turns and move deep work to async workflows.

Tokens per second, or throughput, also changes the picture. A low TTFT model that emits tokens slowly can still delay TTFS. If you get the first token quickly but the model outputs only a few tokens per second, you’ll wait to accumulate a clause. Conversely, a slightly slower TTFT with high throughput can reach TTFS sooner. Practical knobs:

  • Limit max_tokens for real-time turns.
  • Prefer greedy or streaming decoding over complex beam or sampling when latency matters.
  • If available, use early shallow answers, then kick off deep follow-ups asynchronously.
  • Measure tokens/sec on your exact prompts and client region, don’t rely on vendor marketing numbers alone.

Prioritized engineering playbook

Do this first (highest ROI)

  • Colocate models and media, reduce cross-region RTTs. LiveKit highlights colocation as “very high impact.” Expect meaningful median and tail latency reductions when inference and media servers share region or AZ.
  • Measure user-facing metrics, instrument TTFS and TTFA end-to-end and capture p95/p99 in production. Users remember long tails; p95 matters more than p50.
  • Cap reasoning for real-time turns, limit max_tokens, set short time budgets, or use a dedicated “fast” LLM config for conversational turns.

Next (tactical, moderate effort)

  • Implement eager end-of-turn, fold endpointing into STT when possible (Deepgram Flux shows 200 to 600 ms potential reductions). A/B test false positive EOT rate and token waste; start with conservative thresholds.
  • Optimize TTS selection and tokens/sec, pick TTS models and generation settings that reach a clause quickly; measure TTS latency including network and client playout.
  • Control sync tool calls, minimize blocking external calls during a turn and make long retrievals asynchronous.

Optional / longer-term

  • Hybrid architectures, run a lightweight local model for immediate short replies and escalate to cloud models for deep reasoning.
  • Custom colocation & edge placements, for global products, place inference near major user clusters and use smart routing to minimize tail latency.

Success criteria examples

  • Target median voice-to-voice: 700-800 ms. Aim p95 < 1.5 s for acceptable UX (Kwindla Hultman Kramer, Daily guidance).
  • After colocation, expect measurable reduction in median RTT and p95 TTFS, validate with before and after runs (N ≥ 50 warm requests).
  • If you enable eager EOT, keep false EOT rate < 2-5% in production or tune thresholds.

Speech‑to‑speech vs cascaded pipelines: when to pick which

Short decision criteria:

  • Choose cascaded STT → LLM → TTS when you need observability, deterministic tracing, heavy tool integration, or advanced instruction following. Cascades give you intermediate artifacts (transcript, reasoning tokens) for debugging and compliance.
  • Choose single-model speech‑to‑speech if you want a smaller operational surface, simpler deployment, and your tasks are well-covered by the model’s capabilities. Speech‑to‑speech models can hit sub-second TTFA, but Daily’s evaluations show capability gaps on complex multi-turn tasks compared with cascades.

Put it another way: if your product relies on tool calls, precise logs, or complex multi-turn behavior, cascaded pipelines remain the pragmatic choice today. If your priority is minimal stack complexity and your dialog patterns are short and well-scoped, speech-to-speech is rapidly maturing.

Reproducible experiment you can run this afternoon

  • Environment: colocate components in the same cloud region as your target users (or use your current production region).
  • Prompt: a single one-sentence user request that should produce a one-clause assistant reply (15-25 tokens).
  • Warm the system: run 10 warm-up requests to prime caches and GPUs.
  • Measurements (repeat N = 50 warm runs): record timestamps for:
    1. request sent
    2. first LLM token received (TTFT)
    3. last LLM token corresponding to the first clause
    4. first TTS audio sample generated
    5. first audio sample played on the client (TTFA)
  • Report p50/p95/p99 for TTFT, TTFS (request → TTS first audio), and TTFA (request → playback start). Also measure tokens/sec by recording time between the first and last tokens of the first clause.
  • Variation: flip decoding to greedy vs sampled, or cap max_tokens, and compare TTFS/TTFA deltas.

Key questions

  • Is TTFT enough to choose an inference API for voice agents?

    No. TTFT shows when generation begins, but perceived responsiveness depends on TTFS/TTFA, tokens-per-second, STT latency/endpointing, TTS behavior, network RTT, and client buffering.

  • What end-to-end latency should I target for a natural feeling voice agent?

    A practical median target is ~700-800 ms voice‑to‑voice; treat 1, 500 ms as a tolerable POC figure. Aim to keep p95 well under 1.5 s where possible (Kwindla Hultman Kramer; Daily testing).

  • Which single changes cut the most perceptible latency?

    Colocation (model & media), folding end‑of‑turn into STT (eager EOT), capping reasoning effort for live turns, and optimizing tokens/sec/TTS selection deliver the biggest wins.

  • How should I read vendor latency claims?

    Assume many vendor numbers are model‑inference‑only. Ask whether reported values include network, warm vs cold, and client playout buffering. Always validate with your own prompts, region, and clients.

  • Are speech‑to‑speech models ready to replace cascaded stacks?

    They’re promising and often faster on TTFA in benchmarks, but cascaded systems still lead for observability, tool integration, and complex multi‑turn instruction following. Test on your tasks before switching.

One actionable rule

Instrument what your users actually hear, not just the vendor’s TTFT. Measure TTFS and TTFA end-to-end, control reasoning and token emission, colocate where you can, and optimize for the p95 your users will notice. Do that, and your voice agent will stop feeling delayed and start feeling conversational.

2