A natural phone conversation swaps speakers in roughly 200 milliseconds. An AI agent on a SIP call has to hear the caller, decide they are done, think, and start speaking — across a real carrier network — fast enough that the pause feels human. Here is our audio pipeline, from RTP packet to first synthesized syllable, and where every millisecond goes.
The budget
We hold the full pipeline to a 700ms p95 from caller end-of-speech to agent first-audio. The allocation: ~80ms network and jitter buffer, ~120ms streaming speech recognition finalization, ~60ms endpoint detection, ~250ms model time-to-first-token, ~120ms text-to-speech time-to-first-byte, ~70ms return path. Every stage streams; nothing waits for anything upstream to finish completely.
Endpointing is prediction, not detection
Waiting for silence to confirm the caller finished costs 400–600ms by itself — the whole budget. Instead we run a turn-prediction model on the live transcript and prosody that estimates, word by word, the probability the utterance is complete. The agent starts formulating (and often synthesizing) its response before the caller technically stops talking, and throws the draft away cheaply if more speech arrives.
Speculative synthesis and the retraction problem
Speculation means sometimes being wrong: the caller inhales, and continues. If the agent has already started speaking, we have a collision. Our rule: within the first 300ms of agent audio, a detected caller continuation triggers an immediate stop with no filler — cutting off cleanly reads as politeness. Past that window, the agent finishes its clause and yields. Callers perceive both as natural; what they punish is the agent barreling on.
Why SIP makes this harder than a web demo
Browser demos enjoy Opus at 48kHz and a single network hop. Production calls arrive as 8kHz G.711 through carrier gateways that add their own buffering, and jitter characteristics vary by originating network. We run per-call adaptive jitter buffers and keep recognition models trained specifically on narrowband telephone audio — a model that shines on podcast audio can lose 15 points of accuracy on a cell call in a car.