Three years ago we put our first AI agent on a production phone line. Since then, agents on AMP Cortex have handled tens of millions of calls — and most of what we believed at the start turned out to be wrong. This is the technical retrospective.
The hardest problem is knowing when to shut up
Early agents were polite conversationalists and terrible phone workers. They waited for clean end-of-utterance signals that real callers never produce: people trail off, restart sentences, and talk over hold music. Our biggest single quality win was not a better model — it was a barge-in system that lets the caller interrupt the agent mid-sentence and have the agent actually stop, keep the audio it already said as context, and re-plan.
We now treat interruption as the normal case, not the exception. Roughly 30% of caller turns begin before the agent finishes speaking, and the agents that score highest on CSAT are the ones that yield fastest.
Latency budgets are product decisions
A phone call gives you about 800 milliseconds of silence before a human thinks something is broken. That budget has to cover speech-to-text, retrieval, the model, text-to-speech, and two network hops. We publish an internal latency budget for every stage and treat a regression like an outage — because to the caller, it is one.
The counterintuitive finding: callers forgive a fast wrong-ish answer that gets corrected far more readily than a slow perfect one. Conversational repair is cheap; dead air is expensive.
Escalation is a feature, not a failure
Our worst early metric was agents trying too hard. An agent that attempts a task outside its knowledge base produces confidently wrong answers — the single fastest way to destroy trust. We now score agents on escalation quality: did it recognize the boundary, summarize the call for the human, and transfer warm? Teams that tune for graceful escalation see higher containment over time, not lower, because callers learn the agent is trustworthy.
What we would tell ourselves three years ago
Build the telephony first; the AI is replaceable, the call infrastructure is not. Instrument everything — you cannot coach what you did not record and score. And design for the queue, not the demo: a two-minute scripted demo call has nothing in common with Monday morning at a 12-line clinic.