What is an AI voice agent?

A voice agent is a pipeline, not a single model. Audio arrives from the phone network, speech-to-text turns it into words, a language model decides what to say and what to do, text-to-speech turns the reply back into audio, and the whole loop repeats every turn. Each stage adds latency, and the sum is what determines whether the call feels natural or exhausting.

What separates a working agent from a demo is almost never the model. It is the handling of everything around the happy path: the caller who interrupts, the eight-second silence, the accent the recogniser was not trained on, the question that falls outside the agent's brief, and the moment a human needs to take over mid-call.

Voice agents are distinct from IVR phone trees, which follow a fixed menu, and from chatbots, which have no real-time constraint. The real-time constraint is the hard part: a chatbot can think for three seconds and nobody minds; a voice agent that pauses three seconds sounds broken.