Building a Production Telephony Voice Agent on a Cascaded Pipeline: An Experience Report

Abstract

We report on building and operating a multi-tenant telephony voice agent that answers phone and WhatsApp calls with a cascaded speech-to-text, dialog-flow and retrieval, and text-to-speech pipeline. The system serves three ingress paths (a carrier WebSocket media stream, a SIP trunk bridged into a WebRTC media server, and WhatsApp Business Calling) through one shared agent. We describe the architecture and then focus on what went wrong in production and how each problem was root-caused. A baseline turn took over 10 s; the dominant cost was a retrieval service that streamed its answer as a single event, not the dialog-flow hop it had been blamed on (measured at 134 to 291 ms). We describe an acoustic endpointer with a rule-based semantic continuation hint, a block-LMS echo canceller that removes 28.7 dB of synthetic echo at 0.074 ms per 20 ms frame but remains disabled in production pending real-call validation, defects that made calls impossible to end, template variables read aloud to callers, and a class of silent failures in which the call connects but the agent is deaf or mute. We separate measured results from projections throughout, and we list what remains unvalidated.

Keywords: voice agents, telephony, turn-taking, acoustic echo cancellation, retrieval-augmented generation