„Real-time Voice AI: Extracting structured data from live phone calls using LLMs"

In the real world, things rarely go according to plan for call centre agents. The customers calling may be stressed (e.g., broken car, flooded house), and they may give unprompted, out-of-order answers. The call centre agents’ job is not easy - they may need to navigate a huge conditional question tree from memory during a high-stakes situation. They will often be required to enter large amounts of structured data during or after the call, based on handwritten or typed notes.

The results are missing, incorrect data for an economically relevant process. What if the agents could follow a dynamic script that changes in real time based on what the caller just said? And what if the structured data could be extracted directly from the audio stream in real time, so that by the time the call is finished, all forms are already filled out? This is what we set out to build.

Our approach has been an LLM-based application that guides call centre agents through the call, adapting to new information as soon as it is available. It both selects the questions and picks out the answers. In order to implement this, we’ve had to solve many challenges - in context engineering, networking, and system design. The aim of the talk is to guide you through the trade-offs and decision that we made.

Referenten: Izer Onadim und Theodor Nguyen, Members of Technical Staff at QuantCo