Luke Oliff.
Luke Oliff · Writing

Full-Duplex Voice AI Needs a New Architecture

·Voice AI·10 min read·Luke Oliff

The current generation of voice AI systems works like a relay race. Speech-to-text passes the baton to an LLM, which passes it to text-to-speech. Each segment runs sequentially. The user hears the output, responds, and the cycle repeats.

Full-duplex voice does not work like this. In full-duplex, both parties can speak at the same time. The assistant needs to process incoming audio while it is still generating its own output. It needs to handle interruptions mid-syllable, adjust its response based on something the user just said, and keep the conversation flowing without awkward pauses.

This is not a harder version of the relay race problem. It is a different problem entirely.

Diagram comparing turn-based relay race architecture to full-duplex architecture with context tracking

The Generative Context Mis-anchoring Problem

A new paper from researchers at multiple institutions introduces the concept of Generative Context Mis-anchoring (GCM). The PACE system names the core failure: LLM-based full-duplex voice services let users speak while the assistant is still responding. Because servers generate output and advance dialogue state faster than clients can play it, the model makes decisions about what to say next based on audio the user never heard.

Think about what that means at a conversational level. You are on a voice call with an AI assistant. The assistant starts answering your question. Halfway through, you interrupt with a clarification. The assistant needs to decide: should it stop talking, finish the sentence, or pivot to address your clarification? The right answer depends on what you actually heard, not what the model decided to generate.

The paper frames this precisely. PACE defines GCM as the failure that occurs “when subsequent user speech is interpreted based on content the user never heard.” The model advances its internal state assuming the user has processed every token it generated. But audio playback is asynchronous. The client buffers, streams, and plays at real-time speed. The model runs ahead. The gap is the problem.

PACE addresses this with a middleware layer that anchors the model’s context to the client playback boundary. It tracks what audio has actually reached the user’s ears and repairs the context after interruptions to exclude assistant content that never played.

This is not about faster TTS. It is about architectural correctness.

Where the Current Architecture Breaks

The relay race model (STT to LLM to TTS) has three fundamental problems in full-duplex scenarios.

Problem 1: Pipeline latency accumulates

Each stage adds latency. STT needs enough audio to make a transcription reliable. The LLM needs the full transcription to generate a coherent response. TTS needs the full text to produce natural speech. In a turn-based system, this is fine. You wait for the response. In full-duplex, every millisecond of pipeline latency pushes the assistant response further behind the conversation.

Problem 2: State is shared but playback is not

The LLM maintains a conversation state that includes what it has said. But the audio playback pipeline operates independently. If the user interrupts, the LLM cannot distinguish between “the user heard everything up to this point and is responding” and “the user heard nothing after the first sentence and is clarifying.” Both look like new user input to the model. The difference matters for how the assistant should respond.

Problem 3: Interruption handling is a hack, not a feature

Current systems handle interruptions by detecting voice activity, stopping TTS, and sending the user’s audio to the STT pipeline. This works for simple cases but falls apart with barge-in (interrupting mid-word), conversational overlap (both speaking briefly together), and repair (the user corrects themselves). These are normal human conversational behaviours. The current architecture treats all of them as edge cases.

What the Research Says About the Path Forward

The PACE paper is not alone in identifying these problems. This week’s research from the speech and audio community shows a coordinated move toward solving different parts of the full-duplex puzzle.

ReLMCodec: Predictable Speech Tokens

The ReLMCodec paper takes on the foundation layer. Current neural audio codecs produce tokens that vary unpredictably based on acoustic content. This makes it hard for LLMs to plan speech output in advance. ReLMCodec introduces pre-quantization phoneme structure that makes speech tokens predictable at the linguistic level before acoustic encoding happens. The result is a codec that an LLM can reason about before it generates audio.

This matters for full-duplex because it means the LLM can plan prosody, timing, and interruptions at the token level rather than generating blind and hoping the audio sounds right. Predictable tokens mean predictable generation time, which means the system can decide whether to keep talking or stop before the audio is even rendered.

Beyond Naturalness: We Are Evaluating the Wrong Thing

The Beyond Naturalness paper probes automated TTS evaluators and finds they correlate poorly with what listeners actually care about in interactive contexts. Evaluators optimise for naturalness in isolated utterances. But in full-duplex conversation, listeners care about timing, responsiveness, the handling of interruptions, and whether the system sounds like it is actually listening. A system that scores well on naturalness can still fail as a conversational partner.

This is an important corrective. If we optimise for the wrong metric, we build systems that score well and perform badly. The paper argues for linguistically grounded evaluation dimensions that capture turn-taking behaviour, prosodic appropriateness, and repair handling.

CtrlSpeech: Expressive Speech Needs Coarse-to-Fine Control

CtrlSpeech addresses expressive speech synthesis with a coarse-to-fine control architecture. Rather than conditioning the entire generation on a single style embedding, it first plans at the prosodic level (coarse) and then fills in acoustic detail (fine). This layered approach maps directly onto the full-duplex problem: the system can commit to a prosodic plan early and adjust the acoustic detail as the conversation evolves.

CuteTTS: Efficiency Without Sacrifice

CuteTTS demonstrates that high-quality zero-shot TTS does not require expensive iterative diffusion sampling. By modelling continuous latents autoregressively, it achieves streaming-quality speech with lower inference cost. Every millisecond of TTS latency reduction matters when the user is waiting for a response while the system processes new input.

The Voiceprint Fallacy: Security Assumptions Under Pressure

The Voiceprint Fallacy paper questions whether voice is a reliable biometric identifier. In a full-duplex world where AI can clone voices from seconds of audio, voice-based security becomes harder to defend. The paper argues that voices are not unique biometric imprints in the way fingerprints are. The variability introduced by emotion, environment, and recording conditions makes voice identification fundamentally probabilistic rather than deterministic.

This matters for full-duplex voice AI because these systems will handle sensitive transactions. If the security model assumes voiceprints are unique, the architecture needs a rethink.

What Full-Duplex Voice Actually Needs

Pull these papers together and a clearer picture emerges of what a full-duplex voice architecture requires.

Context-aware playback tracking

The PACE approach is the right direction. The model must know what the user has actually heard, not just what it generated. This means the architecture needs a feedback loop from the audio playback layer to the dialogue state manager. The middleware layer cannot be an afterthought. It must be a first-class component.

Predictable generation timing

ReLMCodec’s approach to predictable speech tokens enables the model to estimate generation time before it starts speaking. This is critical for interruption handling. If the model knows “this response will take 800ms to generate and 2 seconds to play,” it can make informed decisions about whether to commit to the full response or leave room for the user to interrupt.

Metrics that match the task

The Beyond Naturalness paper is right. We need evaluation frameworks that measure full-duplex conversation quality, not isolated utterance naturalness. Turn-taking latency, interruption handling success rate, repair recovery time, and context alignment after interruptions all matter more than MOS scores on single sentences.

Layered generation with mid-flight adjustment

CtrlSpeech’s coarse-to-fine approach maps to a full-duplex architecture where the system commits to a conversational direction early but adjusts the surface delivery as the conversation unfolds. This is how humans do it. We start a sentence with a general idea of where it is going and adjust based on the listener’s reaction. Voice AI needs the same capability.

What This Means for Developers Building Voice AI Today

If you are building a voice agent today, the relay race architecture will work for simple use cases. Question answering, form filling, and single-turn interactions are fine. The problems start when you want natural conversation.

The practical advice is unglamorous:

Plan for context tracking from day one. Even if your current system is turn-based, design the state management layer so it can accept playback position callbacks. Adding this later is a rewrite.

Choose codecs with your full-duplex future in mind. The codec you pick today constrains what you can do with interruptions and barge-in. Look for codecs that support partial rendering and streaming start.

Measure the right things. Track how long your pipeline takes from user audio end to assistant audio start. Track interruption recovery time. Track how often the assistant continues talking after the user has started speaking. These metrics will tell you whether your architecture is ready for full-duplex.

Assume voice will be cloned. The Voiceprint Fallacy paper is a warning. Do not build a security model that depends on voice biometrics as a primary factor. Voice is a convenience factor, not an identity guarantee.

FAQ

What is Generative Context Mis-anchoring?

GCM is a failure mode in full-duplex voice systems where the LLM interprets user speech based on content the user never heard. It happens because the model advances its dialogue state faster than the audio playback reaches the user. The PACE paper introduces this term and provides a middleware solution.

Can existing voice agent frameworks handle full-duplex?

Partially. Pipecat and LiveKit Agents can manage streaming audio and basic interruptions, but neither implements playback-aware context tracking at the level PACE describes. The middleware layer that anchors model state to client playback is not built into these frameworks. You would need to add it yourself.

Is this relevant for non-voice AI systems?

Yes. Any system where a model generates output that is consumed asynchronously faces a version of this problem. Streaming text responses, real-time translation, and AI game characters all have a gap between generation and consumption. The principle of anchoring context to the consumption boundary applies broadly.

How far away are production ready full-duplex voice systems?

The research is moving fast. PACE is a middleware prototype, not a shipped product. ReLMCodec and CtrlSpeech are pre-prints. The evaluation gap that Beyond Naturalness flags means we lack reliable benchmarks for full-duplex quality. Production-ready systems are probably 12-18 months out for most use cases, with simpler constrained scenarios (customer service, appointment booking) arriving sooner.

Does latency reduction solve the problem?

No. Lower latency narrows the gap between generation and playback but does not eliminate it. The fundamental issue is architectural: the model does not know what the user has heard. Even if you reduce generation latency to near zero, the asymmetry between model-side and user-side state exists as long as the model consumes its own output faster than the user can hear it. The fix is context tracking, not speed.

References

All posts