Speech-to-speech (S2S) models represent a fundamental shift in voice AI: instead of chaining separate speech-to-text, language model, and text-to-speech components together, these end-to-end systems process audio input and generate audio output within a single unified model. This eliminates the latency overhead and information loss inherent in traditional cascaded pipelines, enabling more natural, expressive, and responsive voice interactions.
Voice agents built on S2S technology are driving a rapidly growing market projected to expand from $14.8 billion in 2024 to over $61 billion by 2033. The applications span customer service automation, real-time translation, accessibility tools, and interactive AI assistants. For AI engineers, understanding the architectural tradeoffs between cascaded and end-to-end approaches is critical for building production-grade voice systems that balance latency, quality, and controllability.
The field has evolved rapidly since OpenAI’s GPT-4o introduced native audio capabilities in 2024. Models like gpt-realtime, NVIDIA’s PersonaPlex, and Meta’s Moshi architecture now demonstrate that S2S models can match or exceed cascaded systems in instruction following, tool use, and conversational naturalness while delivering significantly lower latency.
The traditional approach to voice AI uses a pipeline of three independent models working in sequence:
An orchestration layer acts as the “conductor,” managing the real-time data flow between these components, handling interruptions, and coordinating streaming.
Cascaded architectures are modular, debuggable, and allow you to swap individual components (e.g., upgrade just the TTS). End-to-end S2S models offer lower latency and better preservation of paralinguistic cues (tone, emotion, hesitation) but are harder to debug and control. Most production systems in 2025-2026 use a hybrid approach.
End-to-end S2S models process the entire audio-in to audio-out pipeline within a single neural network. The model directly “hears” the input audio, reasons about it, and generates speech output without ever creating an intermediate text transcript. This means the model can perceive and respond to paralinguistic features like emotion, tone, pacing, and emphasis that would be lost in a text-only pipeline.
OpenAI’s gpt-4o-realtime-preview and the newer gpt-realtime model exemplify this approach. The model doesn’t rely on a transcript of the user’s input — it hears emotion and intent directly, filters out noise, and responds in speech. This enables latencies under 300ms for first-byte response, approaching the ~200ms response gap in natural human conversation.
Modern S2S models typically use a multi-stage internal architecture even though they present as a single model:
# Conceptual S2S pipeline (simplified)
# Audio input -> tokens -> transformer -> tokens -> audio output
import numpy as np
class SpeechToSpeechModel:
def __init__(self, encoder, transformer, decoder):
self.encoder = encoder # Audio -> token embeddings
self.transformer = transformer # Multimodal reasoning
self.decoder = decoder # Token embeddings -> audio
def forward(self, audio_input):
# Encode raw audio to latent representations
audio_tokens = self.encoder.encode(audio_input)
# Process through transformer (reasoning happens here)
output_tokens = self.transformer.generate(
audio_tokens,
stream=True # Enable streaming for low latency
)
# Decode back to audio waveform
audio_output = self.decoder.synthesize(output_tokens)
return audio_outputAchieving real-time performance requires streaming at every stage. The STT model transcribes audio in overlapping chunks, the LLM begins generating a response as soon as the first words arrive, and the TTS begins synthesizing audio before the LLM has finished its thought. This parallelism across the pipeline is what makes sub-second response times possible.
OpenAI’s Realtime API supports two transport methods for low-latency communication: WebRTC for peer-to-peer audio streaming with built-in echo cancellation and noise suppression, and WebSocket for server-side integrations with more control over the audio pipeline.
NVIDIA’s PersonaPlex (released January 2026) builds on the Moshi architecture and introduces a hybrid prompting system with two distinct channels: a text prompt defining the agent’s role, personality, and behavior rules, and a voice prompt as an audio embedding capturing vocal characteristics, accent, and prosody. This separation allows developers to define “what the agent says” independently from “how the agent sounds.”
OpenAI’s TTS models similarly support instructable voice control — developers can specify speaking style through text instructions like “talk like a sympathetic customer service agent” to adjust tone, pace, and expressiveness without changing the underlying voice.
Modern voice agents aren’t limited to conversation. They can invoke external tools during a voice interaction — checking databases, calling APIs, or triggering workflows — while maintaining natural conversational flow. The gpt-realtime model shows significant improvements in calling tools with precision during voice interactions, enabling agents that can look up orders, schedule appointments, or process transactions mid-conversation without awkward pauses.
Voice agents handle inbound calls for order status, troubleshooting, and account management. Companies deploy these on existing telephony infrastructure using SIP trunking, with the voice agent connected via WebSocket or WebRTC to process calls in real time. The agent can detect caller frustration from vocal cues and escalate to human agents when needed.
S2S models enable live multilingual conversations where the system translates speech in one language to speech in another with sub-second latency. Meta’s SeamlessStreaming model specifically targets this use case with Efficient Monotonic Multihead Attention (EMMA) for simultaneous translation.
Voice agents serve as interfaces for users who cannot interact through traditional text or visual UI. End-to-end models excel here because they preserve the emotional context and conversational nuance that cascaded systems often lose, making interactions feel more natural and less robotic.
Legacy IVR systems with rigid “press 1 for…” menus are being replaced by conversational voice agents that understand natural language requests. The key advantage is handling ambiguous or complex requests that wouldn’t fit traditional menu trees, while still maintaining the reliability and auditability that enterprise telephony demands.
Frameworks like Pipecat, LiveKit, and Vocode provide the orchestration layer for building production voice agents. These handle the complex real-time plumbing — audio streaming, turn detection, interruption handling, and failover — letting developers focus on the conversational logic and business rules.