Partway through building voice mode, OpenAI released GPT-Live, a native speech-to-speech model. Audio in, audio out, one model, full duplex. It listens and answers the way a person does, one continuous thing, and we explored whether we could build on it.
A speech-to-speech model is a closed box. You can't put an agent in the middle of it, which means no tools, no memory, no skills, no approvals. Maybe the box opens someday. Today, for an assistant whose whole value is the agent loop, that trade kills the product before it ships.
So we kept the classic cascade, the one that's always felt robotic: transcribe the speech, run the full agent loop, synthesize the reply. Three stages bolted together, and every joint between them is a seam you can hear as dead air. We spent the engineering on hiding those seams. The goal, written down early and returned to often, was to make voice mode feel natural.
How it started
Voice began as a feature of the existing chat surface. Press a button, talk instead of type, watch the same transcript fill in. It worked on the first try, and it felt wrong in a way that took a few days to name: watching both sides of the conversation render as text while you talk turns the conversation into a deposition. You stop talking to something and start reading a court record of yourself. The transcript was breaking it.
That observation ended voice as a feature of our chat interface. We built it a room instead. Start a conversation and the assistant grows to fill the screen: the walls take its color, its two cartoon eyes look back at you, a "Listening" label sits beneath them, and that's the whole interface. The surface came together in days. It looked alive.
But it didn't sound alive. Underneath the eyes, the interaction model was a walkie-talkie: you talk, it pauses, it thinks, eventually it talks back. Every silence reminded you there was machinery back there. The gap between how it looked and how it felt lived entirely in the turn-taking.
We could have shipped it that way. It worked, and the launch date was close. We decided to spend another week building, because voice gets one first impression. Someone who tries it once and feels the walkie-talkie closes the window, and no changelog entry brings them back to check whether it got better.
The pipeline
A single WebSocket carries the whole conversation between the browser and the server.
Each turn routes through the same agent loop that text chat uses, so voice inherits tools, memory, and skills for free. That's the naive version, and the naive version feels mechanical. Four mechanisms make it feel alive.
Speculative launch
The longest pause in a cascade sits between the user's last word and the model's first token. We start early to hide it: the moment the voice detector hears trailing silence, the agent turn dispatches, before we've confirmed the person is actually done talking. The endpoint decision runs concurrently with the model's time to first token. If it turns out they were only pausing mid-thought, the whole turn rolls back, persisted message included, and the listening window quietly extends. You pause, it waits. You finish, it's already answering. This overlap is the single biggest latency win in the system.
The mouth and the brain
Every turn goes first to a fast, cheap "front" model, which must open its reply with a verdict.
Three verdicts are possible.
Simple turn: the front model just answers. Its output streams straight to text-to-speech, and first audio arrives fast because a small model is doing the talking.
Hard turn: the front model escalates. It says one short phrase out loud, "let me think about that for a second," and hands the turn to the strong model, which runs it in the background while that phrase plays. The mouth covers for the brain, so the caller never hears the big model's think time as silence.
Not a turn at all: if the wording says the person isn't finished, a trailing "and..." or a thought left hanging, the front model stays quiet and we keep listening.
The front model has no tools, but it knows what the strong model can do, so it hands off hard requests instead of refusing them or making things up.
Floor-holders
Even with the front model in place, there are stretches where nothing is being said: usually a turn running long on tool calls. The assistant fills those gaps out loud. Some of it is plain filler, "one sec, let me think." But on long turns it gets better than filler: the main agent bubbles up what it's actually doing, and the front model speaks it. "Searched the web, reading through it now" is a live progress report on the work happening behind the voice.
The narration is audio-only. It never appears in the transcript and never persists to the conversation, so the record stays clean. And it's timed off silence the user can actually hear: the server tracks when the last audio finishes playing on their end, not when it finished generating, so the quiet clock starts when the room goes quiet. Narration fires only when someone is genuinely sitting in silence, and never talks over the assistant's own voice.
Gapless synthesis
The opening clause of a reply is synthesized the instant it arrives instead of waiting for a full sentence. A prefetch pump keeps the next segment's synthesis hidden behind the current segment's playback, and the client schedules each audio buffer against a running playhead cursor rather than waiting for the previous one to end. Chunks stitch together with no audible gaps.
Interrupting
Anyone who has used a voice assistant has tried to talk over it and been ignored. The assistant keeps talking, your words land on deaf code, and the only fix is to wait it out. Real conversation doesn't work that way. You interrupt, the other person stops.
Ours stops. Cut in anywhere, over a word, mid-sentence, even during the thinking before any audio has played, and the assistant yields immediately and listens to the new thing you're saying. Interrupting doesn't feel like a feature you invoke. It feels like talking to someone with ears.
Getting there meant fighting the browser. Echo cancellation is half-duplex: while the assistant speaks, the microphone ducks your voice, so real interruptions arrive choppy while the assistant's own audio bleeds through and registers as speech. Our first detector kept hearing itself and stopping mid-sentence. The guard we shipped adapted to the echo instead of fighting it: it waits for a quarter second of sustained speech before yielding, so bleed-through can't trigger it, while tolerating the gaps the canceller chops into a real voice. Long enough to ignore its own echo, short enough that you never feel ignored.
Fail-open everywhere
Every component we added in the name of latency has a hard timeout, and every failure mode resolves to the same safe default. When the endpoint decider times out, the turn releases. When the phrase generator errors, a static phrase plays instead. When the progress narrator gets nothing usable from the model, it stays silent. The front model can make the experience a little slower or a little less polished, and that is the worst it can do. It cannot break turn-taking and it cannot wedge a call. The cleverness lives in the happy path, and the floor stays solid underneath it.
Putting it together
No single mechanism here is the trick. Speculative launch hides the endpoint decision, the front model hides the strong model's think time, progress narration hides the tool calls, and the prefetch pump hides the synthesis. Each one covers a different seam, and the seams were the whole difference between a pipeline and a presence.
The cascade is all still in there, transcribing, looping, synthesizing. If we did our job, you'll never hear it.
