GPT Realtime can play two characters. It can't cast two voices.

We wanted one live AI actor to improvise a scene between two characters. The first attempt sounded like a narrator reading material back to us. So we changed the job: use the scene as context, improvise the exchange, play both roles, and do not announce speaker names.

That worked. The response stopped feeling like a read-aloud and started feeling like a scene. The characters pursued different intentions. Their pacing and delivery shifted. The model was acting.

But it was still one actor. Both characters came through the same underlying voice. The useful question was no longer how to polish the roleplay prompt. It was where character identity should live in the audio architecture.

The prompt improved the scene

Our test used a real two-character scene from WriteForYou rather than a generic assistant conversation. We wanted to know whether a realtime model could work from story facts and perform the exchange without simply reciting the source.

The revised prompt made three changes: improvise from the scene facts, perform both conversational roles, and avoid speaker labels. The response differentiated the characters through word choice, intent, rhythm, and emotional delivery.

This matches the control OpenAI documents for Realtime sessions. The Realtime API reference says instructions can guide response content and audio behavior—including speed, emotion, and laughter—while warning that those instructions are guidance rather than guarantees.

The remaining problem was identity

For character audio, performance is only half the job. A listener also needs to know who is speaking without continuously reconstructing the answer from context.

Our two roles had different intentions, but they did not have stable, audibly separate identities. Cadence could change. Emotion could change. The underlying voice did not.

In text, paragraph breaks and quotation structure can carry speaker identity. In a visual scene, faces and cuts can do it. In audio-only fiction, games, and interactive dialogue, the voice itself often carries the label. If two characters sound like the same person, the audience spends attention resolving the speaker instead of following the scene.

Why another prompt was not the answer

A Realtime session has a configured output voice, and the official session reference states that the voice cannot be changed after the model has responded with audio.

That makes voice a session-level property in this architecture, not a character-level field a roleplay prompt can assign on every turn. A prompt can direct impatience, softness, speed, or laughter within the configured voice. Casting requires an explicit mapping from each character or turn to a stable voice identity.

The architecture decision hiding inside the demo

The hybrid sounds attractive, but it is not a feature toggle. It introduces a handoff. The product must preserve character assignment, delivery intent, interruptions, timing, and nonverbal direction between improvisation and synthesis. It must also decide whether the delay is acceptable.

The correct choice depends on whether live or separately cast is the non-negotiable requirement.

Why ElevenLabs is our next experiment, not our conclusion

ElevenLabs Text to Dialogue represents the problem differently: each dialogue turn has its own text and voice, with a voice ID selecting the speaker and optional audio tags guiding delivery. That is much closer to a casting model than a single session voice.

The same documentation is explicit about the tradeoff. Text to Dialogue is not intended for realtime conversational agents, and several generations may be needed. It recommends keeping combined input text at or below 2,000 characters per request for reliable generation, says output is nondeterministic, and currently lists no limit on speakers per dialogue. The API reference shows the per-turn structure.

We have not run the follow-up comparison yet. Documentation tells us that ElevenLabs exposes the control surface we were missing. Only the experiment can tell us whether it preserves the acting quality, timing, consistency, and production economics we need.

A better test than “which model sounds best?”

We will compare three outputs from the same scene:

1. GPT Realtime as a one-voice live improviser.
2. ElevenLabs Text to Dialogue as scripted multi-speaker rendering.
3. A hybrid where GPT creates the dramatic turn plan and a renderer casts each character.

We will measure blind character distinguishability, emotional accuracy, voice consistency, time to first usable audio, regeneration count, manual timing or stitching, and cost per usable minute under the exact test window.

The most important question may be the least technical: does adding separate voices preserve the sense that two characters are reacting to each other, or does it turn a performed scene back into assembled lines?

The rule we are taking forward

If your product needs a live conversational performer, start with Realtime and treat one configured voice as part of the session design.

If your product needs a clearly distinguishable cast, start with explicit character-to-voice assignment and accept that you are building a rendering pipeline.

If you need both, prototype the handoff before committing to the architecture. Measure latency and editing cost, but also listen for what gets lost between improvisation and synthesis.

Our experiment did not reveal a bad model or a winning provider. It revealed a category error in our original question. We asked one realtime actor to become a cast. The acting improved. The cast requires another layer.