Duplex-MPE: Five Open-Weight Voice Models Struggle in Group Conversations, and Four Are Correct Under 2% of the Time They Answer (2609.31948)

Duplex-MPE (arXiv 2609.31948, 25 September 2026) puts a voice assistant into a conversation among three or four people and asks whether it knows when to speak. Of the five open-weight speech models tested, the best, MiniCPM-o 4.5, was correct on 47.30% of the explicit requests it answered. The other four scored below 2% on that measure (accuracy conditional on the model speaking). All numbers below are the authors' own claims on a synthetic benchmark, not independent replications.

The paper evaluates five open-weight systems plus a text-only Gemini 3.1 Pro reference. Paper read: full text, not only the abstract.

What problem does Duplex-MPE solve?

It tests whether a full-duplex speech assistant can take part in a shared conversation: answer when asked, stay quiet when others are talking, and stop when someone else answers first. The authors argue existing benchmarks centre on a designated user rather than an assistant sitting among several people, as in a meeting, living room or car.

Prior work covers parts of the problem. Full-Duplex-Bench v1.5 simulates four overlap scenarios (user interruption, backchannel, talking to others, background speech) inside a user-assistant exchange. Duplex-MPE says it is the only protocol that combines streaming audio, non-addressed speech, no fixed user and yielding on interruption. That is the authors' comparison, drawn from their own Table 1.

How does Duplex-MPE work?

Each scenario is a scripted conversation with one assistant, called Aria, and three or four humans. Claude Opus 5 writes the scripts from combinations of setting, activity, relationship, device context and register. Qwen3-TTS then synthesises every human turn separately, which gives exact turn boundaries.

Every script contains turns with a required behaviour, per the paper and its summary page:

Each scenario exists in two versions. In the explicit version the request names Aria. In the implicit version the name is replaced by a contextual cue. That gives 2,000 pairs and 4,000 conversations, with 61.10 hours of distinct human speech and 110.93 hours of total audio (average 99.8 seconds per conversation). The models get a spoken instruction preamble and then continuous audio, with no transcript, speaker label or turn boundary.

How are the four scores defined?

• Fresh-onset response rate: a new speech onset shortly after the request, with no assistant speech already active.
• Conditional answer accuracy: correctness, judged only on requests where the model spoke.
• Silence preservation: share of N1 to N3 windows with no speech, or only a brief acknowledgement.
• Answering-window yield: whether the model is silent by a deadline after a human resolves the N4 question, scored only on events where it was already speaking.

Scoring uses Qwen3-ASR-1.7B for transcription, Silero VAD for speech detection and Claude Opus 5 for semantic judgments. Six human reviewers checked samples: reviewers agreed with Opus 5 on 99 of 100 answer-correctness judgments and 98 of 100 acknowledgement-versus-intrusion judgments (authors' report).

What are the results?

MiniCPM-o 4.5 is the only model that both speaks when asked and stays quiet otherwise. The table gives the authors' numbers under explicit addressing, over the same 2,000 requests per condition (Table 3).

Implicit addressing gives nearly the same picture: MiniCPM-o 4.5 scores 0.9435, 0.4665, 0.9289 and 0.5925 on the four metrics. The paper's abstract says MiniCPM-o 4.5 leads on three scored capabilities, and the table agrees: Voila holds the top answering-window yield.

What do the failure profiles look like?

Talking a lot is not the same as taking part well. Freeze-Omni has the highest response presence (0.9955) but the lowest fresh-onset rate, and it kept silent in only 5 of 20,360 windows that required silence. FLM-Audio produced only 22 correct answers among 1,901 requests where it spoke.

Moshi has higher response presence than Voila (0.8155 vs 0.6515) but far lower silence preservation (0.2734 vs 0.8351). Choosing fresh-onset rate over response presence changes the rankings: pooled across both conditions, the Spearman correlation between the two orderings is -0.4303.

Why is Voila's stopping score misleading?

Yield is computed only on events where the model stayed quiet during the question, answered within a short window, and was still talking when the human resolved it. In the explicit condition that is 1,633 events for MiniCPM-o 4.5, 219 for Moshi, 977 for FLM-Audio, 620 for Voila and 1 for Freeze-Omni. Voila's 0.8484 describes stopping on its eligible events, not overall N4 performance, as the authors note.

Does naming the assistant change anything?

For a text-only model, yes, massively. For the five speech systems, the paper detects no significant effect. Gemini 3.1 Pro, given speaker-attributed transcripts and no audio, responded to 0.9470 of named requests and 0.3040 of unnamed ones, a 64.3-point gap.

The speech systems' explicit-minus-implicit differences range from -1.75 to +0.60 percentage points, and the smallest paired-test p value is 0.2365. The authors offer two readings: the models infer the addressee from context, or they do not treat the name as a cue and behave the same either way. Their data cannot separate the two.

One more detail on addressing: MiniCPM-o 4.5 spoke after 95.45% of explicit requests to Aria, yet failed to stay silent on 216 of 4,704 requests aimed at other humans or devices (4.6%).

Does the evidence hold up?

The measurement design is careful, but the scope is narrow. Here is where the claims are weakest.

• Open-weight only. Only open-weight systems were run through the audio protocol. MP-Bench (arXiv 2609.13076, 11 September 2026) benchmarks 12 voice agents, including GPT-Realtime, Gemini-2.5 Live and Gemini-3.1 Live, so commercial systems have been tested elsewhere.
• A text reference is not a speech model. The 64.3-point Gemini result measures a language model reading labelled transcripts. It says little about how Gemini's audio products behave.
• Synthetic everything. Scripts come from Claude Opus 5 and audio from Qwen3-TTS. The authors state the English scenes do not represent real populations, room acoustics or social norms.
• Generator and judge overlap. Claude Opus 5 writes the scripts and also scores answer correctness. The human spot checks (99/100 and 98/100) mitigate this but sample only a small slice.
• Floor effects. Four of five models are correct on under 2% of the requests they answer, so there is little room to detect any effect of addressing on answer quality.
• Paired versions are not perfectly matched. The authors say later edits and separate synthesis can change wording, event counts and audio between the explicit and implicit versions.
• Reproducibility. The paper says the authors will release manifests, audio and scoring code. The paper gives no link yet.

On the plus side, the paper reports bootstrap confidence intervals (2,000 scenarios resampled 10,000 times). They support the silence-preservation ordering: MiniCPM-o 4.5 leads Voila by 8.25 to 9.58 points under explicit addressing.

How does it compare with related work?

Duplex-MPE asks a different question from most neighbours, and it partly agrees with the nearest one.

The two multi-party audio benchmarks reach compatible conclusions by different routes: current realtime systems handle group settings poorly. They are not direct rivals, since they use different tasks, scoring and model lists. MiniCPM-o 4.5 itself is described by its team in arXiv 2604.27393 as a full-duplex omni-modal model.

What does a working researcher take from this?

Report several metrics, not one response rate, and publish the denominators. Duplex-MPE's clearest contribution is showing that response presence hides three different failures: missed requests, content-free speech and intrusion into others' turns.

Practical points for people building or evaluating voice agents:

1. Score silence as a first-class outcome. A model that speaks in nearly every window can look strong on presence-based metrics.
2. Separate new speech onsets from continuation speech, since the two rank models differently.
3. Report how many events reach each conditional metric. A yield of 0.8484 on 620 events and a yield on 1 event are not comparable.
4. Do not assume a name cue is being used. Test with paired inputs, as this paper does.
5. Treat LLM-judged scores as one layer and check them against human labels on a sample.

What's still unclear?

Several questions remain open as of 15 October 2026.

• Why do speech systems ignore the name? The authors say the null result could reflect addressee inference or name-blindness. No experiment here separates them.
• Would commercial audio models do better? Duplex-MPE does not run them. MP-Bench suggests realtime commercial agents also struggle in multi-party settings, but its protocol differs.
• Does synthetic audio transfer to real rooms? The authors caution against drawing conclusions beyond the benchmark scenarios.
• Will the data and code appear? The paper promises a release, and the paper gives no link yet.
• Would the headline model gap hold under a different judge? All semantic scoring uses one LLM judge, checked on 100 answer judgments.