Skip to content

Comfortable Home turf: speech models, VAD, ASR and TTS. New: multi-party turn-taking vocabulary and a paired test (McNemar). No heavy math.

Duplex-MPE: Benchmarking Multi-Party Interaction in Full-Duplex Dialogue

To judge an assistant sitting in a group conversation, score starting, answering, staying silent and stopping on separate denominators, because a single 'did it respond?' number hides which of those it is failing.

Toolbox6 pages. First question: What does full-duplex let a voice model do?
The benchmark7 pages. First question: What does the model hear in one conversation?
What it finds4 pages. First question: How do five open models score on the four?
Your move2 pages. First question: What would a careful reviewer question?
Check yourself3 pages. First question: In the taco scene, which turn is the single T request?
Authors
Chengqian Ma, Wenhao Feng, Weixuan Jin, Gaole Dai, Tianyu Xie, Yuexiao Ma, and 4 more
Published
2026, arXiv preprint
arXiv
2609.31948 (lesson written from v1)
Paper type
Benchmark or dataset

Toolbox 1 Start here. Nothing earlier is needed.

What does full-duplex let a voice model do?

  1. A cascade takes strict turns. VAD waits for you to stop. ASR, an LLM and TTS then reply. While it talks, it can't hear you.
  2. Full-duplex listens while it talks. Room audio keeps streaming in while its own speech streams out (§1). Moshi is one such model.
  3. So silence becomes an output. No harness says "your turn now". At every moment, the model chooses to speak or stay quiet.
  4. Three choices show on the waveform. When to start, when to stay quiet, when to stop. Duplex-MPE scores all three (§1). [[5 | systems tested (App. A1.3)]]

Both timelines are illustrative. The five tested systems are listed in App. A1.3.

A full-duplex model runs its own turn-taking. Next: in a room of five, whose turns are meant for it?

Toolbox 2 Uses the "when to speak" choice from page 1.

Who is a turn meant for, in a room of five?

  1. Every turn has an addressee. The addressee is who a turn targets. "C, be honest…" targets C, not Aria (Table 4).
  2. Some turns target another device. "Hey Siri, how long is the bus?" is for Siri. Aria hears it and stays out (Table 4).
  3. Some turns target nobody. Muttering about money has no addressee at all. The paper calls this self-talk (§3.1).
  4. Only a few turns target Aria. "Aria, which one's the cheapest?" is a turn it must answer (Table 4). [[2,000 | of 26,182 turns need an answer (Table 7)]]

Quotes come from the paper's example scene (Table 4). Counts are real (Table 7, explicit condition). The room layout is illustrative.

Aria must follow every turn but answer very few. Next: how a request shows it is meant for Aria.

Toolbox 3 Uses the addressee idea from page 2.

How can a request be for Aria if nobody says "Aria"?

  1. Explicit addressing names Aria. "Aria, out of those three, which one's the cheapest per person?" The name points the request (Table 5).
  2. Implicit addressing uses a cue. "You've been logging all three numbers, so run it." Only Aria was logging, so listeners infer it (Table 5).
  3. Deleting the name isn't enough. A nameless request might target a human. So the cue must point at Aria (§3.3).
  4. Humans agree the cue works. Reviewers judged all 50 sampled implicit requests to address Aria (Table 11). [[50 / 50 | judged addressed to Aria (Table 11)]]
  5. The cue costs a few words. Unnamed requests average 123.3 characters, named ones 111.6 (Table 8).

Both wordings and all numbers are real (Table 5, Table 8, Table 11). The "name deleted" wording is ours, for contrast.

A request can point at Aria without naming it. Next: when talking over someone is allowed.

Toolbox 4 Uses the "when to speak" choice from page 1.

When may Aria talk while someone else has the floor?

  1. One person holds the floor. The floor is the right to speak right now. Here B is mid-turn, so B holds it.
  2. A backchannel leaves it alone. A brief "Understood." adds nothing and takes no turn. That is a backchannel (App. A3.3).
  3. Content takes the floor. "Yeah, I'm free for all three" adds a claim. Now Aria competes for the turn (Table 20).
  4. Short can still take the floor. Voila's "The ants." lasted 0.45 s. It carried content, so it was penalised (Table 20). [[0.45 s | "The ants." (Table 20)]]

Quotes and durations are real model outputs (Table 20). The floor timeline is illustrative.

A backchannel is fine; content is an intrusion. Next: how a judge tells them apart at scale.

Toolbox 5 Uses the backchannel from page 4.

How do you judge thousands of replies without reading each?

  1. A VAD marks speech first. A causal Silero VAD scans Aria's output waveform. Bursts under 120 ms don't count as speech (§3.4).
  2. Short pauses are merged. Two bursts less than 800 ms apart become one episode. Move the pause slider (§3.4).
  3. No speech means no judge. A silent window passes at once, with no transcript and no LLM call (App. A3.3).
  4. Speech goes to ASR and a judge. Qwen3-ASR-1.7B transcribes it. Claude Opus 5 decides what it was (§3.5). [[2,172 | of 20,360 reached the judge (Table 10)]]
  5. 🧠 We call this VAD-gated judging. A cheap timing check runs on every window. The costly LLM check runs only where speech exists.

Thresholds and counts are real (§3.4, Table 10; MiniCPM-o, explicit). The waveform is illustrative.

Timing comes from the VAD; meaning comes from a judge. Next: how to compare paired yes/no results.

Toolbox 6 A small statistics tool, used again on page 16.

How do you tell whether paired yes/no results really differ?

  1. Run each scene twice. Same model, two versions of one scene. Each pair lands in one of four cells (App. A5.1).
  2. Agreeing pairs say nothing. "Spoke in both" and "spoke in neither" can't show which version gets more speech. Only the disagreeing cells matter.
  3. Treat each disagreement as a coin flip. With no real difference, disagreeing pairs should split evenly. The exact McNemar test measures how lopsided the split is.
  4. MiniCPM-o splits 90 to 78. That is close to even, so there's no evidence of a difference (Table 16). [[p = 0.3961 | 90 vs 78 (Table 16)]]
  5. Now try 90 versus 40. The p value falls far below 0.05. The split decides, not the 2,000 total. Move the sliders yourself.

MiniCPM-o's four cell counts and its p value are real (Table 16). The sliders run the exact two-sided test in your browser.

Only pairs that disagree count, and only their imbalance matters. Next: the conversation the model actually hears.

Benchmark 1 From here on, follow one scene: four classmates picking a place to eat.

What does the model hear in one conversation?

  1. First, a spoken job description. A 31.9 s preamble tells the model it is Aria, and when to stay silent (App. A3.1).
  2. Three options get pitched. Tacos at nine dollars, karaoke at fifteen, Luigi's at eleven. Each is said to the group, so it's N2 (Table 4).
  3. Then the one real request. Turn 4 asks Aria which option is cheapest. Its label is T. The gold answer is the taco place (App. A2.1).
  4. Mentions and mutters need silence. Naming Aria without asking anything is N1. Thinking aloud is N3 (§3.1).
  5. One question gets resolved. Turn 10 asks Aria whether tacos take card (N4Q). A human answers it in turn 11 (N4R).
  6. The model gets only the audio. No transcript, speaker labels or turn boundaries (§3.4). Flip the toggle. [[1 of 13 | turns is a T (Table 4)]]

Labels and turn order follow the paper's scene (Table 4); texts are shortened. Row spacing and bar lengths are not time.

Answer turn 4, start on turn 10, and otherwise stay quiet. Next: how 2,000 such scenes were made.

Benchmark 2 How the taco scene from page 7 was made, twice.

How do you make 2,000 conversations, each in two versions?

  1. Start from five attributes. Setting, activity, relationship, devices and register. Ours: classmates in a dorm common room, with another smart speaker (§3.2).
  2. An LLM writes the script. Claude Opus 5 writes each turn, its label and T's gold answer. T is never first and never last (§3.2).
  3. Rewrite T the other way. Each scene gets an explicit and an implicit T. 1,958 pairs changed only T; 42 also changed the turn before (App. A2.4).
  4. TTS voices every turn alone. Qwen3-TTS gives speakers A–D four fixed voices. Separate clips give exact turn boundaries (App. A2.3).
  5. Humans spot-check a sample. All 1,282 checked TTS turns were intelligible. 98 of 100 gold answers held up (Table 11). [[4,000 | conversations (§3.3)]]

All counts are real (§3.2, App. A2.2–A2.4, Table 11). Boxes are not to scale.

Scripts supply the labels; TTS supplies the timing. Next: the four scores computed on them.

Benchmark 3 Scores the taco scene from page 7. The next four pages each zoom into one row.

What are the four scores, and why not just one?

  1. Start: did it begin answering? After T, Aria must start fresh within 5 s. Counted over all 2,000 T turns (Table 2).
  2. Answer: was the answer right? A judge compares Aria's reply with the gold answer. Counted only where Aria spoke (Table 2).
  3. Shush: did it stay quiet? Each N1, N2 and N3 window should hold no speech, or only a backchannel (Table 2).
  4. Stop: did it yield when resolved? After N4R starts, Aria must fall silent within 2 s. Only eligible events count (Table 2).
  5. No total is formed. Each score keeps its own denominator. A win on one can't pay for a failure on another (§3.5). [[4 | separate denominators (§3.5)]]

Definitions and totals are real (Table 2, Table 7, Table 19). Start, Answer, Shush and Stop are our nicknames for the paper's four metric names.

Four behaviours, four denominators, no average. Next: Start, on turn 4.

Benchmark 4 Start, from page 9, on turn 4 of our scene.

Why doesn't "spoke after the request" mean "started answering"?

  1. Measured: a fresh onset. Aria is silent when T ends, then starts speaking within 5 s (Table 2). Drag the onset slider.
  2. On every T, out of 2,000. In our scene, that's turn 4: "which one's the cheapest?" Every T counts, spoken to or not (Table 2).
  3. Speech already running doesn't pass. A model talking straight through turn 4 never decided to answer. Its onset came before T ended (App. A3.2).
  4. Too late is a miss. No onset within 5 s fails Start. Once Aria speaks, playback waits for 2 s of silence, at most 60 s (§3.4).
  5. Three outcomes, one passes. Fresh, already speaking, or miss. Response presence adds the first two, so it can look high without fresh starts (Table 9). [[0.9500 | MiniCPM-o's Start (Table 9)]]

The 5 s rule and MiniCPM-o's counts are real (§3.4, Table 9, explicit). Aria's onset on the slider is illustrative.

Start counts decisions, not airtime. Next: whether the answer was right.

Benchmark 5 Answer, from page 9, on turn 4. Uses the judge from page 5.

What counts as a right answer, and out of how many?

  1. Measured: correctness. Qwen3-ASR transcribes Aria's reply. Claude Opus 5 compares its meaning with the gold answer (§3.5).
  2. On T turns with speech. For turn 4, whatever Aria said is judged against "the taco place, nine dollars a head" (App. A2.1).
  3. Counted against response presence. A silent T has nothing to judge, so it leaves this denominator. It still fails Start (§3.5).
  4. Unclear verdicts count as wrong. An empty or unparseable answer is scored incorrect. Humans agreed with the judge on 99 of 100 (§3.5, Table 11).
  5. Speaking isn't answering. FLM-Audio spoke on 1,901 requests and was right on 2. Flip the toggle (§4.1). [[2 of 1,901 | FLM-Audio correct (§4.1)]]

Counts are real (Table 3, Table 9, §4.1; explicit). MiniCPM-o's 903 correct answers are our arithmetic (0.4730 × 1,909). Aria's reply is illustrative.

Answer is judged only where Aria spoke. Next: what counts as staying quiet.

Benchmark 6 Shush, from page 9, on the silent turns of our scene. Uses pages 4 and 5.

When does Aria's talking break the silence?

  1. Measured: silence kept. An N1, N2 or N3 window passes if Aria says nothing, or only a backchannel (Table 2).
  2. On every other turn. In our scene that's ten turns: 1–3, 5–9, 12 and 13 (Table 4). "Understood." in turn 7 would pass.
  3. Counted against every window. Each condition has 20,360 silence windows. Each one passes or fails on its own (Table 19). [[20,360 | silence windows (Table 19)]]
  4. Content fails, however short. Even "Nine dollars" counts as an intrusion. It answers a question, so it takes the floor (App. A3.3).
  5. Unreadable speech fails too. If the VAD hears speech but ASR returns nothing, the window fails. Voila had 171 such windows (Table 10).

Rules and counts are real (Table 2, Table 10, Table 19, App. A3.3). Placing these replies in turn 7 is illustrative.

Shush judges content, not just sound. Next: stopping when a human resolves the question.

Benchmark 7 Stop, from page 9, on turns 10–11: "does it take card?", then "never mind".

When does stopping count as stopping?

  1. Measured: yielding. The scheduler waits exactly 3 s after turn 10. Then turn 11 plays, whatever Aria is doing (§3.4).
  2. On eligible N4 events only. Aria must stay quiet during turn 10, start within the gap, and still be talking at turn 11 (§3.5).
  3. Everything else is left out. Finished early, already talking, or talking over the question: excluded, not failed (App. A4.1).
  4. Pass: silent 2 s into N4R. Silent at that deadline passes. Drag the slider to move when Aria stops (App. A4.1). [[2 s | deadline after R starts (§3.5)]]
  5. And silent until the window ends. Speaking again before the clip ends, plus 0.5 s, fails. Answer quality doesn't matter here (App. A4.1).

The rules and gaps are real (§3.4, §3.5, App. A4.1). Clip lengths and Aria's speech are illustrative.

Stop is a rate over eligible events only. Next: how five open models score on all four.

Finding 1 The four scores from page 9, on five open models.

How do five open models score on the four?

  1. MiniCPM-o 4.5 leads three of four. Start 0.9500, Answer 0.4730 and Shush 0.9242, on explicit requests (Table 3).
  2. Answers are weak everywhere. Even the leader is right less than half the time. The other four score 0.0123 or less (Table 3). [[0.4730 | best Answer score (Table 3)]]
  3. Shush splits the field. Voila 0.8351, FLM-Audio 0.3957, Moshi 0.2734. Freeze-Omni keeps silence in 5 of 20,360 windows (§4.1).
  4. Stop has a surprise leader. Voila yields on 0.8484 of its scored events, MiniCPM-o on 0.5964 (Table 3). Page 17 explains the catch.
  5. Implicit requests look the same. Flip the toggle. No score moves by more than 0.0438 (Table 3).

All values from Table 3, to four decimals. Freeze-Omni's Stop isn't reported: only one event qualified (§4.3).

One model is nearly usable; the rest fail in different ways. Next: why "it responded" can mislead.

Finding 2 Uses Start versus response presence from page 10.

Why does Freeze-Omni look best and turn out worst?

  1. Rank by presence first. Freeze-Omni has speech after 0.9955 of T requests. That's first place (Table 3).
  2. Now split each bar. Of its 2,000 T turns, 1,486 found it already talking. Only 505 were fresh onsets (Table 9).
  3. The leader drops to last. Re-rank by Start: Freeze-Omni scores 0.2525. It rarely stopped talking long enough to begin (§4.5).
  4. FLM-Audio hides the same way. 825 of its "responses" were already running. Presence 0.9505, Start 0.5380 (Table 9).
  5. The two rankings disagree. Over ten model-and-condition entries, they correlate at −0.4303. No model keeps its explicit rank (§4.5). [[−0.4303 | rank correlation (§4.5)]]

Counts from Table 9, explicit requests. Each bar is one model's 2,000 T requests.

"It responded" can mean "it never stopped talking". Next: does naming Aria change anything?

Finding 3 Uses explicit vs implicit requests (page 3) and the McNemar test (page 6).

Does saying "Aria" change what the models do?

  1. A text model reacts strongly. Gemini 3.1 Pro reads speaker-labelled transcripts. It responds to 0.9470 of named requests and 0.3040 of unnamed ones (Table 17).
  2. So the rewrite matters to a reader. That 64.3-point gap gives \(p<10^{-371}\) (§4.2). [[64.3 pts | Gemini's gap (§4.2)]]
  3. Speech models barely move. Pick a model. Explicit minus implicit ranges from −1.75 to +0.60 points (Table 16).
  4. No test reaches significance. The smallest p is 0.2365, for Voila's 395 versus 430 split (Table 16).
  5. Two readings fit this. Equal skill at reading context, or deafness to the name (§4.2). Also, 339 pairs differ beyond T (App. A2.4).

Rates from Table 17; pair counts and p values from Table 16, recomputed in your browser. Gemini read text, not audio: it is a reference, not a competitor.

Naming Aria changes a reader's decision, not the speech models'. Next: why Voila's Stop score needs a second look.

Finding 4 Uses the eligibility rule from page 13.

Why isn't Voila's 0.8484 Stop score a win?

  1. Start from all N4 events. Each condition has 1,911 resolved questions (Table 7). Every model faces the same ones.
  2. Most never qualify. Voila first speaks after the question in 913. Only 620 are still talking when N4R begins (Table 14).
  3. Of those, 526 yield. 526 / 620 gives 0.8484, the reported Stop score (Table 15).
  4. MiniCPM-o qualifies far more often. 1,633 scored events and 974 yields: 0.5964. A lower rate, but more yields (Table 15). [[620 vs 1,633 | scored events (§4.3)]]
  5. 🧠 Divide by all 1,911 instead. MiniCPM-o then yields on 51.0%, Voila on 27.5%. This is our arithmetic, not a paper metric.

Counts are real (Table 7, Table 14, Table 15; explicit requests). The per-1,911 rates are our arithmetic.

A conditional rate needs its denominator beside it. The paper reports Stop only when at least 30 events qualify (§4.3).

Your move 1 Our critique of pages 7–17.

What would a careful reviewer question?

  1. 🧠 Preamble untested alone. A 31.9 s spoken instruction sets the rules (App. A3.1). §3.4 mentions a probe; we found no results.
  2. 🧠 Writer and grader overlap. Claude Opus 5 writes gold answers and judges replies (§3.2, §3.5). Human checks used balanced samples (App. A3.4).
  3. 🧠 Implicit may be ambiguous. Reviewers said yes for 50 of 50 (Table 11). Gemini, reading clean transcripts, responded to only 0.3040 (Table 17).
  4. 🧠 N4 counts don't reconcile. 968 scenarios have an N4 event after T (App. A2.2). Yet each condition has 1,911 N4 events (Table 7).
  5. 🧠 Steelman: control first. Clean TTS turns with 0.5 s gaps never overlap (App. A3.1). That isolates the speaking policy; fresh draws can come later (§5).

Pointers mark the evidence behind each concern. The pills are our judgment.

Trust the Shush order most and the Answer ranking least. Every Shush pair differs significantly; most Answer pairs don't (App. A7.1).

Your move 2 Goal picked by us: a voice agent for meetings or a family car. Uses pages 14–17.

Should you put an open full-duplex model in a shared room today?

  1. Take the best model, MiniCPM-o 4.5. It starts fresh on 0.9500 of the requests put to it (Table 3).
  2. Count its intrusions. Its Shush is 0.9242, so 7.58% of silence windows fail. A scene holds about 10.18 such windows (Table 3, Table 19).
  3. Scale to your meeting. A scene averages 99.5 s (Table 6). Move the slider: an hour of similar talk gives about 28 intrusions. [[≈ 28 | intrusions per hour (our arithmetic)]]
  4. Then check its answers. Only 0.4730 of them were judged correct (Table 3). In all 100 checked scenes, the answer had been said aloud (Table 11).
  5. 🧠 Verdict: not yet, unattended. Use Duplex-MPE as your acceptance test instead. We'd want Shush above 0.99 and Answer above 0.9.

Rates are real (Table 3). Per-hour counts are our arithmetic: 20,360 windows over 2,000 scenes of 99.5 s (Table 6, Table 19).

Not ready to sit unattended in a shared room. But it gives you a ready yardstick for your own duplex model.

Check 1 From page 7.

In the taco scene, which turn is the single T request?

  • Turn 5: "this is exactly why we keep Aria in here"
  • Turn 4: asking Aria which option is cheapest per person
  • Turn 6: "Hey Siri, how long is the bus?"
  • Turn 10: asking Aria whether the taco place takes card

Turn 4 is the only T. Turn 5 mentions Aria without asking (N1), turn 6 targets another device (N2), and turn 10 is an N4 question that a human resolves (Table 4).

Check 2 From pages 6 and 16.

Gemini drops 64 points without the name; speech models don't move. What follows?

  • The speech models infer addressees from context as well as from names
  • The implicit requests were badly written
  • The name doesn't change the speech models' decision, for reasons this test can't separate
  • The speech models handle implicit requests better than Gemini

No detected difference is not equal skill. Context inference and ignoring the name both fit the data (§4.2, App. A5.1).

Check 3 Spot the flaw. From pages 13 and 17.

"Voila's Stop is 0.8484, so it's best at giving up the floor." What's wrong?

  • Nothing: 0.8484 beats MiniCPM-o's 0.5964
  • The rate covers only 620 eligible events; MiniCPM-o yields far more often overall
  • Stop also penalises wrong answers, which Voila avoids by rarely answering
  • Voila's Stop was measured on implicit requests only

Read a conditional rate with its denominator. Voila yields on 526 events, MiniCPM-o on 974, out of the same 1,911 (Table 15).