arXiv · 2026 · Preprint
Yinhao Bai et al. (JD.com) · → Paper · Demo: ✗ · Code: ✗
JD.com’s technical report on a modular full-duplex voice agent. A 48.9B-parameter MoE Thinker is trained jointly on speech and text from mid-training through DPO. A plug-and-play state-token gate (Joy-Duplex) decides when the agent may speak. The evaluation covers text, speech-understanding, empathy and full-duplex turn-taking benchmarks, but never the spoken output itself.
Problem
End-to-end speech LMs tend to lose text-domain reasoning when speech is added during multimodal training (“cognitive degradation”). They also give flat responses that ignore speaker attributes such as age, gender and emotion. Full-duplex systems built on energy-based VAD or dual-stream modeling either react to background speech and backchannels or fail to yield when the user really interrupts. The authors argue that putting fine-grained turn-taking inside the main backbone, as dual-stream Moshi-style models do, also dilutes its reasoning capacity.
Method
JoyAI-Talker separates three functions into modules: duplex state coordination, a Thinker for understanding and reasoning, and a Talker for speech generation (§2).
Thinker. An encoder-projector-decoder speech LM (§2.1). The speech encoder is an attention-based encoder-decoder pretrained on ASR. It downsamples log-Mel filter-bank features 8x with Conv2D blocks to a 12.5 Hz token rate, and MLP projectors feed it into JoyAI-LLM Flash. That backbone is a 40-layer sparse MoE (one dense layer plus 39 MoE layers, 256 experts, Top-8 routing plus a shared expert, MLA attention) with 48.9B total and about 3.28B active parameters. Speech-text joint training starts at mid-training instead of being postponed to SFT, and runs through four stages (§2.1.1):
- joint mid-training, first as an adapter-only warmup with the encoder and LLM frozen, then as full-parameter training;
- joint context extension to 64K tokens;
- joint SFT, which includes S2T data synthesized with an in-house TTS model;
- joint DPO on self-generated text responses scored by LLM and rule-based judges.
For speech prompts, preference pairs are built only within the speech-conditioned candidate pool, not pooled with text-conditioned candidates. The authors call this a “speech-only strategy”. The loss is masked to assistant turns only (§2.1.2). The Persona-Adaptive Empathetic Response (PAER) framework makes the Thinker infer gender, age and emotion from the input audio in a chain-of-thought step. The Thinker then emits response text plus an optional <instruct> speaking instruction and paralinguistic tokens (§2.1.3, §2.2.8).
Talker. The Talker reuses the JoyVoice architecture (§2.2). A causal autoregressive Transformer predicts supervised speech tokens from a speaker embedding, a natural-language local instruction and the text. Its hidden states condition a dynamic-chunk Diffusion Transformer that predicts mel-spectrograms, and a vocoder produces the waveform. Instructions are hierarchical: a global instruction covers speaker-level traits and a local instruction covers emotion, rate, breathiness and similar per-utterance traits. Ten paralinguistic special tokens (for example [Laughter], [Sigh], [Cough]) are added to the vocabulary (§2.2.2 to §2.2.4). Speaker-specific SFT absorbs the global instruction into one fixed assistant voice. Its training data comes from “style borrowing”: expressive movie and TV performances are voice-converted into the target voice, so the target speaker does not have to record every style (§2.2.5). The Talker accepts streaming text input and produces streaming speech output (§2.2.7).
Joy-Duplex. An external gating module that wraps the half-duplex dialogue model without changing its parameters (§2.3). A streaming audio encoder with dynamic-chunk attention processes 160 ms chunks. A 1.7B decoder-only LM then decodes interleaved incremental transcript tokens, an <|asr_eos|> boundary token, and state tokens (<|partial|>, <|complete|>, <|backchannel|>, <|accept|>, <|reject|>). The downstream model responds only after an <|accept|> token. An optional speaker-verification plugin can restrict activation to the authorized user (§2.3.2, §2.3.3).
Key Results
Text capability (Table 1). The only baseline is Qwen3-Omni (32B, 3B active). JoyAI-Talker-DPO scores highest of the three columns on 13 of 14 benchmarks, for example MATH 94.62 vs 92.04, GPQA 64.51 vs 58.93, AGIEval 54.22 vs 42.87 and IFBench 31.83 vs 26.68. Qwen3-Omni is highest on GSM8K (95.98 vs 95.60 DPO and 95.53 SFT). The SFT checkpoint falls below Qwen3-Omni on C-Eval (87.31 vs 87.79), MMLU (87.72 vs 87.90), CMMLU (85.14 vs 85.87) and GSM8K. DPO scores above SFT on all 14 rows.
Speech understanding (Table 2). Both JoyAI-Talker checkpoints have lower WER than Qwen3-Omni on all ten ASR rows. The DPO checkpoint scores 0.86 vs 1.04 on AISHELL-1, 1.32 vs 1.52 on LibriSpeech clean and 9.39 vs 11.27 on GigaSpeech. On LibriSpeech other, SFT is lowest (2.55, with DPO at 2.58). Qwen3-Omni stays ahead on CoVoST 2 zh2en (28.50 vs 27.95 DPO) and AlpacaEval (4.68 vs 4.61 DPO), and ties DPO on CommonEval (4.53). JoyAI-Talker is ahead on en2zh translation (CoVoST 2 51.29 SFT, 50.37 DPO, 44.80 Qwen3-Omni), OpenBookQA, MMSU and SD-QA. For speaker-attribute perception, gender is 98.25 SFT / 98.20 DPO vs 97.59, age 77.30 DPO vs 65.20, and emotion 79.57 DPO vs 77.00.
Perception inside chain-of-thought (Table 3). Accuracy drops when attributes are read from the model’s own CoT during dialogue instead of being asked directly: gender falls from 98.2 to 89.3 and emotion from 79.6 to 72.8. Age is flat (77.3 to 77.7). Emotion is scored on the authors’ own re-annotation of MER2025, so that column cannot be compared with published MER2025 results.
Empathetic responses (Table 4). An LLM judge scores the text responses on three English EchoMind subsets. JoyAI-Talker beats Qwen3-Omni on all six cells: weighted scores of 9.38/9.08/9.79 vs 8.91/8.91/9.47, and empathy scores of 8.28/7.49/9.73 vs 7.71/7.03/9.20 (Emotion/Age/Gender). No human raters are involved, and the acoustic side of the responses is not scored.
Full duplex (Table 5). On Full-Duplex-Bench v1.5, Joy-Duplex gives the best value in all eight cells. User interruption: C_RESPOND 0.88 (next best GPT-4o 0.78) and C_RESUME 0.07 (next GPT-4o 0.10). Backchannel: C_RESPOND 0.01 (next 0.02, Moshi and Gemini 3.1 Live) and C_RESUME 0.96 (Gemini 3.1 Live 0.95). Talking to other: C_RESPOND 0.17 (Moshi 0.20) and C_RESUME 0.72 (Gemini 0.66). Background speech: C_RESPOND 0.10 (Moshi 0.21) and C_RESUME 0.85 (Gemini 0.66). The Freeze-Omni, Moshi and GPT-4o numbers are copied from the benchmark paper. The authors ran Gemini 3.1 Live and Joy-Duplex themselves. Joy-Duplex is a gating front end, while the baselines are complete end-to-end dialogue systems. No turn-taking latency is reported, although §2 concedes that the modular design adds latency.
Speech tool calling (Table 6). JoyAI-Talker is best on only one of five benchmarks, Speech-ACEBench Single (98.56 vs 96.63 for Qwen3.6 text-only and 92.79 for Qwen3-Omni). The text-only Qwen3.6 leads on ACEBench Parallel (61.36 vs 44.32), BFCL Single (94.81 vs 89.62) and SmartInteract (83.08 vs 80.39). Qwen3-Omni leads on BFCL Parallel (84.68 vs 78.72).
Novelty Assessment
The report mostly integrates existing parts at large scale. Thinker-Talker decoupling follows earlier omni models. The Talker is JoyVoice with instruction-aware fine-tuning. Plug-and-play state-token duplex predictors have direct precedents that the paper cites (Easy Turn, SoulX-Duplug, FastTurn). Joy-Duplex’s interleaving of incremental ASR text, an <|asr_eos|> boundary and an explicit accept/reject gate is a specific design choice, but no ablation isolates its effect. Two smaller ideas are more distinctive: building DPO pairs only within the speech-conditioned pool, and creating expressive target-voice SFT data by voice-converting performances from other speakers. The paper does not test either one with an ablation. The “cognitive degradation” claim is never measured against the text-only JoyAI-LLM Flash backbone. Table 1 compares only with Qwen3-Omni, so the amount of text capability retained is not shown.
Field Significance
Moderate — The report gives a large-scale data point for a modular voice-agent design: a separate small gating LM handles turn-taking for an MoE speech LM whose parameters it does not change, with strong reported Full-Duplex-Bench v1.5 behaviour. Its evidence covers understanding, text-response empathy and turn-taking decisions only, so it says nothing about the quality or controllability of the generated speech.
Claims
- supports: A separate streaming gating model that decodes incremental transcripts with explicit interaction-state tokens can handle overlapping speech in full-duplex dialogue, yielding to interruptions and holding the floor against backchannels and background speech, without changing the dialogue model’s parameters.
Evidence: Joy-Duplex, a 1.7B decoder that emits
<|partial|>,<|complete|>,<|backchannel|>and<|accept|>/<|reject|>tokens, gives the best value in all eight Full-Duplex-Bench v1.5 cells, including interruption C_RESPOND 0.88 and background-speech C_RESPOND 0.10 against Freeze-Omni, Moshi, GPT-4o and Gemini 3.1 Live. (§2.3.2, §3.4.2, Table 5) - refines: Training speech and text jointly from mid-training onward can let a speech LM match or exceed a comparable omni model on text reasoning, but whether this avoids degradation relative to its own text backbone has to be measured against that backbone.
Evidence: JoyAI-Talker-DPO leads Qwen3-Omni on 13 of 14 text benchmarks (for example MATH 94.62 vs 92.04), but no row reports the text-only JoyAI-LLM Flash backbone, and the SFT checkpoint trails Qwen3-Omni on C-Eval, MMLU, CMMLU and GSM8K. (§2.1.1, §3.1, Table 1)
- complicates: Speaker-attribute perception in speech LMs can weaken when the attributes are inferred inside a chain-of-thought dialogue step instead of being queried directly.
Evidence: JoyAI-Talker’s gender accuracy falls from 98.2 (Direct) to 89.3 (CoT) and emotion accuracy from 79.6 to 72.8, while age stays at 77.3 vs 77.7. (§3.3.1, Table 3)
- supports: Preference optimization on a speech LM’s text responses to spoken prompts can improve speech-conditioned reasoning beyond supervised fine-tuning.
Evidence: DPO with preference pairs built only within the speech-conditioned candidate pool raises OpenBookQA from 93.84 to 94.73, MMSU from 82.40 to 84.65 and SD-QA from 73.96 to 75.77 over the SFT checkpoint, while three of four translation scores fall slightly (for example CoVoST 2 en2zh 51.29 to 50.37). (§2.1.1, §3.2, Table 2)
- complicates: A speech LM’s spoken-language agent ability can still trail a text-only model of similar active size on multi-call and parallel tool use.
Evidence: JoyAI-Talker beats the text-only Qwen3.6 only on Speech-ACEBench Single (98.56 vs 96.63) and trails it on ACEBench Parallel (44.32 vs 61.36), BFCL Single (89.62 vs 94.81) and SmartInteract (80.39 vs 83.08). (§3.5, Table 6)
Limitations and Open Questions
Warning
The Talker’s speech output is never evaluated. The report has no TTS quality, intelligibility, speaker-similarity, instruction-following, paralinguistic-event, streaming-latency or human listening results. §2.1.3 states that the system achieves superior “acoustic empathy metrics”, and §3.3.2 says acoustic realization is “evaluated separately”, but no table or section reports such results. All claims about expressive, instruction-controlled speech are therefore descriptive only.
- Nothing has been released (no code, weights or demo link), and the training data is in-house. Results cannot be reproduced.
- Full-duplex comparisons mix numbers copied from the benchmark paper with runs by the authors, and they compare a gating front end with complete end-to-end systems. Turn-taking latency is not reported, although §2 acknowledges a latency penalty from modular routing.
- The empathy evaluation relies only on an LLM judge scoring text responses. Emotion recognition uses the authors’ re-annotated MER2025 labels.
- The text-capability comparison omits the JoyAI-LLM Flash backbone, so the paper never directly measures how much text capability the speech training costs.
- None of the components (speech-only DPO pairing, VC-based style borrowing, the
<|asr_eos|>state interleaving) is ablated. - The authors name three open issues (§5): no general audio or environmental-sound understanding, a large-scale RL stage that is still in progress, and the modular duplex design versus end-to-end dual-stream modeling, which they plan to compare.
Wiki Connections
- Spoken Language Model — The Thinker is an encoder-projector MoE speech LM trained jointly on speech and text from mid-training through DPO, evaluated on ASR, speech QA, translation and speaker-attribute perception.
- Speech-to-Speech — JoyAI-Talker is a full-duplex spoken dialogue system whose state-token gating is evaluated on overlap handling in Full-Duplex-Bench v1.5.
- RLHF Speech — The Thinker undergoes DPO with LLM and rule-based judges, with preference pairs for spoken prompts built only within the speech-conditioned candidate pool.
- Moshi — Dual-stream full-duplex baseline whose design the paper argues against; it trails Joy-Duplex on all Full-Duplex-Bench v1.5 cells.
- Freeze-Omni — Full-duplex baseline on Full-Duplex-Bench v1.5, with numbers taken from the benchmark paper.
- Qwen3-Omni — Main baseline for the text, speech-understanding, empathy and tool-calling comparisons.
- Full-Duplex-Bench v1.5 — Benchmark for the overlap-handling evaluation and source of the cited baseline numbers.
- JoyVoice — Supplies the Talker’s autoregressive Transformer plus Diffusion Transformer architecture and its speech data recipe.
- EchoMind — Its Emotion, Age and Gender subsets are used for the LLM-judged empathetic-response evaluation.
- AIR-Bench — Benchmark for the gender and age perception tests.
- SoulX-Duplug — Earlier plug-and-play streaming state predictor for full duplex, cited as a precedent for Joy-Duplex’s modular design.