arXiv · 2026 · Preprint
Kyudan Jung et al. (KAIST AI, NAVER Cloud) · → Paper · Demo: ✓ · Code: ✓
Introduces an open-source, scalable audio pre-processing pipeline that curates multi-turn, multi-speaker conversational speech data for training full-duplex speech language models, and validates it by fine-tuning Moshi on the resulting corpus.
Problem
Full-duplex spoken dialogue models such as Moshi need to learn how to listen and speak simultaneously, which requires training data that captures overlapping speech, backchannels, and rapid turn-taking. Existing large-scale speech corpora fall short of this need: web-scale ASR-oriented pipelines (WenetSpeech, Emilia) are optimized for single-stream, non-overlapping speech and treat overlaps as noise to be filtered out, while small conversational corpora with genuine multi-speaker dynamics (Fisher, Switchboard) are narrowband, dated, and too limited in scale for large-scale pre-training or fine-tuning. Building suitable full-duplex training data from in-the-wild recordings is further complicated because real conversations trigger diarization errors and ASR hallucinations, and because unlike text-pipeline efforts (Dolma, RedPajama, FineWeb), speech data curation “recipes” for speech language models remain largely proprietary and undocumented.
Method
Sommelier is a modular, toggleable data-processing pipeline that converts raw, in-the-wild conversational audio (radio, podcasts) into structured multi-turn training corpora while deliberately preserving overlaps and backchannels rather than removing them. The pipeline has five stages. First, audio standardization converts inputs to a common 16kHz/16-bit mono format with loudness normalization, then segments long recordings into sub-five-minute chunks using voice activity detection to avoid out-of-memory failures in downstream models. Second, speaker diarization identifies who speaks when; the authors replace the commonly used pyannote 3.1 model with NVIDIA’s Sortformer, motivated by its stronger handling of short utterances such as backchannels. Third, overlapping speech is handled by extracting non-overlapping reference segments per speaker, computing speaker embeddings, feeding the overlapped interval into a two-speaker separation model (SepReformer), and using cosine similarity between the separated candidates and the reference embeddings to assign each separated stream to the correct speaker before re-concatenating with the non-overlapping context. Fourth, an optional background-music removal step uses a pretrained audio tagger (PANNs) to flag segments likely to contain music, applying a source-separation model (Demucs) only to those segments to avoid unnecessary quality loss elsewhere. Fifth, transcription uses a three-model ROVER ensemble (Whisper, Canary, Parakeet) with word-level majority voting, defaulting to Whisper on ties, followed by an n-gram repetition filter to prune residual hallucinations; word-level timestamps are extracted via forced alignment for token-level synchronization required by streaming speech LMs.
To validate the pipeline’s downstream utility, the authors LoRA fine-tune the open-source Moshi model (moshiko-pytorch-bf16) on 83 hours of Sommelier-processed data, restricting training to short-turn conversational segments (individual turns capped at 10 seconds, valid regions requiring at least three consecutive turns) after finding that longer single-speaker turns destabilized training.
Key Results
Fine-tuning Moshi on Sommelier-processed data improved performance on Full-Duplex-Bench 1.0 across backchanneling, smooth turn-taking, and user interruption handling, while pause handling stayed roughly at base-model levels. On Full-Duplex-Bench v1.5, the fine-tuned model showed large post-distractor audio-quality gains (e.g., SI-SDR on the “Background Speech” scenario improved from 5.43 dB to 20.76 dB) and markedly reduced stop/response latencies across scenarios. In component-level ablations, Sortformer outperformed the pyannote 3.1 baseline on VoxConverse diarization (DER 7.16% vs. 8.4%, JER 14.69% vs. 17.68%), with the largest relative gains on short-duration and turn-boundary regimes. The three-model ASR ensemble reduced WER on LibriSpeech test-other by about 37% relative to standalone Whisper-large-v3 (6.26% → 3.92%) and qualitatively eliminated repetitive Whisper hallucinations. The overlap-separation module consistently improved WER, STOI, and UTMOS over the raw mixed-signal baseline across nine synthesized SIR/overlap conditions, with UTMOS approaching the oracle (clean-source) upper bound even at full overlap. On a single A100, the full pipeline processes audio at an RTF of 0.1746 (0.133 without optional denoising), which the authors project to about 55 hours to process 10,000 hours of audio using eight A100 GPUs with parallel processes.
Novelty Assessment
Sommelier’s contribution is primarily an engineering integration: none of its individual components (Sortformer diarization, SepReformer separation, PANNs-gated Demucs denoising, ROVER ASR ensembling) are novel in isolation, and the paper is explicit that its role is analogous to open text-pipeline efforts (Dolma, RedPajama, FineWeb) for the speech domain. The genuine contribution is the systematic combination and validation of these components specifically for full-duplex conversational data, including a concrete design choice (preserving rather than discarding overlaps and backchannels) that differs from prior large-scale speech-corpus pipelines, plus a from-scratch empirical evaluation of each pipeline stage (diarization model choice, ASR ensembling, overlap-separation quality) and an end-to-end demonstration that the resulting data measurably improves a fine-tuned full-duplex model’s turn-taking behavior. It is a data-engineering and validation contribution rather than an architectural one.
Field Significance
moderate — the paper targets a genuine and previously under-addressed bottleneck for full-duplex speech language model training: the scarcity of open, well-documented pipelines for curating multi-speaker conversational data with realistic overlap and backchannel structure. It provides useful, reproducible engineering evidence (component-level ablations plus an end-to-end fine-tuning demonstration) rather than a new modeling paradigm, and its downstream validation is limited to a single base model and benchmark family.
Claims
- supports: Open-source data-curation pipelines can measurably improve full-duplex conversational model behavior through better training data alone, without changes to model architecture.
Evidence: LoRA fine-tuning Moshi on 83 hours of pipeline-processed data improved Full-Duplex-Bench 1.0 scores on backchanneling, smooth turn-taking, and user interruption handling relative to base Moshi. (§3.1, Table 1)
- supports: Preserving overlapping speech and backchannels in training data, rather than filtering them out as noise, better equips full-duplex speech language models to model realistic conversational dynamics.
Evidence: the pipeline explicitly retains overlap and backchannel regions through diarization-guided separation and re-concatenation, in contrast to prior large-scale corpora (WenetSpeech, Emilia) whose pipelines were optimized for single-stream speech and excise overlaps. (§2, §A.2)
- refines: Diarization models tuned to standard benchmarks are not automatically well suited to conversational speech, where robustness to short utterances and rapid speaker turns matters more than aggregate accuracy.
Evidence: on the VoxConverse common subset, Sortformer outperformed pyannote 3.1 on overall DER/JER, with the gap most pronounced on short-duration segments (≤1.0s) and turn-taking boundary regions. (§3.2, Table 2)
- supports: Ensembling multiple ASR models with output-level voting reduces the transcription hallucinations that a single ASR model produces in noisy or silent conditions, improving the quality of derived training labels.
Evidence: a ROVER ensemble of Whisper, Canary, and Parakeet reduced WER on LibriSpeech test-other by about 37% relative to standalone Whisper-large-v3 (6.26% to 3.92%) and corrected qualitative repetition hallucinations such as “Yeah., Yeah., Yeah…”. (§2.5, §3.4, Table 4)
- complicates: Automated speaker separation for overlapping speech recovers most, but not all, of the perceptual quality and intelligibility of cleanly recorded (non-overlapping) audio, with the gap widening as overlap severity increases.
Evidence: under synthesized overlap conditions, separated-speech UTMOS and STOI approached but did not fully reach oracle (clean-source) values, and the baseline mixed-signal condition degraded sharply as overlap ratio increased toward 1.0. (§3.3, Tables 3 and 8)
Limitations and Open Questions
Downstream validation is limited to a single full-duplex model (Moshi, via LoRA fine-tuning on 83 hours of data) evaluated on one benchmark family (Full-Duplex-Bench 1.0/1.5); it is unclear whether the pipeline's benefits generalize to other full-duplex architectures, larger training scales, or languages beyond English.
The authors themselves note that the pipeline focuses exclusively on speech and does not account for non-speech acoustic events or general sound scenes, limiting its scope relative to omni-modal audio processing approaches. They also acknowledge that separated overlapping-speech audio is inevitably slightly inferior in fidelity to originally isolated, distinct-channel recordings, since the artificial separation process can introduce minor acoustic artifacts. Separately, fine-tuning on Sommelier-processed data did not improve pause handling relative to base Moshi, which the authors attribute tentatively to the Moshi architecture itself or to the absence of prompt audio conditioning, without further diagnosis.
Wiki Connections
- Spoken Language Model — targets the full-duplex speech language model paradigm (exemplified by Moshi) by addressing its core training-data bottleneck rather than its architecture.
- Speech-to-Speech — the pipeline’s downstream validation directly targets the dialogue sub-paradigm, improving a full-duplex spoken dialogue model’s turn-taking, backchanneling, and interruption handling.
- Evaluation Metrics — contributes systematic component-level benchmarking (diarization model comparison, ASR ensemble evaluation, overlap-separation quality assessment across SIR/overlap conditions) for full-duplex data-curation pipelines.
- Moshi — the full-duplex speech-text foundation model that Sommelier-processed data is used to LoRA fine-tune, serving as the paper’s primary downstream validation target.
- Full-Duplex-Bench — the turn-taking benchmark used to evaluate base vs. Sommelier-fine-tuned Moshi in the paper’s main results.
- Full-Duplex-Bench v1.5 — the updated overlap-handling benchmark used for the paper’s extended latency and audio-quality evaluation in Appendix D.3.
- Qwen3-Omni — its Captioner variant is used by Sommelier to generate context-aware audio captions (emotion, gender, situation) as pipeline metadata.
- Ming-UniAudio — cited as an example speech LLM that benefits from rich captioned metadata of the kind Sommelier’s context-aware captioning stage produces.
- SpeechWeave — cited as a prior speech-data generation pipeline; contrasted with Sommelier’s focus on curating real multi-speaker conversational audio rather than synthesizing single-speaker TTS training data.