LREC 2026 · 2026 · Conference
Wataru Nakata et al. (University of Tokyo) · → Paper · Demo: ✓ · Code: ?
Builds and releases J-CHAT, a 76,000-hour Japanese spoken dialogue corpus mined automatically from YouTube and podcasts, and shows that a textless spoken dialogue language model trained on the full corpus is rated more natural and meaningful than the same model trained on either source alone.
Problem
End-to-end spoken dialogue systems need tens of thousands of hours of dialogue audio: the authors point to roughly 20k hours used for the generative spoken dialogue language model dGSLM and 100k hours for SoundStorm dialogue synthesis. Open dialogue corpora are much smaller (CallHome-JP, the largest spontaneous Japanese dialogue set, is 49 hours) or are scripted studio recordings (STUDIES, 8.2 hours), and the largest open dialogue resource, Seamless Interaction (about 4k hours), was recorded manually, which the authors argue is too costly to repeat for other languages. Large web-mined corpora such as GigaSpeech or LibriTTS exist, but they are segmented at the utterance level for ASR or TTS and lose turn structure and cross-turn coherence (§1, §2, Table 1).
Method
The contribution is an automated, language-independent corpus construction pipeline plus the corpus it produced (§3, Figure 1).
- Collection. About 600k YouTube audio files (about 180k hours) were found by searching random Wikipedia page titles, and about 880k podcast episodes (about 140k hours) were downloaded from the RSS feeds of every PodcastIndex station labelled Japanese (§3.1).
- Language filtering. Whisper’s language-identification model keeps segments with Japanese probability above 0.8, retaining 55.7% of YouTube and 84.7% of podcast audio (§3.2.1).
- Dialogue extraction. PyAnnote speaker diarization yields speaker-labelled turns. Audio is split into dialogues at silences of 5 seconds or more, and dialogues in which one speaker holds more than 80% of the speaking time are discarded as monologues; any dialogue with two or more speaker IDs is kept. This retained 41.9% (YouTube) and about 45.0% (podcast) of the Japanese audio (§3.2.2).
- Cleansing. The Demucs source separation model is run on all audio to remove background music (§3.3).
- Transcription. Transcripts with subword alignments come from the reazonspeech-nemo-v2 ASR model, intended for systems such as Moshi that train on text alongside speech (§3.4).
The result totals 76,036 hours in 5,424,514 dialogues: 11,017 hours from YouTube and 65,019 hours from podcasts. Mean dialogue length is 50.23 s with 10.10 turns and 3.14 speakers (Table 2). The corpus ships with predefined train/valid/test splits and is released under CC BY-NC 4.0 for “information analysis” under Article 30-4 of the Japanese Copyright Act (§1, §4.1, §7).
Validation model. To test whether the corpus is usable for training, the authors train dGSLM, a textless spoken dialogue language model, using its official implementation and original configuration (§5). Speech is discretised with k-means (1,000 clusters) over features from a pretrained Japanese HuBERT. A two-tower unit language model is trained autoregressively on two-channel unit streams. The two channels are built by switching the output channel at each diarized turn-taking event, because J-CHAT audio is single-channel (§5.1). Training ran for 100k steps on 32 V100 GPUs with 36,864-token batches and up to 3,000 tokens (about 1 minute) per sample (§5.2). A HiFi-GAN vocoder conditioned on x-vectors converts units back to speech. It is trained on the read-speech JVS and JVNV corpora, not on J-CHAT (§5.2). Three dialogue LMs are trained: one on the YouTube subset, one on the podcast subset and one on all of J-CHAT. A resynthesis condition (units to vocoder, with no language model) serves as an upper reference (§5).
Key Results
Dialogue generation (Table 3, verified against the table image). Each model received the first 5 seconds of a J-CHAT test dialogue and generated the next 25 seconds with beam search (beam 5). The two channels were voiced as JVS speakers 001 (male) and 002 (female). Each MOS test used 60 native Japanese listeners rating 8 samples each, following the dGSLM protocol (§5.3).
| System | Naturalness | Meaningfulness |
|---|---|---|
| resynth | 2.55 ± 0.18 | 2.48 ± 0.18 |
| dGSLM-YouTube | 1.44 ± 0.13 | 1.56 ± 0.14 |
| dGSLM-podcast | 1.44 ± 0.13 | 1.52 ± 0.13 |
| dGSLM-J-CHAT | 2.28 ± 0.19 | 2.18 ± 0.19 |
The model trained on all of J-CHAT clearly beats both single-source models, with no overlap between the 95% confidence intervals. The YouTube and podcast models are statistically indistinguishable, even though the podcast subset has about 4.3 times as many dialogues (Table 2). The authors read this as evidence that scale alone is not enough (§5.4). A significant gap to resynthesis remains on both axes, and the authors say the generated dialogue “occasionally produces sensible words” but often lacks coherence (§5.4). All absolute scores are low: even resynthesis falls below 2.6, so the unit and vocoder pipeline itself caps perceived quality.
Acoustic quality (§4.2). Mean NISQA scores, an automatic quality predictor rather than a listening test, are 2.99 for the Podcast subset and 2.37 for YouTube. These sit above telephone-recorded CallHome-JP (1.98) and below studio-recorded STUDIES (4.01).
Diversity (§4.3, Figures 2 and 3). Over 120 sampled dialogues per corpus, the mean pairwise cosine similarity of transcript sentence embeddings is lower for J-CHAT YouTube (0.2390) and Podcast (0.3457) than for STUDIES (0.5186) and CallHome-JP (0.6164), which the authors take as broader topic coverage. In a t-SNE of frame-level HuBERT features, both J-CHAT subsets and CallHome-JP cover regions that scripted STUDIES does not reach. The authors read this as evidence that J-CHAT contains spontaneous-speech characteristics. The evidence is qualitative.
Novelty Assessment
The pipeline is built entirely from existing tools: web search seeding as in JTubeSpeech, Whisper language ID, PyAnnote diarization, Demucs and an off-the-shelf ASR model. The new elements are the dialogue-unit segmentation and monologue filter, the use of podcast directories as a high-yield dialogue source, and above all the scale of the open corpus (76k hours, more than 1,000 times CallHome-JP). The validation experiment reuses dGSLM without modification. The contribution is a data resource, not a modelling advance. The most informative experimental finding is that combining sources helps a dialogue LM where a larger single source did not.
Field Significance
moderate: J-CHAT provides an open Japanese dialogue corpus at the tens-of-thousands-of-hours scale that the authors argue end-to-end spoken dialogue models need, together with a recipe for reproducing it in other languages from web audio. Its own validation is limited to one textless dialogue LM with low absolute MOS, so the paper shows that the corpus is usable for training rather than that it yields strong dialogue models.
Claims
- supports: Web-mined, automatically filtered in-the-wild audio can supply open spoken dialogue corpora at the scale needed to train end-to-end spoken dialogue language models, avoiding the cost of manual dialogue recording.
Evidence: A pipeline of language ID, diarization-based dialogue segmentation with a single-speaker-dominance filter, and music removal turns about 320k hours of YouTube and podcast audio into 76,036 hours of Japanese dialogue in 5.4M dialogues. A dGSLM trained on it reaches naturalness MOS 2.28 against 2.55 for resynthesis. (§3, Table 2, Table 3)
- refines: For spoken dialogue language models, source diversity in the training corpus can matter more than raw data volume from a single source.
Evidence: dGSLM trained on the podcast subset (about 4.3 times as many dialogues as YouTube) scores the same as the YouTube-trained model (naturalness 1.44 vs 1.44, meaningfulness 1.52 vs 1.56). The model trained on both sources scores 2.28 / 2.18, with non-overlapping 95% CIs. The combined model also has more total training data, so diversity and volume are not fully separated. (§5.4, Table 3)
- complicates: Textless unit-based dialogue language models trained on large in-the-wild corpora still produce largely incoherent dialogue, with quality limited by both the language model and the unit-to-speech pipeline.
Evidence: Even the best J-CHAT-trained dGSLM is significantly below resynthesis on naturalness (2.28 vs 2.55) and meaningfulness (2.18 vs 2.48). The authors note that generated dialogue only occasionally contains sensible words. Resynthesis itself, through a vocoder trained on read speech, scores only 2.55. (§5.4, Table 3)
- complicates: Web-sourced dialogue audio remains measurably noisier than studio dialogue corpora even after source separation, and podcast sources are cleaner than video platforms.
Evidence: After Demucs cleansing, mean NISQA is 2.99 (Podcast) and 2.37 (YouTube), against 4.01 for studio-recorded STUDIES and 1.98 for telephone CallHome-JP. (§4.2)
Limitations and Open Questions
The two-channel dialogue streams are reconstructed from single-channel audio
J-CHAT is mono. Channels for dGSLM are assigned by switching at diarized turn boundaries, yet dialogues average 3.14 speakers (Table 2) and the paper does not say how three or more speakers, or overlapping speech, are mapped onto two channels. Overlap and backchannel modelling, a central motivation for dialogue LMs, therefore depend on diarization quality that is never evaluated (§3.2.2, §5.1).
- Narrow validation. The corpus is validated with only one textless model, dGSLM, using one vocoder and two fixed JVS voices. No transcript-conditioned model such as Moshi is trained, and no baseline dGSLM is trained on another corpus (for example CallHome-JP), so the gain over existing data is not measured directly.
- Volume and diversity are confounded. The full-corpus model has more training data (about 68k train hours) than the podcast-only model (about 57k), so the conclusion’s statement that multi-source training “significantly improves” results cannot be attributed to source diversity alone (§4.1, §6).
- Unvalidated pipeline components. Diarization accuracy, Demucs artifacts and ASR transcript error rates are not reported. The claim that the podcast noise level is “sufficiently low for use in speech synthesis” rests on NISQA alone, with no TTS experiment (§4.2).
- Proxy diversity measures. Topic and phonetic diversity are measured on 120 dialogues per corpus with sentence-embedding cosine similarity and a t-SNE plot. These are coarse proxies, and the phonetic analysis is qualitative (§4.3).
- Licensing. Release under CC BY-NC 4.0, justified by Japanese copyright law, may limit commercial use and reuse outside Japan (§1, §7).
- Minor reporting gap. The split-duration sentence in §4.1 gives three numbers for four splits (train/valid/test/other), so the size of the “other” split cannot be recovered from the text.
Wiki Connections
- Spoken Language Model — Provides a 76k-hour open Japanese training corpus for spoken dialogue language models and validates it by training a textless, unit-based dialogue LM (dGSLM) that continues real dialogue audio prompts.
- Speech-to-Speech — Falls in the dialogue sub-paradigm: the validation model generates two-channel spoken dialogue continuations directly from speech units, without text.
- Self-Supervised Speech — The validation model’s speech-to-unit stage uses k-means (1,000 clusters) over a pretrained Japanese HuBERT, and HuBERT features also underpin the phonetic-diversity analysis.
- GAN Vocoder — Trains an x-vector-conditioned HiFi-GAN unit vocoder on JVS and JVNV, whose resynthesis MOS (2.55) sets the quality ceiling for the dialogue samples.
- Subjective Evaluation — Rates naturalness and meaningfulness MOS with 60 native Japanese listeners per test, following the dGSLM protocol, and reports 95% confidence intervals.
- Moshi — Cited as a dialogue model that needs transcripts in training, which motivates J-CHAT’s ASR transcripts with subword alignments; no Moshi model is trained in this paper.
- HiFi-GAN — Provides the vocoder architecture, trained here with x-vector speaker conditioning to turn HuBERT units back into speech.
- SoundStorm — Cited for its roughly 100k hours of dialogue training data, used to argue that end-to-end dialogue synthesis needs corpora of tens of thousands of hours.
- Whisper — Its language-identification model filters the crawled audio to Japanese (probability above 0.8).
- GigaSpeech — Compared in Table 1 as a large open in-the-wild corpus that is segmented at the utterance level and so lacks dialogue structure.
- JVS Corpus — Supplies vocoder training data and the two fixed voices (JVS001, JVS002) used to render the dialogue channels in the listening tests.
- Effects of Dialogue Corpora Properties on Fine-Tuning Moshi — A later paper that pre-trains a Moshi-based spoken dialogue model on J-CHAT and a quality-filtered J-CHAT subset, then compares fine-tuning curricula over Japanese dialogue corpora.
- Towards a Japanese Full-duplex Spoken Dialogue System — A later paper that pre-trains J-Moshi on about 60k hours of J-CHAT, converting the mono audio to two-channel stereo with speaker diarization.