arXiv · 2025 · Preprint
Chu et al. (Olewave) · → Paper · Demo: ? · Code: ✗
A large-scale tiered conversational speech dataset (OleSpeech-IV) with 5,000+ hours of multispeaker, multilingual recordings at the Tier IV level, accompanied by a proprietary speech-to-text alignment module (Olign) that produces calibrated word-level timestamps and handles speaker overlap.
Problem
Large-scale conversational speech datasets with reliable speaker labels, fine-grained timestamps, and calibrated confidence scores are scarce. Public sources like podcasts and teleconferences contain naturalistic multi-party speech, but converting them into high-quality training corpora is difficult: end-to-end ASR aligners produce boundary errors and overconfident posterior scores, speaker overlap confounds word-level annotation, and manual labelling at scale is prohibitively costly. OleSpeech-IV addresses this gap by combining human-sourced speaker labels with a proprietary alignment pipeline designed to handle these failure modes.
Method
The dataset is organized into four tiers of increasing annotation depth. Tier I covers 1M+ hours of untranscribed audio with language and topic labels. Tier II adds machine-generated transcripts with word and utterance confidence scores (200K+ hours). Tier III replaces automatic transcripts with human-validated ones (40K+ hours). Tier IV (OleSpeech-IV) builds on Tier III with full speaker diarization, word-level timestamps, translated transcripts, and custom annotation layers (5,000+ hours). All audio originates from publicly available English podcasts, talk shows, teleconferences, and conversations spanning 12 languages and 15 topic categories.
The central component of the curation pipeline is Olign, a proprietary speech-to-text aligner accessible via SDK or on-premises deployment. Olign produces sentence- and word-level timestamps by leveraging explicit alignment rather than the CTC spike or attention-weight heuristics used by E2E systems such as Whisper. This makes it robust to transcription errors (hallucinated words are assigned zero duration rather than consuming real speech time), overconfident ASR output, and speaker overlap (per-speaker word timings are maintained even in simultaneous speech segments). Confidence scores at both the word and sentence level are designed to be calibrated rather than uniformly high, enabling downstream filtering of low-quality segments without wholesale exclusion.
The open-source subset, OleSpeech-IV-2025-EN-AR-100, contains 100 hours of English speech with all regional accent variants. It is split 8:1:1 (train:dev:test) and released for non-commercial research use. The full Tier IV collection and the Olign pipeline remain proprietary.
Key Results
No quantitative evaluation of downstream TTS or spoken conversational agent performance on OleSpeech-IV is reported. The paper provides qualitative comparisons between Olign and a Whisper-based E2E aligner, illustrating boundary smearing in the E2E case and correct word boundaries in Olign output, including in overlapping-speech segments. Confidence score calibration is demonstrated via examples where over-confident Whisper outputs contrast with Olign’s mid-range scores on genuinely ambiguous or erroneous words. The paper notes that benchmarking of Olign’s alignment accuracy is ongoing and no numerical accuracy figures are currently available.
Novelty Assessment
The primary contribution is the dataset itself, positioned within a tiered family of proprietary Olewave speech resources. The Olign aligner is described as an engineering integration combining established forced-alignment principles with pragmatic handling of ASR errors and speaker overlap; its design details are not disclosed in sufficient depth for independent replication. The open research contribution is modest: 100 hours of annotated English conversational speech, which is meaningful but smaller than several contemporaneous publicly available corpora. The lack of any downstream evaluation limits the paper’s ability to demonstrate that the dataset and pipeline confer practical benefits over existing alternatives.
Field Significance
Moderate — OleSpeech-IV provides a documented methodology for constructing tiered, multispeaker conversational corpora from publicly available sources, and the open subset offers a modestly sized resource for multilingual TTS and spoken conversational agent research. Its broader impact is constrained by the proprietary nature of both the full dataset and the Olign aligner, and by the absence of any trained-model evaluation demonstrating downstream gains.
Claims
-
supports: Constructing large-scale conversational speech corpora with fine-grained speaker diarization and word-level timestamps is feasible from publicly available broadcast sources when specialized alignment pipelines are used in place of general-purpose ASR.
Evidence: OleSpeech-IV Tier IV assembles 5,000+ hours from English podcasts, talk shows, and teleconferences with human-sourced speaker labels and Olign-produced word-level timestamps and confidence scores. (§2.4, §3)
-
complicates: End-to-end attention- and CTC-based speech aligners introduce systematic boundary errors and overconfident posterior scores that degrade the reliability of automatically annotated training data.
Evidence: Whisper-based E2E aligners produce boundary smearing (segment endings placed at the onset of the next word) and assign high posterior probabilities (>0.8) to misrecognized words; Olign assigns zero duration to hallucinated insertions and provides calibrated per-word confidence scores. (§3.1.1.2, §3.1.2)
-
complicates: Speaker overlap in naturalistic multi-party speech remains an open challenge for current alignment pipelines, with quantitative accuracy benchmarks largely unreported.
Evidence: Olign supports per-speaker word-level timestamps in overlapping segments, but the paper explicitly notes that benchmarking for this capability is ongoing and no numerical accuracy figures are available. (§3.1.1.3)
-
supports: Calibrated word-level confidence scores in speech annotation pipelines can reduce human review effort by directing annotators to mid-confidence ambiguous segments rather than requiring complete audio re-examination.
Evidence: The Olign pipeline provides per-word confidence scores that highlight words likely to contain errors, enabling annotators to focus effort without reviewing every word in a recording. (§3.1.2)
Limitations and Open Questions
Warning
Olign, the central pipeline component, is proprietary and not open-sourced, making the data curation methodology unreproducible. No quantitative accuracy benchmarks for Olign are provided; the paper explicitly states these are work in progress.
The open-source release covers only 100 of the 5,000+ hours in the Tier IV collection, limiting reproducibility of experiments at scale. The paper does not evaluate any downstream speech generation or understanding model trained on OleSpeech-IV, so the practical utility of the dataset for TTS or spoken conversational agent training remains undemonstrated. The dataset currently supports full Olign alignment only for English; Spanish, Japanese, and Mandarin support is listed as under development.
Wiki Connections
- Multilingual TTS — OleSpeech-IV provides a multilingual conversational corpus spanning 12 languages that can serve as training data for multilingual TTS systems.
- Spoken Language Model — the dataset’s conversational structure, speaker diarization, and turn-taking annotations make it directly relevant to training spoken language models for interactive speech generation.
- Speaker Adaptation — per-speaker labels and word-level timestamps enable speaker-aware training objectives, supporting zero-shot speaker conditioning and adaptation research.