arXiv · 2022 · Preprint
Radford et al. (OpenAI) · → Paper · Demo: ? · Code: ✓
Whisper demonstrates that scaling weakly supervised pretraining on 680,000 hours of web-sourced audio trains a multitask speech recognition model that matches human-level robustness across out-of-distribution conditions, without any dataset-specific fine-tuning.
Problem
Prior speech recognition research had separated two objectives that should be unified: self-supervised pretraining produced high-quality encoders, but required fine-tuning on each target dataset, which made models brittle outside their training distribution. Supervised models trained on academic datasets like LibriSpeech achieved near-superhuman WER on that benchmark but performed far worse than humans on other recording conditions, speakers, and domains. The field had conflated in-distribution performance with genuine robustness.
Method
Whisper is an encoder-decoder Transformer trained to predict transcripts directly from raw audio in a supervised fashion, using web-scraped (audio, transcript) pairs. All audio is resampled to 16 kHz and encoded as 80-channel log-magnitude Mel spectrograms with a 25 ms window and 10 ms stride. The encoder uses two convolution layers as a stem, followed by standard Transformer blocks with pre-activation residuals. The decoder uses learned position embeddings and tied input-output token representations.
The training data pipeline filters machine-generated transcripts using heuristics (e.g., all-uppercase or all-lowercase text detection, ASR-system stylistic artifacts) and matches audio language to transcript language using VoxLingua107 classification. The final dataset covers 680,000 hours: 438,000 hours of English speech recognition, 117,000 hours of other languages, and 125,000 hours of X-to-English translation.
Rather than training a separate model per task, Whisper encodes all tasks and conditioning information as a sequence of special tokens prepended to the decoder input: a language token (one of 99), a task token (transcribe or translate), and an optional timestamp flag. This allows a single model to perform multilingual ASR, speech translation, voice activity detection, and language identification. The model family ranges from 39M (Tiny) to 1550M (Large) parameters (Table 1).
Training uses AdamW with gradient norm clipping and linear learning rate decay, with 2^20 update steps and batch size 256. No data augmentation is applied; generalisation relies entirely on dataset diversity.
Key Results
On LibriSpeech test-clean, the best zero-shot Whisper model achieves 2.7% WER, roughly matching the mid-2019 supervised state of the art. The headline contribution is not this LibriSpeech number but its out-of-distribution robustness: compared to a wav2vec 2.0 Large model with identical LibriSpeech performance, Whisper achieves 55.2% lower average WER across 14 held-out datasets including CHiME-6 (65.8 vs. 25.5%), Common Voice (29.9 vs. 9.0%), and CORAAL (35.6 vs. 16.2%) (Table 2, §3.3).
On the CoVoST 2 X-to-English translation benchmark, zero-shot Whisper achieves 29.1 BLEU, outperforming Maestro, mSLAM, and XLS-R in the overall and low-resource settings (§3.5, Table 4). On the Multilingual LibriSpeech benchmark, zero-shot Whisper achieves 7.3% WER, outperforming XLS-R (10.9%), mSLAM (9.7%), and Maestro (Table 3), though it trails significantly on VoxPopuli where other models train on in-distribution data.
In human comparison tests on Kincaid46, Whisper’s English ASR performance is within a fraction of a percentage point of pure-human transcription services and better than all four commercial ASR systems tested (§3.9, Figure 7).
Dataset scaling ablations show consistent improvement across tasks, with English WER roughly following a power law from 3,405 to 54,000 hours before showing diminishing returns at 681,000 hours (§4.2, Table 6). Multitask and multilingual joint training hurts small models but benefits larger ones, with large models slightly outperforming English-only counterparts (§4.3, Figure 9).
Note
Text normalisation was developed iteratively against Whisper outputs, creating a risk of overfitting the normalizer to Whisper’s transcription style. The authors validate against an independent FairSpeech normalizer and find mostly similar results, with differences concentrated on specific contractions and numerical formats (§4.4).
Novelty Assessment
The architectural choices in Whisper are deliberately standard (off-the-shelf encoder-decoder Transformer) to isolate the effect of data scale. The genuine contributions are: (1) the multitask token format that collapses ASR, translation, VAD, and language identification into a single decoder, and (2) the empirical demonstration that weak supervision at sufficient scale achieves human-level robustness without self-supervised pretraining. The filtering pipeline for machine-generated transcripts addresses a practical data-quality problem not previously studied at this scale. The scaling analysis (model size, dataset size, multitask/multilingual transfer) constitutes a thorough empirical characterisation of weakly supervised ASR. This is an empirical and engineering contribution rather than an architectural one.
Field Significance
Tip
High — Whisper demonstrates robust zero-shot ASR generalisation at human-level accuracy across diverse conditions and speakers without dataset-specific fine-tuning. Its public release across five model sizes (39M–1550M parameters) makes it practically accessible for automatic intelligibility evaluation in TTS research, where WER on held-out speech is the standard downstream metric. Its significance for this wiki derives primarily from its role as an evaluation instrument, not as a speech synthesis system.
Claims
- Weakly supervised ASR training at sufficient scale produces models that generalise robustly across recording conditions, speakers, and languages without dataset-specific fine-tuning. (§3.3, Table 2)
- Models evaluated solely on in-distribution benchmarks can appear superhuman while remaining significantly below human performance out-of-distribution, indicating that benchmark-focused evaluation overstates robustness. (§3.3, Figure 2)
- Multitask and multilingual joint training introduces negative transfer in small models but confers benefits at large scale. (§4.3, Figure 9)
- Per-language zero-shot ASR performance scales predictably with per-language training data, with WER roughly halving for every 16-fold increase in supervision hours. (§3.4, Figure 3)
Limitations and Open Questions
Long-form transcription remains unreliable without explicit decoding heuristics (beam search, temperature fallback, voice activity detection, initial timestamp constraints) to prevent repetition loops, truncation of edge segments, and hallucination. These are workarounds rather than principled solutions (§4.5). Performance on languages with non-Indo-European scripts (Hebrew, Telugu, Chinese, Korean) is worse than the training-data trend predicts, likely reflecting tokenisation mismatch and linguistic distance. Language identification accuracy on Fleurs is penalised by having no training data for 20 of 102 Fleurs languages, upper-bounding accuracy at 80.4% regardless of model capacity (§3.6).
The text normalisation procedure carries a risk of being overfitted to Whisper’s transcription style, which could bias WER comparisons in Whisper’s favour (§4.4). Fine-tuning properties are not studied in this paper, leaving open how the model’s robustness advantage transfers to supervised fine-tuning settings where prior work performs.
Wiki Connections
Related evaluation infrastructure: evaluation-metrics, subjective-evaluation. Related multilingual corpus: 1912.06670 (Common Voice). Related concepts: self-supervised-speech, multilingual-tts.