arXiv · 2018 · Preprint
Devlin et al. (Google) · → Paper · Demo: ? · Code: ✓
BERT introduces deep bidirectional pre-training of Transformer encoders using masked language modelling and next sentence prediction, establishing a unified fine-tuning paradigm that achieved state-of-the-art results across eleven NLP benchmarks.
Citation Stub
This paper is not a speech generation paper but is cited by the corpus. See Context in Speech Generation below for why it is relevant.
Context in Speech Generation
BERT establishes the masked language modelling (MLM) pre-training objective, enabling a Transformer encoder to attend to both left and right context simultaneously rather than operating unidirectionally. The resulting contextual text representations are rich in syntactic and semantic information, making BERT a natural candidate as a text encoder in TTS systems that require linguistically informed embeddings beyond simple lookup tables. TTS architectures can leverage a pre-trained BERT encoder to extract contextually grounded phonetic and prosodic features from input text, replacing or supplementing conventional grapheme or phoneme embeddings. The fine-tuning paradigm BERT introduces, where a pre-trained model is initialised with shared parameters and then adapted to a downstream task with minimal added components, provides a blueprint that speech generation pipelines apply to conditioning components, text-front-end modules, and prosody predictors.
Wiki Connections
No in-corpus papers cited by this work. Cross-links from corpus papers that cite BERT are added by the integration agent.