arXiv · 2023 · Preprint
Gemini Team et al. (Google) · → Paper · Demo: ? · Code: ?
Introduces the Gemini model family, a set of natively multimodal transformer decoders trained jointly on text, image, audio, and video that achieve state-of-the-art performance across 30 of 32 evaluated benchmarks, including the first model to exceed human-expert performance on MMLU.
Citation Stub
This paper is not a speech generation paper but is cited by the corpus. See Context in Speech Generation below for why it is relevant.
Context in Speech Generation
Gemini models are designed to process audio natively, ingesting raw 16kHz signals through Universal Speech Model (USM) features as part of an interleaved multimodal input stream. This native audio understanding allows Gemini to handle automatic speech recognition and speech translation in a single model, outperforming both Whisper and USM on ASR benchmarks including multilingual and multilingual LibriSpeech tasks. The family spans Ultra, Pro, and Nano (1.8B and 3.25B parameter) sizes, making it a candidate backbone for spoken conversational agent systems requiring joint reasoning over text and speech. Its post-training pipeline (SFT plus RLHF) and instruction-following capabilities provide an alignment recipe that multimodal speech LM work can draw on.
Wiki Connections
- spoken-language-model — Gemini’s native multimodal architecture (audio + text in a single decoder) and post-training alignment recipe inform spoken conversational agent systems that require joint speech and language reasoning