arXiv · 2026 · Preprint
Zhao et al. (SB Intuitions) · → Paper · Demo: ? · Code: ?
Applies Direct Preference Optimization to align a Japanese SpeechLLM’s text outputs toward concise, conversational, TTS-ready style, and introduces SpokenElyza, the first benchmark for Japanese speech-worthiness.
Problem
SpeechLLMs that combine ASR-trained encoders with text-based LLM backbones inherit written-style output habits: markdown formatting, bullet points, and complex nested sentences that read well but synthesize poorly and sound unnatural when spoken. Prior work addressing this “speech-worthiness” mismatch has focused exclusively on text-based LLMs, leaving the growing ecosystem of audio-input SpeechLLMs unaddressed. The problem is sharper in Japanese, where written and spoken registers diverge substantially in politeness markers, sentence-final particles, and syntactic complexity, so a SpeechLLM can produce grammatically correct written Japanese that nonetheless sounds robotic or socially inappropriate when synthesized as speech. No evaluation resource previously existed to measure this specific failure mode for Japanese spoken dialog.
Method
The base system follows a Qwen2-Audio-style architecture: a Whisper-large speech encoder is connected to an LLM (Sarashina-7B, chosen for its Japanese language capability) via a shallow projector layer. In a pretraining stage, the LLM and audio encoder are frozen and only the projector is trained to align speech and text representations, using ReazonSpeech and in-house paired audio-text data.
To adapt the pretrained model toward speech-worthy outputs, the authors frame the problem as preference optimization: speech-worthy responses are preferred over written-style responses for the same input. They apply Direct Preference Optimization (DPO) combined with a supervised fine-tuning (SFT) term that directly maximizes likelihood of the chosen (speech-worthy) response, with a combined objective L = w·L_DPO + (1-w)·L_SFT, where w is swept over {0.5, 0.9, 0.95, 0.99}. Preference pairs are drawn from three datasets translated or adapted into Japanese: SpeechPref (an audio-first preference corpus derived from Databricks-Dolly-15K, filtered to exclude non-speech-suitable content), InstructS2S-200K (originally built for training unified speech-to-speech systems, here re-scored with gpt-oss-120B to construct high-margin preference pairs from multiple rollouts), and DeepDialog (casual chit-chat spoken dialogue, similarly re-scored). During preference training, they compare tuning only the top four transformer layers (“TopLayers”) against tuning key/query projections and LayerNorm parameters across all layers (“KQ-LN”), on the hypothesis that written-to-spoken style transfer mainly requires adjusting attention patterns rather than learning new factual knowledge. A textual “spoken system prompt” instructing the model to produce speech-friendly conversational responses is also tested as an inference-time addition.
To evaluate the approach, the authors construct SpokenElyza, a benchmark derived from the widely used ELYZA-tasks-100 Japanese instruction-following dataset. Construction proceeds through four stages: (1) modality filtering removes tasks that cannot be verbalized (e.g., text string manipulation), yielding a 36-example subset called Elyza; (2) an external LLM (gpt-oss-120B) rewrites the original written-style ground-truth responses under speech-friendly constraints (no markdown/lists, polite spoken predicates, simplified syntax), producing 34 SpokenElyza examples after excluding two responses that remained too long; (3) a native Japanese researcher listens to TTS-synthesized versions of the rewritten responses and corrects any that fail auditory intelligibility; (4) the original per-example absolute scoring criteria are standardized into relative deductions from a maximum of 5. Evaluation combines an LLM-as-judge score (Qwen2.5-32B-Instruct, using a custom 1-5 speech-suitability rubric for SpokenElyza and the standard ELYZA rubric for Elyza) with automatic surface-form proxies: word count (via the Janome tokenizer) for conciseness, dependency-parse depth (via SpaCy) for syntactic complexity, and the percentage of non-vocalizable characters (NV%).
Key Results
On the LLM-as-judge evaluation, the pretrained checkpoint scores 2.91 on SpokenElyza and 3.97 on Elyza. Adding DPO+SFT training raises the SpokenElyza score to 2.97, and combining DPO+SFT with the spoken system prompt reaches 3.44, an 18% relative improvement over the pretrained baseline, while Elyza performance drops only modestly to 3.78 (a 5% relative decrease) (Table 2). The spoken system prompt alone is a strong lever: it cuts average word count from 325.91 to 65.53, dependency depth from 6.38 to 5.06, and NV% from 13.46% to 3.69%; combining it with DPO+SFT training achieves the lowest dependency depth (4.97) and NV% (3.24%) of all configurations, though with a slightly higher word count (77.79) than prompting alone, which the authors attribute to preference training encouraging more complete responses (Table 3).
In the ablation over trainable parameters, KQ-LN tuning (key/query and LayerNorm parameters across all layers) outperforms TopLayers tuning on both benchmarks: 3.78 vs. 3.61 on Elyza and 3.44 vs. 3.38 on SpokenElyza-with-prompt (Table 4). Sweeping the DPO loss weight w shows that higher w consistently improves SpokenElyza scores across the individual preference datasets, most pronouncedly for DeepDialog, while on Elyza the trend is less consistent: the combined-dataset configuration dips from w=0.5 to w=0.9 and then plateaus, and per-dataset results for InstructS2S-200K and SpeechPref continue to decline from w=0.9 to w=0.99 (Figure 2).
Novelty Assessment
The architecture itself is not new: a Whisper encoder feeding an LLM via a shallow projector is a well-established SpeechLLM pattern, and DPO is an established preference-optimization method borrowed directly from the text-LLM alignment literature. The paper’s contribution is in application and evaluation infrastructure rather than modeling: it is the first to apply preference-based alignment specifically to the SpeechLLM setting (rather than text-only LLMs) for register control, and it introduces SpokenElyza, the first benchmark purpose-built to measure Japanese speech-worthiness, with a novel four-stage curation pipeline that includes human auditory verification. The KQ-LN vs. TopLayers ablation and the finding that increasing DPO weight trades text-benchmark performance for speech-worthiness are useful empirical findings for practitioners adapting SpeechLLMs to register-specific outputs, but they extend known preference-optimization behavior to a new domain rather than introducing a new training principle.
Field Significance
moderate — this paper demonstrates that preference-based alignment, previously validated only for text-based LLMs, transfers effectively to SpeechLLMs for correcting written-to-spoken register mismatch, and it fills a concrete gap by releasing the first Japanese speech-worthiness benchmark. Its scope is narrow (a single language, a single base model family, and a benchmark derived from just 34-36 curated examples), which limits how far its specific findings generalize, but the curation methodology and evaluation rubric provide a template that could be replicated for other languages and SpeechLLM backbones.
Claims
- supports: Combining preference optimization with supervised fine-tuning on chosen responses can substantially improve a SpeechLLM’s adherence to a target output register while largely preserving its performance on the original text-oriented evaluation.
Evidence: DPO+SFT with a spoken system prompt improves the LLM-as-judge score on SpokenElyza from 2.91 (pretrained) to 3.44 (an 18% relative gain), while the same model’s score on the original written-style Elyza benchmark drops only from 3.97 to 3.78 (a 5% relative decrease). (§4.1, Table 2)
- supports: A textual system prompt instructing a language model to produce speech-friendly output is a highly effective, low-cost complement to preference training for reducing verbosity and non-vocalizable content.
Evidence: Adding a spoken system prompt alone reduces average word count from 325.91 to 65.53 and non-vocalizable content from 13.46% to 3.69%, and the effect combines with DPO+SFT training to reach the lowest dependency depth (4.97) and NV% (3.24%) among all tested configurations. (§4.1, Table 3)
- refines: When lightly fine-tuning a pretrained LLM to shift its output register (e.g., written to spoken) without altering its factual knowledge, tuning attention-related parameters (key/query projections and LayerNorm) across all layers is more effective than restricting adaptation to only the top transformer layers.
Evidence: KQ-LN tuning outperforms TopLayers tuning on both benchmarks: 3.78 vs. 3.61 on Elyza and 3.44 vs. 3.38 on SpokenElyza with the spoken prompt. (§4.2, Table 4)
- complicates: Increasing the relative weight of a preference-optimization loss against a supervised fine-tuning loss in a combined training objective improves the target style-alignment metric but can degrade performance on the original text-oriented benchmark, requiring a trade-off rather than a free improvement.
Evidence: Raising the DPO loss weight w from 0.5 to 0.99 consistently improves SpokenElyza scores across most preference datasets, but on Elyza the combined-dataset score dips from w=0.5 to w=0.9 and then plateaus, with per-dataset scores for InstructS2S-200K and SpeechPref continuing to decline up to w=0.99. (§4.2, Figure 2)
- complicates: Evaluation benchmarks and rubrics built for text-based instruction-following do not reliably capture whether a response is suitable for auditory delivery, since factually correct but written-style responses (markdown, URLs, dense structure) can still score highly under a text-oriented rubric.
Evidence: The standard ELYZA-tasks-100 rubric does not distinguish written from spoken style, motivating the authors to build a separate scoring rubric (Table 1) that explicitly penalizes non-verbalizable formatting, and to construct SpokenElyza with human auditory verification rather than reusing the original written-style ground truth. (§2.3, §3)
Limitations and Open Questions
The evaluation of speech-worthiness improvement relies primarily on LLM-as-judge scoring (Qwen2.5-32B-Instruct) over a small benchmark (34-36 curated examples); the only direct human involvement is a single native-speaker researcher correcting synthesized audio during benchmark construction, not a rated listening study over the model comparisons reported in Table 2-4.
The work is scoped entirely to Japanese, chosen specifically because of its pronounced written-spoken register divergence; the authors explicitly note that generalization to other languages is left to future work. Several training datasets (in-house pretraining pairs, and the in-house TTS system used to synthesize benchmark audio) are proprietary and not otherwise specified, limiting reproducibility. The paper also inherits a known limitation of the broader SpeechLLM paradigm that it cites but does not address: prior work has shown SpeechLLMs underperform cascaded ASR+LLM+TTS systems, so the speech-worthy alignment studied here is one piece of a pipeline whose end-to-end quality is not evaluated in this paper.
Wiki Connections
- Spoken Language Model — the paper’s base system is a SpeechLLM (Whisper encoder plus an adapted LLM) and its contribution targets a known failure mode of this architecture family: written-style rather than speech-appropriate text generation.
- Speech-to-Speech — frames speech-worthy text generation as the critical intermediate step in cascaded SpeechLLM-plus-TTS spoken dialogue systems, which the paper identifies as the dominant paradigm over end-to-end speech-to-speech models.
- RLHF Speech — applies Direct Preference Optimization, a preference-based alignment method from the RLHF family, to align SpeechLLM outputs toward a speech-appropriate register rather than a text-quality objective.
- Evaluation Metrics — introduces a new LLM-as-judge scoring rubric and a set of automatic surface-form proxies (word count, dependency depth, non-vocalizable percentage) specifically for measuring speech-worthiness.
- Qwen2-Audio — the paper’s SpeechLLM architecture (speech encoder plus LLM via a shallow projector) directly follows the Qwen2-Audio design pattern.
- LLaMA-Omni — cited as an end-to-end speech-to-speech model that, along with SLAM-Omni, underperforms cascaded systems, motivating the paper’s focus on improving the cascaded pipeline’s text-generation stage.
- SLAM-Omni — cited alongside LLaMA-Omni as an end-to-end speech interaction system that suffers from catastrophic forgetting and trails cascaded approaches on benchmarks like VoiceBench.
- VoiceBench — cited as evidence that cascaded ASR+LLM+TTS systems outperform end-to-end SpeechLLMs, which underpins the paper’s argument that improving the cascaded pipeline’s text output remains a critical intermediate step.