arXiv · 2026 · Preprint
Wen-Chin Huang et al. · → Paper · Demo: ✗ · Code: ✗
Introduces CodecMOS-Accent, a 4,000-sample mean opinion score (MOS) benchmark covering 9 neural codec resynthesis systems and 15 LLM-based TTS voice-cloning systems evaluated across 10 English accents drawn from VCTK, with 19,600 human annotations spanning naturalness, speaker similarity, and accent similarity.
Problem
Neural audio codecs (NACs) and the LLM-based TTS systems trained on top of them are usually benchmarked with reconstruction-quality objective metrics on standard, single-accent speech. Existing codec benchmarks (DASB, Codec-SUPERB) and individual NAC papers largely omit subjective listening tests in the TTS setting, and prior accent-similarity studies have been limited in scale (few accents, systems, or listeners). This leaves an open question: does the in-context “voice cloning” capability of modern codec-based TTS systems, which is known to replicate speaker identity and even recording environment, generalize to accent as well, and can existing objective metrics stand in for costly subjective accent evaluation?
Method
The authors build CodecMOS-Accent on top of VCTK, selecting 32 speakers spanning 10 accents (20 female, 12 male), downsampled to 16kHz with silence trimmed via forced-alignment labels. Five ground-truth utterances (3-7 seconds) per speaker yield 160 reference samples; for the voice-cloning condition, a separate same-speaker utterance is used as the prompt.
Two tasks are evaluated. In resynthesis, 9 open-source neural codecs (EnCodec, DAC, SpeechTokenizer, FACodec, Mimi, SNAC, WavTokenizer, NanoCodec, NeuCodec) encode and decode the reference speech, with some models run at low-bitrate configurations to widen the quality range. In voice cloning, 15 open-source LLM-based TTS systems (VALL-E-X, TorToiSe, XTTS, FireRedTTS, MaskGCT, OpenAudio S1-mini, VevoTTS, CosyVoice 2, Llasa-1B, MetaVoice, OrpheusTTS, VoiceStar, IndexTTS2, Chatterbox, NeuTTS Air) generate speech for a target text conditioned on the reference prompt, so that speaker identity and accent must be replicated from a short audio cue.
A crowdsourcing company (Intergroup) recruited 25 listeners across four self-reported accent regions (19 US, 2 Canadian, 3 English, 1 Scottish) to rate 4,000 samples (784 per listener) on a five-point scale along three dimensions: S-NAT (naturalness), S-SPK-SIM (speaker similarity to the reference), and S-ACC-SIM (accent similarity to the reference, independent of speaker identity). This produced 19,600 annotations (4.9 per sample on average) after rejecting 55 low-quality samples flagged in listener comments. Four objective metrics are computed in parallel for comparison: O-WER (Whisper-based), O-SPK-SIM and O-ACC-SIM (cosine similarity from ECAPA-TDNN speaker- and CommonAccent-trained accent-embedding models respectively), and O-UTMOS (predicted quality from the UTMOS model, trained on the 2022 BVCC dataset).
Key Results
At the system level (Table 1), ground truth ranked only 9th in naturalness (S-NAT), reflecting known VCTK recording artifacts, while several TTS systems (CosyVoice 2, OpenAudio S1-mini, Llasa-1B) exceeded the ground truth’s S-NAT score. Ground truth retained the top rank in both S-SPK-SIM and S-ACC-SIM. Subjective speaker and accent similarity scores had a much narrower floor (lowest 2.84 and 3.13 respectively) than naturalness (lowest 1.36), indicating that even heavily degraded codecs (e.g. low-bitrate SpeechTokenizer, EnCodec) retained perceptible speaker and accent identity.
Correlation analysis (Table 2) found S-SPK-SIM and S-ACC-SIM strongly correlated at both the utterance level (r=0.75) and system level (r=0.97). Among objective proxies, O-SPK-SIM correlated most strongly with both S-SPK-SIM (r=0.86) and, unexpectedly, with S-ACC-SIM (r=0.90), outperforming the purpose-built O-ACC-SIM (r=0.81) as an accent-similarity predictor. O-UTMOS correlated very strongly with S-NAT at the system level (r=0.96) despite being trained on pre-2020, largely single-speaker BVCC data. O-WER correlated weakly with all subjective dimensions.
A same-accent bias analysis (Table 3) found listeners sharing the accent of the sample rated ground-truth S-SPK-SIM and S-ACC-SIM significantly higher than listeners with a different accent (p<0.05), while ground-truth S-NAT showed no such bias (p=0.160). Across the full system set, the same-accent bias also became significant for S-NAT (p<0.001), which the authors attribute to a US-listener-majority pool interacting with predominantly US-English-trained models. A cross-accent agreement analysis (Table 4) found generally moderate-to-strong Spearman correlation across listener accent groups (mostly ρ>0.4), with the lowest agreement for Welsh (min ρ=0.51) and Irish (min ρ=0.43) accents.
Novelty Assessment
The contribution is a dataset and evaluation study, not a new model or architecture. Its novelty lies in scale and analytical scope relative to prior accent-similarity work: 24 systems, 32 speakers, 10 accents, and 19,600 annotations, versus prior studies that the authors characterize as limited in accent, system, or listener count. The specific analytical findings, the strong correlation between speaker and accent similarity, the surprising strength of a speaker-similarity embedding as an accent-similarity proxy, evidence that early codec layers retain speaker/accent information previously assumed to be encoded only in later layers, and the same-accent perceptual bias, are genuine empirical observations rather than incremental confirmations of known results.
Field Significance
moderate — This paper provides a comprehensive, up-to-date empirical snapshot of how current neural codecs and codec-based TTS systems generalize to accented speech, and it surfaces concrete methodological findings (a speaker-similarity embedding as a strong accent-similarity proxy, a listener same-accent perceptual bias) that are directly usable in designing future accent-focused evaluation protocols. Its scope is that of a benchmark and analysis study rather than a new architecture or training method, so its significance depends on the released dataset seeing use as a shared evaluation resource.
Claims
- supports: Perceived speaker identity and perceived accent are tightly coupled in synthesized and resynthesized speech, such that systems that preserve one tend to preserve the other.
Evidence: Pearson correlation between S-SPK-SIM and S-ACC-SIM was 0.75 at the utterance level and 0.97 at the system level across 24 codec and TTS systems. (§3.2, Table 2)
- refines: An objective speaker-similarity embedding can be a stronger predictor of perceived accent fidelity than a dedicated accent-similarity embedding.
Evidence: System-level correlation of O-SPK-SIM (ECAPA-TDNN speaker embeddings) with S-ACC-SIM was 0.90, exceeding the correlation of the purpose-built O-ACC-SIM (CommonAccent-trained embeddings) with S-ACC-SIM at 0.81. (§3.2, Table 2)
- contradicts: Early-layer codebooks in residual-vector-quantization codecs encode only low-level linguistic/phonetic content, without high-level speaker or accent identity.
Evidence: Resynthesis using only the first two SpeechTokenizer layers preserved perceptible speaker and accent identity (S-SPK-SIM 2.838, S-ACC-SIM 3.131) despite near-floor naturalness (S-NAT 1.386), directly contradicting the assumption that low-layer codes are purely linguistic. (§3.1, Table 1)
- complicates: Predicted-quality models trained on older, largely single-speaker listening-test datasets remain valid proxies for subjective naturalness of modern multi-speaker, codec-based TTS and resynthesis systems.
Evidence: O-UTMOS (trained on the pre-2020 BVCC dataset) correlated with S-NAT at r=0.96 system-level across 24 contemporary systems, which the authors attribute to modern codec-based systems facing similar naturalness bottlenecks as older single-speaker TTS rather than to genuine generalization of the predictor. (§3.2)
- complicates: Subjective ratings of speaker and accent similarity, and in some conditions naturalness, are biased upward when the listener shares the accent of the evaluated speaker.
Evidence: Same-accent listeners rated ground-truth S-SPK-SIM and S-ACC-SIM significantly higher than different-accent listeners (p=0.011 and p<0.001); across the full system set this bias extended to S-NAT as well (p<0.001), plausibly reflecting the predominantly US-English listener pool and US-trained systems. (§3.3, Table 3)
Limitations and Open Questions
The benchmark deliberately excludes commercial, black-box TTS and codec systems, so its rankings characterize only currently available open-source systems and may not reflect the state of the art overall. The accent coverage is limited to 10 English-language accent groups present in VCTK, several with only a handful of speakers, and listener representation is skewed toward US English (19 of 25 listeners), which the authors themselves note may drive part of the observed naturalness bias. Cross-accent agreement (Table 4) was weakest for the least-represented accents (Welsh, Irish), suggesting the per-accent reliability of the dataset varies. The dataset relies on a single crowdsourcing platform for annotation collection, and 55 of 4,000 samples were excluded post hoc for quality issues identified from listener comments rather than a pre-registered quality filter.
Wiki Connections
- Neural Audio Codec — benchmarks 9 open-source neural codecs on resynthesis quality and their capacity to preserve speaker and accent identity across bitrate configurations.
- Zero-Shot TTS — evaluates 15 zero-shot voice-cloning TTS systems on their ability to replicate both speaker identity and accent from a short reference prompt.
- Subjective Evaluation — conducts a large-scale MOS listening test (19,600 annotations, 25 listeners) across naturalness, speaker similarity, and accent similarity, and analyzes listener accent bias.
- Evaluation Metrics — systematically compares four objective metrics against three subjective dimensions to assess which objective proxies best predict human judgments.
- NeuCodec (Finite scalar quantization codec) — evaluated as one of the 9 resynthesis codecs in the benchmark’s system-level comparison.
- CosyVoice 2 — evaluated as one of the 15 voice-cloning TTS systems; achieved the top S-NAT score in the benchmark.
- IndexTTS2 — evaluated as one of the 15 voice-cloning TTS systems in the benchmark.