arXiv · 2026 · Preprint
Meza, Ferrer & Riera (Universidad de Buenos Aires / ICC, CONICET-UBA) · → Paper · Demo: ? · Code: ✓
Trains autoencoders on F0, loudness and voicing contours alone to produce utterance-level prosody embeddings, and introduces a classification benchmark with speaker-, text- and spurious-correlation-controlled protocols showing that these embeddings hold up where waveform-based embeddings collapse.
Problem
General-purpose speech representations (WavLM, HuBERT, wav2vec 2.0, emotion2vec) encode prosody mixed with speaker identity, lexical content and channel. For tasks where prosody is the distinguishing factor, such as question/statement intonation or contour classification, that entanglement lets downstream classifiers latch onto text or speaker cues that do not transfer. Prior prosody-only representations either rely on hand-crafted statistics (eGeMAPS) or were learned for reconstruction alone (the F0/energy VQ-VAE of Prosody Embedding Codebook) or end to end inside a TTS model (prosody encoders for Tacotron-style transfer), and none of these were checked on downstream tasks designed so that only prosody separates the classes. The paper also argues that existing prosody benchmarks (SUPERB-prosody) mix in lexical and speaker cues, so they cannot tell whether a representation is purely prosodic.
Method
The input is frame-level prosody at a 10 ms hop: Praat autocorrelation F0 (two-pass speaker-adapted range, log scale, linearly interpolated over unvoiced frames), a binary voicing track, and an eGeMAPS-style perceptual loudness signal (26-band mel filterbank with equal-loudness weighting and cubic-root compression). Pitch and energy are z-normalized with global training-set statistics. Restricting the input to these signals is the paper’s mechanism for keeping speaker timbre and phonetic content out of the embedding; there is no adversarial or factorization loss.
Three autoencoder families are compared, and only the encoder is kept at inference:
- GRU: a bidirectional multi-layer GRU encoder whose final forward and backward states are concatenated and projected to a single vector z of dimension d; an autoregressive GRU decoder reconstructs pitch and energy conditioned on z at every step, with a non-autoregressive GRU head for voicing. Training uses a three-stage duration curriculum and teacher forcing linearly annealed to zero over 80 epochs.
- TransfSeq: a 3-layer, 8-head Transformer encoder whose full output sequence is attended by a non-autoregressive Transformer decoder (no bottleneck); the downstream embedding is the concatenated mean and standard deviation of encoder outputs, so it is not directly shaped by training.
- TransfCLS: the same Transformer with a learnable CLS token whose output alone is repeated and fed to the decoder, forcing a global bottleneck.
Three reconstruction targets are compared: EPv (energy plus pitch on voiced frames), EPi (energy plus interpolated pitch on all frames), and EPvV (EPv plus a binary cross-entropy voicing term, which the authors note may indirectly encode speech rate). Each is trained either as a standard autoencoder (AE) or as a masked autoencoder (MAE), where contiguous spans are replaced by a learned mask token (GRU) or dropped from the sequence (Transformer). All training data from LJSpeech and VCTK is segmented into intonation units with a pretrained automatic IU-boundary model, so every training example is a complete prosodic contour. Model parameter counts are not reported.
The benchmark uses three datasets: SynthID, a new set of 608 utterances from 19 ElevenLabs synthetic voices reading 16 sentences that are natural as either a question or a statement (the only difference in the text prompt is the final punctuation); RAVDESS emotion classification (two fixed sentences, 24 speakers); and a rebalanced subset of the Bestiary intonational-contour corpus (358 utterances, 3 classes) in which text is made uninformative of class. Each is evaluated under three cross-validation protocols of increasing difficulty: speaker-independent (SI), speaker-and-text-independent (STI), and text-label correlation (TCC), where the text/class combinations of each test sample are withheld from training so that a classifier relying on lexical cues is pushed toward the wrong class. A small MLP, with hyperparameters tuned once by Bayesian search on SynthID SI, is trained on each embedding; results average three seeds with speaker-conditioned bootstrap 95% confidence intervals. Baselines are eGeMAPS pitch/energy subsets (with and without jitter and shimmer), WavLM (layer-averaged or last layer, mean-pooled), emotion2vec, and the ProsodyVQ-VAE of Prosody Embedding Codebook pooled to mean and standard deviation.
Key Results
All results are reported in figures (bar and scatter plots with confidence intervals), with no tables, so the comparisons below are qualitative except where the text gives numbers.
- Reconstruction error does not predict embedding quality (Figure 2, SynthID). Across GRU, TransfSeq and TransfCLS, AE and MAE, and the three losses, pitch-plus-energy MSE spans about two orders of magnitude while SI and STI accuracy stays nearly flat; under TCC, lower MSE does not go with higher accuracy.
- Bottleneck trade-off. On SynthID, single-vector bottleneck models (GRU especially) tend to be stronger under SI and STI, while the sequence-decoder TransfSeq models tend to be stronger under TCC; the authors describe this pattern as almost reversed but noisy. The three losses show no clear winner, and EPvV is used for the remaining experiments. A 32-dimensional embedding is comparable in accuracy to 128 and 512 dimensions, and for MAE models under TCC it is close to the best result.
- RAVDESS emotion (Figure 3). WavLM and emotion2vec give the highest SI and STI accuracy, but under TCC they drop to near or below the majority-class line, indicating they rely on lexical content. The authors state their best configurations outperform ProsodyVQ-VAE under SI and STI; in Figure 3 that margin is small. Under TCC on RAVDESS the authors name eGeMAPS and ProsodyVQ-VAE as the best systems; in Figure 3 the prosody-only systems (both eGeMAPS subsets, ProsodyVQ-VAE and TransfSeq-AE) sit close together with overlapping confidence intervals, and the authors attribute eGeMAPS’s robustness to its statistical functionals being largely agnostic to the text.
- Bestiary intonation contours (Figure 3). The gap between waveform baselines and the prosody-only embeddings under SI and STI is smaller than on RAVDESS, and under TCC the waveform embeddings fall below chance. The proposed models outperform eGeMAPS and ProsodyVQ-VAE across all three protocols, and under TCC they give the best results of all systems.
- Leakage probe (§5.2, SynthID). Linear classifiers predict speaker and sentence at 72% and 100% from WavLM-mean embeddings, against 40% and 41% from TransfSeq-AE embeddings, so the prosody-only embeddings carry much less speaker and lexical information, though not none.
Comparisons are fair in the sense that every representation goes through the same MLP and protocols, but MLP hyperparameters were selected on SynthID with each configuration, and no single proposed configuration is best on every dataset and protocol.
Novelty Assessment
The encoders themselves are standard: GRU and Transformer autoencoders, CLS pooling and MAE-style masking applied to F0/energy contours, a setting already explored with a VQ-VAE by Prosody Embedding Codebook. The more original parts are the evaluation design (the TCC protocol that withholds text/class combinations to expose lexical shortcuts, the rebalanced Bestiary subset, and the SynthID set with all speaker by text by intonation combinations by construction) and the empirical finding that reconstruction MSE is a poor proxy for downstream prosodic usefulness. The paper does not test the embeddings inside a speech synthesis system, so their value as TTS conditioning, which the authors list as an intended use, is untested here.
Field Significance
Low. The paper provides a controlled, prosody-isolating evaluation protocol and evidence that waveform SSL embeddings exploit lexical shortcuts on prosodic tasks, which is useful for anyone choosing a prosody or style representation to condition expressive synthesis. Its evidence is English-only, uses small classification sets with automated scoring and no human evaluation, and never closes the loop to generated speech.
Claims
- complicates: Reconstruction error of a prosody autoencoder is not a reliable proxy for how much task-relevant prosodic information its learned embedding carries.
Evidence: On SynthID, pitch-plus-energy MSE varies by about two orders of magnitude across architectures, losses and pretext tasks while SI/STI accuracy is nearly constant, and under TCC lower MSE does not correspond to higher accuracy; 32-dimensional MAE embeddings have markedly worse MSE but near-best TCC accuracy. (§5.1, Figure 2)
- complicates: Waveform-level self-supervised speech representations encode enough lexical content that classifiers trained on them for prosodic distinctions can learn text shortcuts which fail when text/class pairings shift.
Evidence: WavLM and emotion2vec lead on RAVDESS and Bestiary under speaker-independent splits but fall to near or below the majority-class baseline under the TCC protocol; a linear probe recovers sentence identity at 100% from WavLM-mean embeddings versus 41% from TransfSeq-AE embeddings. (§5.2, Figure 3)
- supports: Restricting a learned representation’s input to F0, energy and voicing contours substantially reduces speaker and lexical information in the resulting global prosody embedding while preserving information about intonation categories.
Evidence: Speaker and sentence probe accuracies on SynthID are 40% and 41% for TransfSeq-AE versus 72% and 100% for WavLM-mean; on Bestiary the prosody-only embeddings come close to the waveform baselines under SI/STI and give the best TCC results. (§5.2, Figure 3)
- refines: Prosody-only embeddings are best suited to tasks whose classes differ mainly in intonation; for emotion, waveform representations retain an advantage because voice quality and spectral cues also carry the signal.
Evidence: The gap between WavLM/emotion2vec and the proposed embeddings under SI and STI is notably larger on RAVDESS emotion than on Bestiary contour classification, and on RAVDESS TCC hand-crafted eGeMAPS statistics are at least level with the learned embeddings, whereas on Bestiary TCC the learned embeddings clearly lead. (§5.2, Figure 3)
- complicates: For learned prosody embeddings, a tight global bottleneck that preserves fine temporal detail improves accuracy when text and class are independent but reduces robustness when they are spuriously correlated.
Evidence: Single-vector GRU and TransfCLS models tend to lead under SI and STI on SynthID, while the mean/std-pooled TransfSeq models tend to lead under TCC, a pattern the authors describe as almost reversed though noisy. (§5.1, Figure 2)
Limitations and Open Questions
Warning
No speech is generated: the embeddings are evaluated only as inputs to emotion and intonation classifiers, so their usefulness as conditioning for TTS or voice conversion, which the paper names as a target application, is not demonstrated.
- Training data is limited to clean read English (LJSpeech, VCTK) because F0 and IU extraction need clean audio; noisier corpora such as LibriSpeech were excluded, and robustness to noise is left to future work.
- Downstream sets are small (608, 1,440 and 358 utterances) and English-only. One of them (SynthID) is synthetic speech from a single commercial TTS system, and all development decisions, including MLP hyperparameters, were made on it.
- No single configuration is best everywhere: the strongest proposed models on RAVDESS SI/STI are not the strongest on TCC, and the learned embeddings give no clear gain over eGeMAPS on RAVDESS TCC.
- The embedding dimension of the Figure 3 systems is given as d = 128 in §5.2 but as 256 in the Figure 3 caption.
- Duration and speech-rate information enters only indirectly through the voicing track; no explicit timing features are modelled.
- Model sizes, training compute and the code URL are not reported in the parsed text, although the authors state that the benchmark, models and code are released.
Wiki Connections
- Prosody Embedding Codebook — closest baseline (ProsodyVQ-VAE on F0 and energy); this paper adds the downstream evaluation that work lacked and outperforms it on Bestiary across all protocols, with a small margin on RAVDESS SI/STI and no clear separation on RAVDESS TCC.
- emotion2vec — waveform-based baseline that leads on RAVDESS speaker-independent emotion classification but falls near the majority-class line under the text-label-correlation protocol.
- NaturalSpeech 2 — cited as a synthesis system that models prosody through latent diffusion, an example of task-specific prosody modelling that the paper contrasts with general-purpose embeddings.
- VALL-E 2 — cited as a neural codec language model whose prosody is modelled implicitly inside the synthesis pipeline rather than through a reusable prosodic embedding.