IWSDS · 2025 · Workshop
Miyazawa et al. (Fairy Devices Inc.) · → Paper · Demo: ? · Code: ?
A HuBERT-based classifier that recognizes four paralinguistic attitudes (agreement, disagreement, questions, stalling) in user speech without linguistic input, achieving macro-F1 of 0.912 and outperforming human listeners (macro-F1 of 0.829 on the same test set).
Problem
Most spoken dialogue systems rely on a cascaded ASR-then-linguistic-understanding pipeline, which ignores the prosodic and paralinguistic cues present in user speech. When a user expresses disagreement through intonation while producing a formally neutral phrase, or signals a question through rising pitch alone, these cues are discarded. This creates systematic miscommunication when user intent is primarily conveyed through paralinguistic features rather than lexical content. Prior recognition work addressing paralinguistic information typically incorporates linguistic features and focuses on specific domains such as meeting speech or news delivery, limiting domain portability and adding ASR latency to the response pipeline.
Method
The model takes raw waveform as input and produces a probability distribution over four mutually exclusive paralinguistic attitude classes: agreement, disagreement, question, and stalling. These four classes correspond to the four main types of boundary pitch movement at the end of prosodic phrases in Japanese, making them recoverable from acoustic features alone.
The backbone is HuBERT-large (frozen during training), which provides 1024-dimensional hidden representations per frame. The optimal extraction layer is selected on validation data, landing at layer 12, consistent with prior findings that paralinguistic information concentrates in intermediate SSL layers. Embeddings are averaged over time and passed through two fully connected layers (1024 units each) with a final softmax output. Linguistic features are deliberately excluded to avoid ASR latency in the response pipeline and to maintain domain independence.
Training uses six-fold cross-validation on a Japanese reading-speech corpus of crowd workers (19,821 quality-filtered utterances) and actors (1,606 utterances). Training data is augmented four-fold with noise drawn from DEMAND, MUSAN, and FSD50K, and with room impulse responses from the BIRD database, using signal-to-noise ratios from -10 to 10 dB.
Key Results
On actors’ speech under clean conditions, the model achieves macro-F1 of 0.909, compared to human listener performance of 0.829 on the same material (Tables 4 and 5). Human confusion between the disagreement and question classes is notably reduced in the model. On the full dataset combining crowd workers and actors, macro-F1 reaches 0.912 (Table 6), confirming that the quality-estimation method used to filter crowd-worker speech is effective.
Under environmental conditions, additive noise degrades performance to macro-F1 of 0.625, and the combined noisy-reverberant condition drops it to 0.449 (Table 7). Applying MP-SENet speech enhancement as preprocessing recovers noisy performance to 0.844, but offers only marginal gains under reverberation (0.492). The gap between noise and reverberation robustness is attributed to the difficulty of recovering prosodic cues from reverberant speech, since prosody is the primary carrier of attitude in this task.
Novelty Assessment
The contribution is engineering integration: HuBERT-large, a well-established self-supervised model, is applied as a frozen feature extractor for a four-class paralinguistic classification task. The network above the encoder is minimal (mean pooling plus two FC layers). The methodological choice to exclude linguistic features is deliberate and empirically validated, and the human perception baseline provides a useful calibration point. The main contribution is demonstrating that a standard SSL model suffices for super-human paralinguistic attitude recognition in clean conditions, and characterizing the degradation profile under noise and reverberation. The noise robustness evaluation is the more informative half of the paper.
Field Significance
Moderate - This paper provides evidence that frozen self-supervised speech representations are sufficient for paralinguistic attitude classification relevant to spoken conversational agents, without linguistic features and with a latency advantage over ASR-dependent approaches. The contribution is application-focused, extending SSL feature extraction to an understanding task in the SCA pipeline. The systematic noise and reverberation evaluation offers useful ground-truth data for deployment considerations, particularly the finding that reverberation is more persistent than noise and largely unrecovered by speech enhancement postprocessing.
Claims
-
supports: Self-supervised speech representations capture sufficient prosodic information to classify paralinguistic attitudes from acoustic input alone, without linguistic features.
Evidence: A frozen HuBERT-large model using layer 12 embeddings averaged over time achieves macro-F1 of 0.909 on a four-class attitude task in Japanese reading speech, exceeding human listener performance of 0.829 on the same data. (§4.1, Table 4, Table 5)
-
complicates: Environmental reverberation poses a more persistent challenge than additive noise for paralinguistic recognition models, and is not adequately addressed by speech enhancement postprocessing.
Evidence: Under noisy conditions, MP-SENet speech enhancement recovers macro-F1 from 0.625 to 0.844; under reverberant conditions, the same model raises F1 only from 0.449 to 0.492, attributed to the difficulty of recovering prosodic structure from reverberant speech. (§4.3, Table 7)
-
complicates: Data augmentation with noise and reverberation during training does not close the performance gap when those conditions appear at inference time.
Evidence: Despite four-fold augmentation using noise (DEMAND, MUSAN, FSD50K) and room impulse responses (BIRD) at varying SNR levels, macro-F1 drops from 0.912 (clean) to 0.625 (noisy) and 0.449 (noisy-reverberant) on held-out test conditions. (§4, §4.3, Table 7)
-
complicates: Paralinguistic attitude recognition systems trained on reading-speech datasets may overstate real-world spoken dialogue performance, where speech production is more spontaneous and less controlled.
Evidence: The dataset consists entirely of scripts read with intended attitudes by crowd workers and actors; the authors identify generalisation to naturally-occurring speech directed at dialogue systems as unresolved future work. (§3, §5)
Limitations and Open Questions
The corpus is a Japanese reading-speech dataset where speakers deliberately produce each attitude; it does not reflect the ambiguity or variability of spontaneous human-machine dialogue speech. The four-class taxonomy covers core boundary attitudes but may miss finer-grained paralinguistic distinctions. Reverberation robustness remains an open problem: speech enhancement postprocessing helps only marginally, and joint enhancement-recognition models are proposed but not explored. The paper does not address how an inferred attitude label should influence downstream response generation policy within the dialogue system.
Wiki Connections
- Self-Supervised Speech - the system uses HuBERT-large as its core feature extractor; layer 12 embeddings provide prosodically-rich representations sufficient for attitude classification without any linguistic input.
- Evaluation Metrics - macro-F1 is used as the primary metric, and human perception accuracy (0.829) serves as the reference benchmark for evaluating model performance.
- Subjective Evaluation - crowd workers evaluate actor speech utterances to establish human perception accuracy, providing a perceptual baseline against which the automated model is compared.