IWSDS · 2026 · Workshop

Hyuga Nakaguro et al. (Nara Institute of Science and Technology / Institute of Science Tokyo / RIKEN) · → Paper · Demo: ? · Code: ?

Introduces “emotional dialogue breakdown” detection, a paralinguistic extension of text-based dialogue breakdown detection, backed by a new parallel-emotion speech dataset and a FiLM-based detector that treats emotion as a control signal over text interpretation.

Problem

Spoken dialogue systems can produce a response that is textually coherent yet emotionally inappropriate: the same reply text can be correct or jarring depending on the tone in which the user’s utterance was spoken. Prior dialogue breakdown detection research (Higashinaka et al., 2016) evaluates only whether a response is consistent with the textual dialogue context, leaving paralinguistic mismatches unaddressed. Existing emotional speech dialogue resources such as IEMOCAP and MELD confound emotion with textual content, since utterance text varies along with emotion, which makes it impossible to isolate the effect of paralinguistic cues alone on response appropriateness. The paper’s goal is to define and benchmark a form of dialogue breakdown driven purely by mismatches between the emotional tone of an utterance and the emotional appropriateness of its response, holding the text fixed.

Method

The authors construct paraling-dial, a dataset in which 149 short dialogue-style utterance texts (drawn from the copyright-free Aozora Bunko corpus) are each recorded by six professional voice actors (three male, three female) expressing five discrete emotions spanning Russell’s valence-arousal circumplex: happiness, relaxation, anger, sadness, and neutral. For each utterance-emotion combination, a human annotator (assisted by ChatGPT-generated candidates) wrote a response sentence appropriate to that emotional rendition. This yields 149 × 5 × 6 = 4,470 utterance-response pairs (about 284.9 minutes of audio, mean utterance duration 3.92 seconds). Acoustic validity was checked via F0 and RMS energy distributions per emotion (§3.2), which showed separable trends for most emotion pairs.

From this resource, the authors build a binary emotional dialogue breakdown detection task: “Matched” pairs use the response collected for that specific emotion; “Mismatched” pairs are constructed by shuffling in a response originally written for a different emotion label while keeping the same utterance audio, producing pairs where the text is contextually plausible but paralinguistically inconsistent. Matched and Mismatched pairs are balanced 1:1 and split 8:1:1 into train/validation/test.

The detector itself is a discriminative classifier, not a speech generation model. The utterance text and candidate response text are concatenated with a [SEP] token and encoded by a pre-trained SentenceBERT into a 768-dimensional feature vector x. A control signal c representing the emotion (a 5-way one-hot vector in the main configuration) is passed through a fully connected layer to produce a scale γ(c) and shift β(c), which are applied to x via Feature-wise Linear Modulation (FiLM; Perez et al., 2018): FiLM(x, c) = γ(c) ⊙ x + β(c). The modulated vector is passed through a classification head (Linear 768→128 → ReLU → Dropout(0.3) → Linear 128→1 → Sigmoid) to predict Matched/Mismatched. The model is trained with Adam (learning rate 0.001) using standard supervised binary classification. To test the effect of control-signal abstraction, the authors also swap the one-hot emotion vector for two alternatives: a mid-level emotion embedding (from a two-layer network trained for emotion classification on acoustic features, 88.6% classifier accuracy) and low-level raw acoustic features (MFCC, MFCC Δ, MFCC ΔΔ, RMS energy, F0, spectral centroid mean/std). As a separate baseline, the pretrained speech language model Qwen2-Audio is evaluated directly on the breakdown-detection pairs without any task-specific training.

Key Results

Qwen2-Audio, evaluated zero-shot on the emotional dialogue breakdown task, achieves only 50.0% accuracy, statistically indistinguishable from chance on this balanced binary task. The FiLM-based classifier, by contrast, reaches 93.8% accuracy when given gold emotion labels as the control signal, and 91.2% accuracy when given predicted emotion labels from a separately trained classifier (88.3% accuracy). Comparing control-signal abstraction levels (Table 3): the one-hot emotion label (high-level abstraction) gives 91.2% accuracy, the intermediate emotion embedding gives 72.0%, and raw low-level acoustic features give 50.4%, close to chance. The ordering is monotonic in abstraction level, not in raw information content: the lowest-level features, despite containing the most raw signal, perform worst.

Novelty Assessment

The contribution is a dataset and an evaluation task rather than a new generative model or a new architecture. Feature-wise Linear Modulation itself is an existing mechanism (Perez et al., 2018) originally proposed for visual reasoning; the paper’s engineering contribution is applying it to modulate a text encoder’s representation using emotion as a control signal for a dialogue-consistency classification task, which is a straightforward if effective repurposing rather than a structural innovation. The more durable contributions are (1) paraling-dial, a controlled resource that holds text fixed while varying only emotional speech rendition and its appropriate response, isolating a variable that confounds existing emotional dialogue datasets such as IEMOCAP and MELD, and (2) the empirical finding that a general-purpose speech language model (Qwen2-Audio) fails to exploit paralinguistic cues for this task even though it consumes raw audio directly, which is a useful negative result about current SLM capabilities.

Field Significance

moderate — This is a workshop-scale paper (149 base utterances, acted speech, single annotator for response writing) that introduces a genuinely new evaluation axis, paralinguistic consistency between an utterance’s emotional tone and its response, that prior text-only dialogue breakdown detection does not capture. It provides a controlled resource and a concrete negative result about a widely used speech language model’s paralinguistic capability, but the dataset’s small scale, acted delivery, and purely objective (non-human-validated) evaluation limit how far its conclusions can be expected to generalize to natural spoken dialogue.

Claims

  • supports: Paralinguistic (emotional/tonal) mismatch between a spoken utterance and its textual response is a distinct dialogue-breakdown failure mode from textual inconsistency, and can be systematically constructed and measured when parallel emotional recordings of identical text are available.

    Evidence: paraling-dial pairs 149 fixed utterance texts with recordings in five emotions (happiness, relaxation, anger, sadness, neutral) across six professional speakers; Mismatched pairs are built by holding the utterance audio fixed and swapping in a response collected for a different emotion label. (§3.1, §3.3)

  • complicates: A speech language model that consumes raw audio directly is not guaranteed to exploit the paralinguistic cues present in that audio for a downstream consistency-judgment task.

    Evidence: Qwen2-Audio, evaluated zero-shot on the emotional dialogue breakdown detection task, achieves only 50.0% accuracy, at chance level for the balanced binary task. (§6.1)

  • supports: Feature-wise Linear Modulation of a text representation by a discrete emotion signal is an effective way to make a model’s interpretation of response text conditional on the emotional tone of the paired utterance.

    Evidence: The FiLM-conditioned SentenceBERT classifier reaches 93.8% accuracy with gold emotion labels and 91.2% with predicted emotion labels, far exceeding the Qwen2-Audio chance-level baseline. (§4.1, §6.1)

  • refines: When selecting a control signal to represent paralinguistic information for a downstream task, the appropriate level of abstraction matters more than the raw richness of the signal; a higher-abstraction categorical signal can outperform lower-level continuous features.

    Evidence: Accuracy is 91.2% for a one-hot emotion-label control signal, 72.0% for an intermediate emotion embedding, and 50.4% for raw low-level acoustic features (MFCC, F0, RMS, spectral centroid) used as the FiLM control signal. (§6.2, Table 3)

  • complicates: Conclusions drawn from acted, single-annotator emotional speech resources evaluated only with objective accuracy metrics may not transfer directly to natural spontaneous emotional dialogue or to human judgments of conversational breakdown.

    Evidence: The authors note paraling-dial consists of acted speech from professional voice actors instructed to emphasize emotional differences, and that the current evaluation relies solely on objective accuracy without a subjective human study of perceived breakdown. (§6.3)

Limitations and Open Questions

The paraling-dial dataset is composed entirely of acted, deliberately exaggerated emotional speech from professional voice actors and trained-annotator-written responses, and all reported results (93.8%/91.2% accuracy) are objective classification metrics with no human perceptual validation that detected "breakdowns" correspond to how real listeners would judge conversational appropriateness.

Beyond the acted-speech and objective-only-evaluation limitations the authors themselves flag, the task is restricted to five discrete emotion categories rather than continuous emotional variation, and the dataset is limited to Japanese short dialogue-style utterances drawn from a single literary source (Aozora Bunko), leaving open how the approach transfers to other languages, domains, and more fine-grained or continuous emotional representations. The authors explicitly propose extending to spontaneous speech, continuous emotion vectors, and subjective user studies as future work (§7).

Wiki Connections

  • Evaluation Metrics — introduces a new benchmark task and paired dataset for measuring whether spoken dialogue systems can detect mismatches between an utterance’s emotional tone and its response, extending text-only dialogue breakdown detection into the paralinguistic domain.
  • Spoken Language Model — evaluates a general-purpose speech language model (Qwen2-Audio) as a baseline and finds it performs at chance level on paralinguistic mismatch detection, contributing evidence about a specific capability gap in current speech language models.
  • Emotion Synthesis — the paraling-dial resource is a parallel emotional speech corpus (identical text rendered in five emotions per speaker) that could support future emotion-conditioned TTS or voice conversion training and evaluation, although this paper’s own contribution is detection rather than synthesis.
  • Qwen2-Audio Technical Report — used as an off-the-shelf speech language model baseline for the emotional dialogue breakdown task, where it performs at chance level.
  • WavReward — cited as related work on evaluating spoken dialogue quality by jointly considering cognitive and emotional aspects of speech.