ACL 2026 (Findings) · 2026 · Conference

Maike Züfle et al. (Karlsruhe Institute of Technology / University of Edinburgh / NatWest) · → Paper · → Preprint · Demo: ? · Code: ✓

F-Actor fine-tunes a 1B-parameter text LLM into a dual-stream full-duplex speech model whose voice, topic, dialogue initiation, and backchannel and interruption counts are set by a prompt prefix. It trains in a single stage on about 2,000 hours of synthetic dialogue, and the paper ablates the text stream, text alignment, audio delay, loss scope, and codec type.

Problem

Full-duplex speech models (dual-stream designs such as Moshi and the NVIDIA duplex S2S model, or interleaved single-stream designs) learn to cope with user interruptions and backchannels. They are rarely evaluated on whether the system itself backchannels or interrupts, so evaluation focuses on reactive rather than proactive behaviour. Most released models also offer little control over voice, persona, topic, or interaction style. The few instruction-following duplex systems the authors identify (MinMo, Voila, and the dialogue generator in Behavior-SD) are not released, need costly multi-stage training, and do not evaluate their control abilities systematically. The authors describe F-Actor as the first open, controllable full-duplex model.

Method

Backbone and speech representation. The backbone is Llama-3.2-1B-Instruct (Llama 3). Speech enters and leaves through NanoCodec (22 kHz, 0.6 kbps, 12.5 fps), whose four FSQ codebooks (size 4032) are independent. This lets the model predict all four codes of a frame in parallel without a depth transformer. The codec encoder and decoder stay frozen; only the LLM and the output heads are trained.

Dual-stream input. The user and system streams each produce four discrete acoustic unit (DAU) sequences. Each codebook of each stream has its own embedding layer, sized to the LLM hidden dimension, rather than extending the LLM vocabulary. The eight embeddings are summed into one input vector per frame. The authors choose dedicated embeddings to avoid a large joint softmax over text tokens and DAUs.

Output heads. Eight linear heads (four user, four system) map the final hidden state to DAU logits. At inference only the four system heads are sampled, and the NanoCodec decoder produces the waveform. An optional system-side text stream is embedded, added to the system speech embedding, and predicted with the LLM’s original language-model head (an inner-monologue design in the style of Moshi and the NVIDIA duplex model).

Instruction prefix. A text prompt states the narrative from the system’s perspective, whether the system starts the dialogue, and exact counts of backchannels and interruptions. An ECAPA-TDNN speaker embedding from five seconds of target speech is linearly projected into the token space. Speaker embedding and prompt embeddings are prepended to the audio stream. The Behavior-SD narratives were rewritten into second person (Gemma-1.1-7b-it for training, GPT-5.1 for test), because about 85% of original narratives open with the speaker who starts the conversation, which let the model ignore the explicit start instruction.

Training. A single supervised stage on Behavior-SD (2,164 hours of English two-speaker dialogue synthesized with CosyVoice, with narrative, backchannel, and interruption annotations). Kaldi forced alignment provides word- or utterance-level text-audio alignment; conversations with text-speech mismatches are dropped, leaving 74.4% of the training set. Speakers are restricted to the 52 Behavior-SD voices, a deliberate choice against voice cloning. Training uses four A100-40GB GPUs for about 46 to 48 hours per model, learning rate 5e-5, early stopping on validation (Table 6). Inference samples at temperature 0.9 with top-k 40.

Ablated choices. Loss on system tokens only (s) or on both user and system tokens (s/u); with or without text stream; utterance- or word-level alignment; audio delay of 0, 1, or 2 tokens after text; special tokens before backchannels and interruptions; and Mimi (RVQ, 8 codebooks, still predicted in parallel) in place of NanoCodec.

Evaluation protocol. Two model instances converse with each other, each with its own instructions. Metrics are DAU and text perplexity, segment-averaged UTMOS, WER between the Parakeet transcript of generated audio and the model’s own text stream, speaking-time difference, correct-start accuracy, ECAPA-TDNN speaker similarity and drift, narrative adherence from a Llama-3.1-8B-Instruct judge (1 to 5, three seeds), and Pearson correlation between prompted and detected backchannel and interruption counts. Event detection uses Parakeet word timestamps with grid-searched thresholds. This replaced the FD-Bench Silero-VAD defaults, which misclassified events on Behavior-SD. On the Behavior-SD test set, Parakeet word-level detection misses 1.0 ± 1.4 backchannels and 0.5 ± 0.7 interruptions per dialogue, versus 1.6 ± 1.7 and 1.0 ± 1.0 for Silero-VAD (Table 5).

Key Results

General modelling (Table 1). Adding a text stream lowers DAU perplexity from 25.23 (audio only) to 21.45 to 25.47 across text-stream variants. Text-audio WER depends on alignment and delay. Utterance alignment gives 23.19 / 17.35 / 15.18% at delay 0 / 1 / 2, and word alignment gives 33.40 / 19.53 / 8.35%. UTMOS rises from 3.17 to 3.40 as word-aligned delay grows from 0 to 2. The system-only loss variant has the lowest DAU perplexity (21.45), with WER 7.59% and UTMOS 3.41. Removing the speaker embedding gives the highest UTMOS (3.47) and lowest WER (7.16%) among the trained models. Behavior-SD ground truth scores UTMOS 3.78 and WER 4.5%. The authors attribute at least 1 point of WER to words seen fewer than 100 times in training, which appear correctly in text but are mispronounced.

Instruction following (Table 2).

ConfigurationCorrect start (%)Spk. sim.Narrative (1-5)BC corr.Interr. corr.
Behavior-SD dialogues (topline)100.000.623.90 ± 0.030.920.74
Lower baselines50.0 (random)0.35 (swapped)1.26 ± 0.02 (random narrative)--
s/u, no text stream54.00.521.50 ± 0.030.560.21
s/u, utt. align, delay 299.60.532.30 ± 0.000.360.16
s/u, word align, delay 089.80.521.74 ± 0.020.530.23
s/u, word align, delay 299.80.542.78 ± 0.020.540.25
s/u, no speaker emb., word, delay 299.40.362.53 ± 0.050.500.19
s only, word, delay 299.60.542.46 ± 0.060.570.17
s/u + BC/I tokens, word, delay 2100.00.542.49 ± 0.020.480.28
  • Dialogue initiation depends on the text stream. Audio-only models are near chance (54.0%), while text-stream models with audio delay reach 99.0 to 100.0%.
  • Voice control works within the closed speaker set. With the speaker embedding, similarity is 0.52 to 0.54, against 0.36 without it and 0.35 for swapped embeddings. Drift (1 minus cosine, first vs. last segment) is 0.61 to 0.67, against 0.62 for the test set.
  • Narrative adherence peaks at 2.78 with word alignment and delay 2, still well below the 3.90 topline. The system-only loss lowers it to 2.46.
  • Behaviour counts correlate significantly with the prompt (all p < 0.01), but correlations stay far below the topline: 0.54 for backchannels and 0.25 for interruptions in the main configuration. Special BC/interruption tokens raise interruption correlation (0.28) but lower backchannel correlation (0.48) and narrative adherence (2.49). The Limitations section states the model consistently produces fewer events than requested.

Human check of the LLM judge (Table 3, §C.3). Twelve computer-science researchers rated narrative fit of 80 dialogue transcripts (240 ratings) for Behavior-SD and the two strongest configurations: human 4.17 / 3.00 / 2.68 vs. judge 3.88 / 2.68 / 2.33. System-level Kendall’s tau is 1.00 and item-level tau is 0.507; inter-annotator item-level tau is 0.517. Raters judged text, not audio, so this validates the judge’s ranking but is not a listening test.

Turn-taking statistics (Table 4, cumulative seconds per minute). F-Actor produces IPU 59.3 s, pause 10.4 s, gap 3.0 s, overlap 5.4 s, against its own training data (Behavior-SD ground truth) at 55.8 / 10.8 / 3.8 / 3.0 s. Reported reference rows are dGSLM best non-cascaded 41.4 / 13.8 / 10.7 / 6.1 s and best cascaded 54.8 / 0.0 / 5.3 / 0.0 s (ground truth 53.5 / 5.5 / 4.4 / 3.6 s), and Moshi 50.8 / 7.0 / 4.5 / 4.1 s (ground truth 51.1 / 6.4 / 4.2 / 3.3 s). The dGSLM and Moshi numbers come from their own papers against their own ground-truth data, so this is a distributional comparison rather than a same-test-set benchmark. F-Actor overlaps more than its training data (5.4 vs. 3.0 s) and leaves shorter gaps (3.0 vs. 3.8 s).

Sampling temperature (Table 7, dev set). Raising temperature from 0.6 to 1.0 increases backchannel correlation (0.30 to 0.52) and lowers the speaking-time difference (24.29 to 12.07 s). It also lowers UTMOS (3.48 to 3.38) and raises WER (6.2 to 10.09%). Narrative adherence is non-monotonic (2.57, 2.26, 2.78, 2.08 at 0.6, 0.8, 0.9, 1.0). Temperature 0.9 is used throughout.

RVQ vs. FSQ (Table 8). Swapping NanoCodec for Mimi while still predicting all codebooks in parallel keeps instruction-following close (narrative 2.83 vs. 2.78, correct start 100.0 vs. 99.8%) and WER lower (7.18 vs. 8.35%). Speech quality falls sharply: UTMOS 2.50 vs. 3.40, speaker similarity 0.40 vs. 0.54, backchannel correlation 0.40 vs. 0.54.

No released instruction-following duplex model exists for direct comparison, so all instruction-following numbers are judged against the training-data topline and constructed lower bounds.

Novelty Assessment

The architecture is close to the dual-stream, FSQ-codec, multi-head design of the NVIDIA duplex S2S model, with two changes: dedicated per-codebook embeddings instead of vocabulary extension, and a prepended instruction prefix (text prompt plus projected ECAPA-TDNN embedding). The behaviour-count prompts follow Behavior-SD, which already conditioned a dialogue generator on similar annotations. The contribution is therefore an engineering integration that turns an annotated synthetic dialogue corpus into a controllable, system-side duplex model on an academic budget, plus a release of model and code.

The more durable value is the controlled ablation. It shows that a text stream is what makes discrete instructions like dialogue initiation learnable, and that word alignment and audio delay are needed together for text-audio consistency. It also reports negative findings: special BC/interruption tokens and system-only loss do not improve overall instruction following, and parallel RVQ prediction costs speech quality. The evaluation protocol (self-play, tuned event detection, judge validated against humans) is useful but self-referential. Training, test, speakers, and topline all come from the same synthetic corpus.

Field Significance

Moderate — The paper shows that system-side conversational behaviour in a full-duplex model can be steered by a prompt and measured, and that a 1B backbone fine-tuned in one stage on about 2,000 hours suffices for coarse control. Its ablations give concrete guidance on text streams, alignment, and audio delay for dual-stream speech LMs. The weak correlation for interruptions and the synthetic-only setting make it a starting point rather than a solved capability.

Claims

  • supports: In dual-stream full-duplex speech language models, an auxiliary text stream is needed for the model to follow discrete control instructions such as who should start the dialogue.

    Evidence: Audio-only models reach 54.0% correct-start accuracy against a 50% chance baseline, while text-stream models with an audio delay reach 99.0 to 100.0%; narrative adherence rises from 1.50 to as high as 2.78. (§7.2, Table 2)

  • supports: Word-level text-speech alignment combined with delaying audio tokens behind text tokens improves consistency between the generated text and speech streams in full-duplex speech LMs.

    Evidence: WER between the transcribed audio and the generated text falls from 33.40% (word alignment, no delay) to 8.35% (word alignment, delay 2), against 15.18% for utterance alignment at delay 2; UTMOS rises from 3.17 to 3.40 over the same word-aligned delay sweep. (§7.1, Table 1)

  • complicates: Prompting full-duplex speech models for specific counts of backchannels and interruptions yields directional rather than precise control, especially for rare events.

    Evidence: Prompted and generated counts correlate at 0.54 (backchannels) and 0.25 (interruptions) against 0.92 and 0.74 for the training data; the model consistently produces fewer events than requested, which the authors link to interruptions averaging 0.9 per training conversation. (§7.2, Table 2; §Limitations)

  • complicates: Predicting all codebooks of a residual vector quantizer in parallel preserves dialogue-level behaviour but degrades speech quality compared with an independent-codebook (FSQ) codec.

    Evidence: Replacing NanoCodec with Mimi under the same parallel heads keeps narrative adherence (2.83 vs. 2.78) and correct start (100.0 vs. 99.8%) but drops UTMOS from 3.40 to 2.50 and speaker similarity from 0.54 to 0.40. (§7.5, Table 8)

  • refines: In full-duplex speech LMs, sampling temperature trades conversational dynamics against speech quality.

    Evidence: From temperature 0.6 to 1.0, backchannel correlation rises from 0.30 to 0.52 and speaking-time imbalance falls from 24.29 to 12.07 s, while UTMOS falls from 3.48 to 3.38 and text-audio WER rises from 6.2 to 10.09%. (§7.4, Table 7)

Limitations and Open Questions

Warning

Training, testing, the 52 target voices, and the topline all come from Behavior-SD, a corpus synthesized with CosyVoice TTS. Evaluation is self-play between two F-Actor instances with no real human interlocutor and no listening test, so naturalness, voice control, and behaviour control on real conversational speech remain untested.

  • Backchannel and interruption counts act as a direction (more or fewer) rather than an exact target, and the model under-produces both (§Limitations).
  • NanoCodec has no streaming encoder, so deployment is limited to chunk-wise processing; latency is not reported.
  • Voice control is limited by design to a closed set of training speakers; generalization to unseen voices is not evaluated.
  • Speech quality (UTMOS 3.40 vs. 3.78 for the data) and narrative adherence (2.78 vs. 3.90) trail the training data. Some WER comes from mispronounced rare words.
  • The human study rates transcripts for narrative fit only; it does not assess audio quality or conversational naturalness.
  • Comparison with dGSLM and Moshi uses numbers reported against different ground-truth corpora.
  • English only, because comparable duplex data with behaviour annotations does not exist for other languages.

Wiki Connections

  • Spoken Language Model — F-Actor adapts a text LLM to consume a live user speech stream as codec tokens and to generate system speech under prompt-specified behavioural constraints.
  • Speech-to-Speech — The model is an end-to-end full-duplex spoken dialogue system that listens and speaks simultaneously, with ablations of text-stream, alignment, and delay choices.
  • Behavior-SD — Provides the full training and test corpus, the behaviour annotations, and the prompt format that F-Actor adapts to a system-side duplex model.
  • Efficient and Direct Duplex Modeling for Speech-to-Speech Language Model — The closest architectural precedent (dual stream, FSQ codec, parallel heads); F-Actor freezes the codec encoder, uses dedicated DAU embeddings, and adds an instruction prefix.
  • Moshi — Source of the dual-stream and inner-monologue text-stream design, the Mimi codec used in the RVQ ablation, and a reference point for turn-taking statistics.
  • Llama 3 — Llama-3.2-1B-Instruct is the fine-tuned backbone, and Llama-3.1-8B-Instruct serves as the narrative-adherence judge.
  • Finite Scalar Quantization — FSQ’s independent codebooks motivate predicting all codec streams in parallel without a depth transformer.
  • FD-Bench — Its Silero-VAD event-detection defaults were tested and replaced with tuned Parakeet timestamps for counting backchannels and interruptions.
  • MinMo — Cited as an instruction-controllable duplex system that is unreleased and evaluated on control only in its TTS setting, motivating F-Actor’s open release.
  • UTMOS — Used as the automatic speech-quality metric throughout the ablations.