SIGDIAL 2026 · 2026 · Workshop
Yash Raj Singh (Independent Researcher) · → Paper · Demo: ✓ · Code: ?
Describes a deployed cascade Hinglish voice assistant that cuts perceived response latency by flushing the LLM token stream to a commercial TTS engine at script-aware clause boundaries and synthesizing those chunks concurrently with ongoing generation and playback.
Problem
Cascade spoken dialogue systems (ASR, then an LLM, then TTS) add up the latency of each stage, which puts them far from the roughly 200 ms gaps of human turn-taking. Streaming text to TTS word by word lowers latency but, the paper argues, deprives the synthesizer of the intonation context it needs. Waiting for whole sentences keeps prosody but raises latency. The problem is sharper for Hinglish (Hindi-English code-switching), where Devanagari and Latin script, and SOV and SVO word orders, mix inside one utterance. The paper argues that end-to-end speech-to-speech models such as AudioPaLM are costly at scale and handle dense intra-sentential code-switching poorly, and that cascades keep modularity and deterministic control over persona guardrails.
Method
CrossOracle is a Node.js WebSocket middleware between Azure Cognitive Services (ASR and LLM) and ElevenLabs Turbo v2.5 (TTS). Audio travels as 16 kHz PCM over bidirectional WebSockets. No model is trained. The contribution is the orchestration logic around the commercial services, in four parts.
Hinglish semantic boundary detection (§3.1). A checkFlush() heuristic runs in the LLM token callback and decides when to send the text buffer to TTS. It checks three trigger classes in priority order: the Devanagari danda or double danda; sentence-final ., ? or ! after at least three alphabetic characters, with an abbreviation blocklist (Dr., Mr.); and structural boundaries (newlines, em dashes, ellipses). A 30-character minimum keeps fragments out of the TTS engine, and a 220-character cap bounds latency when punctuation is sparse.
Asynchronous TTS worker (§3.2). Flushed chunks go into a ttsQueue. A background worker (the “First-In-First-Stream” worker) sends each chunk to the ElevenLabs API independently of the LLM loop and forwards the returned PCM to the client at once. Perceived latency becomes the sum of ASR latency, LLM time-to-first-token, the time to accumulate the first chunk, and TTS latency for that first chunk (Eq. 1). Later chunks are synthesized while earlier ones play. Playback stays gapless as long as the cumulative synthesis time of the first k chunks stays below the cumulative audio duration of the first k-1 chunks (Eq. 2).
Deterministic barge-in (§3.3). An interrupt message, triggered by voice activity detection or a UI button, makes the server call a global AbortController. This cancels the LLM stream and any in-flight TTS requests and clears the queue, so no stale “ghost audio” plays after the user speaks.
ASR mishear filter and personas (§3.4, §3.5). Azure STT is locked to the hi-IN locale to skip language identification. Each transcript’s Devanagari character ratio r is computed. Transcripts with r < 0.1 and more than 8 characters are discarded as hallucinated Latin strings. Accepted transcripts had a mean r of 0.61, against about 0.02 for the hallucinated ones. Eleven preset expert personas (doctor, lawyer, therapist and others) each have a system prompt that enforces Devanagari for Hindi words, a distinct ElevenLabs voice, and a pre-synthesized, edge-cached greeting. Custom personas are built from a free-text description through a keyword-to-voice-bucket routing table.
Key Results
All results are server-side telemetry from 8,000 logged turns across the preset personas (§4, Table 1). There is no human evaluation.
- Time to first byte: mean TTFB is 4.9 s for CrossOracle against 6.5 s for a “Naive Buffer” that flushes after a fixed token count with no boundary detection, a 1.6 s reduction. The paper does not say whether the Naive Buffer numbers come from the same 8,000 turns or from a separate run.
- TTS real-time factor: mean RTF (synthesis time over audio duration) is 0.107 against 0.134 for the Naive Buffer. The paper gives no explanation for why the chunking policy changes the API’s RTF.
- Gapless playback: the continuous-playback condition (Eq. 2) held across all 8,000 turns, with no mid-sentence buffering (§3.2). One edge case broke it: an isolated emoji as the final LLM token took 233 ms to synthesize 93 ms of audio (RTF 2.513). A regex filter that strips non-lexical Unicode before TTS was later deployed (§4).
- Barge-in: 100% of interruptions were resolved, with a server-side abort time under 1 ms. This measures
AbortControllerexecution, not the time until the client’s audio stops. - Greetings: pre-synthesized greetings start in 866 ms, against 4,900 ms for a cold LLM turn (§3.5).
Even with the pipeline changes, the 4.9 s mean TTFB is far above the 200 ms turn gap the paper uses as its target (§1). The paper does not break down which stage of Eq. 1 dominates.
Novelty Assessment
The work is engineering integration on top of closed commercial services. Sending LLM output to TTS in chunks while generation continues is a common pattern in cascade voice agents. The specific additions are the script-aware flush rules for Hinglish (danda plus Latin punctuation with length bounds), the explicit gapless-playback condition stated in terms of RTF, the Devanagari-ratio mishear filter, and a cancel-everything barge-in path. None of these is evaluated in isolation beyond the single Naive Buffer comparison. The claim that semantic chunking “preserves prosodic integrity” across code-switching boundaries is not measured. The paper defers MOS evaluation to future work.
Field Significance
Low — The paper is a single-author demonstration that gives a deployment data point for cascade Hinglish spoken conversational agents: clause-level flushing to a commercial TTS engine lowers time to first byte against fixed-token flushing, and per-chunk RTF can be used to reason about gapless playback. With automatic-only telemetry, one baseline, and no listener study, it serves as an engineering reference rather than evidence about synthesis quality.
Claims
- supports: In cascade spoken dialogue systems, dispatching LLM output to TTS at linguistically motivated clause boundaries, rather than after a fixed token count, lowers time to first audio.
Evidence: With script-aware flushing (Devanagari danda, Latin sentence punctuation, structural breaks, 30-220 character bounds), mean TTFB is 4.9 s against 6.5 s for fixed-token-count flushing over 8,000 logged Hinglish turns. (§3.1, §4, Table 1)
- supports: Overlapping TTS synthesis of earlier text chunks with ongoing LLM generation and playback removes TTS time from all but the first chunk of perceived response latency, provided synthesis runs well below real time.
Evidence: Perceived latency reduces to ASR latency, LLM time-to-first-token, first-chunk accumulation and first-chunk TTS latency (Eq. 1); at a mean RTF of 0.107 the gapless condition (Eq. 2) held across all 8,000 turns. (§3.2, Eq. 1, Eq. 2, Table 1)
- complicates: A low mean real-time factor does not guarantee gapless chunked playback, because short or non-lexical chunks can have a per-chunk RTF above one.
Evidence: An isolated emoji emitted as the final LLM token took 233 ms to synthesize 93 ms of audio (RTF 2.513), inverting the playback condition at end of turn and requiring a pre-TTS filter for non-lexical Unicode. (§4)
- complicates: Chunking and parallel dispatch in a cascade pipeline can lower time to first audio while leaving it well above human turn-taking gaps, because ASR and LLM time-to-first-token remain on the critical path.
Evidence: The mean TTFB of 4.9 s is far above the roughly 200 ms inter-turn gap cited as the target, and pre-synthesized greetings that skip the LLM start in 866 ms. (§1, §3.5, Table 1)
Limitations and Open Questions
Warning
There is no human evaluation of the spoken output. The central claim that clause-level flushing preserves prosody across Hinglish code-switching is asserted, not measured, and MOS testing is left to future work (§4, §6).
- All speech and language components are closed commercial APIs (Azure ASR and LLM, ElevenLabs Turbo v2.5), so the latency and RTF numbers depend on those services and their versions.
- The only baseline is a fixed-token-count Naive Buffer. The paper does not report sentence-level buffering, word-level streaming, or a no-parallelism condition, and does not say how the Naive Buffer numbers were collected.
- Latency is reported only as means, with no distributions, percentiles, or per-stage breakdown of the 4.9 s TTFB.
- The barge-in figure is server-side
AbortControllerexecution time. Client-side audio stop time and false or missed interruptions from voice activity detection are not measured. - The mishear filter discards transcripts silently. Its false-rejection rate on genuine English-heavy Hinglish speech is not reported.
- No code or log data release is stated; the system is available only as a live deployment.
Wiki Connections
- Speech-to-Speech — CrossOracle is a cascade voice-in, voice-out dialogue system (ASR, LLM, TTS) that targets end-to-end response latency for code-switched Hinglish conversation.
- Streaming TTS — the system streams LLM text to a commercial TTS engine in clause-level chunks synthesized concurrently with playback, and measures TTFB and per-chunk RTF against fixed-token flushing.
- AudioPaLM — cited as an end-to-end speech-to-speech alternative that the paper argues is too costly at scale and weak on dense code-switching, motivating its cascade design.
- FastSpeech 2 — cited in related work as the example of streaming TTS whose word-level streaming the paper contrasts with its clause-level flushing.