arXiv · 2026 · Preprint
Hugo L. Hammer et al. (Oslo Metropolitan University, SimulaMet) · → Paper · Demo: ? · Code: ✓
Introduces Calliope, an open-source, offline pipeline that converts a standard EPUB e-book into an EPUB 3 Media Overlay narrated e-book, using neural TTS to generate the narration and capturing synchronization timestamps directly from the synthesis process rather than via post-hoc forced alignment.
Problem
Narrated e-books, which highlight the currently spoken word or sentence in sync with audio playback, support early literacy and accessibility for readers with dyslexia or other reading challenges. Commercial services (ElevenReader, NaturalReader) already convert text e-books into narrated e-books using neural TTS, but no open-source solution performs this task end to end. The open-source and partly open-source alternatives that do exist have structural weaknesses: forced-alignment tools such as Syncabook and Storyteller require pre-existing audio and map it back onto text using dynamic time warping or transcription-based fuzzy matching, both of which are prone to synchronization drift; Epub2tts produces plain audiobooks without synchronized text highlighting at all; and Audible-epub3-maker produces synchronized EPUB 3 output but depends on commercial cloud TTS APIs, incurring recurring cost and requiring upload of potentially copyrighted or sensitive text to third-party servers. Existing open-source pipelines also typically use a “reconstruction” strategy that strips the publisher’s original CSS and layout when rebuilding the EPUB, degrading visual fidelity. Calliope targets all three gaps at once: exact synchronization, privacy-preserving offline operation, and layout preservation.
Method
Calliope is a three-phase pipeline that transforms a static EPUB into an EPUB 3 file with Media Overlays (the SMIL-based standard that binds text spans to audio time intervals).
Phase 1: text extraction and layout preservation. The pipeline traverses the DOM of each XHTML chapter to find block-level elements (paragraphs, headers) containing narrative text, normalizes the text, and segments it into sentences with a standard tokenizer. For each sentence, a <span> element with a globally unique ID is injected directly into the existing DOM node, strictly inside the original block, which preserves the parent element’s CSS attributes instead of rebuilding the document from scratch. This segmentation simultaneously defines the input units for TTS synthesis and the anchor points for text highlighting.
Phase 2: TTS synthesis and synchronization. Each sentence is synthesized independently by a neural TTS model conditioned on a short (~15 second) reference audio clip that defines the narrator’s voice (a zero-shot voice-cloning setup). The framework supports two backends: XTTS-v2, an autoregressive GPT-2-style transformer that predicts discrete audio tokens from text and a speaker embedding, decoded to waveform via a HiFi-GAN-based decoder, offering zero-shot voice cloning and cross-lingual prosody transfer at the cost of sequential (non-parallel) inference; and Chatterbox, a non-autoregressive flow-matching model built on a 0.5B-parameter Llama transformer backbone that solves a continuous probability-flow ODE for faster, fewer-step synthesis. Because both are Transformer-based models with fixed context windows, the pipeline applies a two-tier safeguard: sentences longer than a character threshold (e.g., 200 characters) are recursively split at the nearest whitespace, and sentences shorter than a minimum threshold (e.g., 60 characters) are merged with neighbors to avoid known degradation on very short inputs; separately, if synthesis raises a token-overflow error at inference time (e.g., from phonetically dense inputs like spoken-out years), the offending sentence is recursively split and the resulting audio segments are concatenated. Per-sentence waveforms are stitched into a continuous chapter-level audio stream: a short linear fade-out (e.g., 50 ms) is applied at each segment boundary to suppress clicks and hallucinated breath sounds, and a fixed silence pad (e.g., 0.15 s) is inserted between segments. Because the pipeline knows the exact duration of every synthesized and padded segment, it derives each sentence’s SMIL clip-begin and clip-end timestamps directly by summing prior segment durations, with no separate alignment step. The active interval of each sentence is extended to the start of the next sentence so that highlighting stays active across the inter-sentence silence, avoiding a highlight “flicker” effect.
Phase 3: packaging and accessibility. A SMIL file is generated per chapter mapping structural span IDs to the computed time intervals. A CSS rule keyed on the prefers-color-scheme media query adapts the highlight color for light versus dark reading modes. The OPF package manifest is updated with media:duration metadata per chapter and in total, and the container is re-zipped into a validated EPUB 3.3-compliant file with Media Overlays. The implementation is released as an MIT-licensed Python library with separate CLI entry points for the Chatterbox and XTTS-v2 backends, runnable on consumer hardware with optional GPU acceleration.
Key Results
The paper’s central empirical comparison (§6) evaluates whether a forced-alignment alternative (generate audio first, then align text to it) would have been an acceptable substitute for Calliope’s built-in timestamp capture. Using Chatterbox-generated narration of a single Project Gutenberg e-book (“The Gift of the Magi”), the authors compared Calliope’s timestamps against two forced-alignment tools: Syncabook (DTW-based, via Afaligner) and Storyteller (Whisper-transcription plus fuzzy string matching), restricting the comparison to the 85% of sentences both methods tokenized identically. Calliope’s drift was exactly zero by construction, for every quantile reported. Syncabook’s drift ranged from -0.96 s to 0.49 s (mean -0.03 s, median -0.01 s), with the paper noting that over 30% of sentences exceeded the [-50, 150] ms window judged acceptable for reading comfort. Storyteller’s drift was far worse and more variable, ranging from -34.91 s to 1.6 s (mean -1.11 s), with only 6.6% of sentences falling inside the acceptable window. A qualitative feature comparison (Table 1) also positions Calliope against Syncabook, Storyteller, Epub2tts, and Audible-epub3-maker on synchronization strategy, privacy (local vs. cloud), layout handling (preserved vs. reconstructed), cost, and synchronization accuracy, with Calliope the only entry marked “preserved” layout, “local” privacy, “free” cost, and “exact” synchronization simultaneously.
Novelty Assessment
Calliope does not introduce a new TTS architecture; it integrates two existing open-source TTS systems (XTTS-v2, Chatterbox) unmodified. The genuine contribution is at the systems-engineering level: deriving exact synchronization timestamps directly from the synthesis process instead of using any alignment algorithm at all, which by construction eliminates the drift class of failure that forced-alignment approaches exhibit; a “surgical” DOM-insertion strategy that preserves original publisher CSS and layout rather than reconstructing the document, addressing a specific weakness the authors identify in prior open-source tools; and a practical recursive text-segmentation and exception-handling scheme for keeping long-form book text within TTS context-window limits. The offline/local design is an engineering choice (privacy, cost, copyright) rather than a technical advance. The quantitative validation in §6 is informative but narrow: it evaluates a single short story rather than a range of books, languages, or TTS backends, and it directly follows from how timestamps are computed rather than testing a nontrivial hypothesis. Overall this is an engineering-integration contribution that assembles existing TTS and EPUB standards into a coherent, evaluated open-source tool, not a new speech-generation method.
Field Significance
moderate — This paper contributes a practical, evaluated open-source system for a task (synchronized narrated e-books) that previously depended on either forced alignment (with measurable drift problems) or proprietary cloud APIs. It demonstrates that timestamp capture during TTS synthesis is a workable alternative to post-hoc alignment for this application, and quantifies the drift shortfall of two forced-alignment baselines on one test book. Its contribution is bounded to an applied integration of off-the-shelf TTS models rather than to TTS methodology itself.
Claims
- supports: Deriving audio-text synchronization timestamps directly from the TTS synthesis process, rather than aligning independently generated audio to text after the fact, can eliminate the alignment-drift failure mode that post-hoc forced-alignment methods exhibit.
Evidence: On a single test e-book, Calliope’s timestamp-derived synchronization showed zero drift across all evaluated sentences, while a DTW-based forced-alignment baseline (Syncabook) showed drift up to 0.96 s (over 30% of sentences exceeding the acceptable [-50, 150] ms window) and a transcription-based baseline (Storyteller) showed drift up to 34.91 s with only 6.6% of sentences within the acceptable window. (§6, Table 2, Figure 3)
- complicates: The fixed context-window limits of Transformer-based TTS models (both autoregressive and flow-matching) can cause synthesis failures or truncation on long-form, sentence-segmented input text unless explicitly mitigated at the pipeline level.
Evidence: The framework applies a two-tier handling strategy: sentences longer than ~200 characters are recursively split at the nearest whitespace, sentences shorter than ~60 characters are merged with neighbors to avoid known short-input degradation, and a runtime exception handler recursively splits any sentence that triggers a token-overflow error during decoding (e.g., from phonetically dense inputs such as spoken years). (§4.2, Equations 2-3)
- complicates: Concatenating independently synthesized sentence-level audio segments into a continuous stream introduces audible boundary artifacts that must be explicitly suppressed, a problem the authors note is common in flow-matching-based TTS output.
Evidence: A linear fade-out filter (~50 ms) is applied to the end of each synthesized segment and a fixed silence pad (~0.15 s) is inserted between segments specifically to mitigate boundary clicks and hallucinated breath sounds before concatenation. (§4.2.1)
- supports: A short reference audio clip can serve as a practical, low-effort conditioning signal for zero-shot voice cloning across an entire long-form document, without per-book or per-speaker fine-tuning.
Evidence: Calliope requires only an approximately 15-second reference WAV sample to define the narrator voice profile for an entire e-book, applied consistently across all chapters via either the XTTS-v2 or Chatterbox backend, running fully offline on consumer hardware. (§5.2)
Limitations and Open Questions
The synchronization-drift evaluation in §6 is conducted on a single short story (“The Gift of the Magi”), synthesized with one TTS backend (Chatterbox) and compared against two forced-alignment tools; it does not establish how drift behavior generalizes across longer books, other languages, or the XTTS-v2 backend. The Storyteller comparison also excludes roughly 15% of sentences where its independent tokenizer disagreed with Calliope’s, which could bias the comparison in either direction. The paper reports no subjective evaluation (e.g., listening tests or MOS) of the narration audio quality itself, only synchronization accuracy; overall narration quality is inherited entirely from the underlying XTTS-v2 or Chatterbox models and is not separately assessed. The character-length thresholds used for sentence splitting and merging (200 and 60 characters) are stated as example values without a reported ablation showing they are optimal. The authors identify a GUI and quantized/mobile deployment as future work rather than complete in this version.
Wiki Connections
- Zero-Shot TTS — Calliope relies on the zero-shot voice-cloning capability of its TTS backends to narrate an entire e-book from a single short reference audio clip, without any per-book fine-tuning.
- Flow Matching — one of the two supported TTS backends, Chatterbox, is a flow-matching model with a 0.5B-parameter Llama backbone, and the paper notes flow-matching models as a source of boundary/breath artifacts that its concatenation pipeline must mitigate.
- Autoregressive Codec TTS — the other supported backend, XTTS-v2, is an autoregressive GPT-2-style transformer that predicts discrete audio tokens from text before decoding them to waveform, the pattern this concept covers.
- XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model — one of the two TTS backends Calliope integrates for zero-shot narrator voice cloning and cross-lingual prosody transfer.
- Flow Matching for Generative Modeling — provides the flow-matching formulation underlying Chatterbox, the second TTS backend Calliope supports.
- Robust Speech Recognition via Large-Scale Weak Supervision — Whisper is the transcription model used by the Storyteller forced-alignment baseline that Calliope’s deterministic synchronization is shown to outperform on drift accuracy.