arXiv · 2026 · Preprint
Ranjith M. S. et al. (Smallest AI) · → Paper · Demo: ? · Code: ?
Demonstrates that a production diffusion-based TTS system can run at over 95% low-fidelity (LoFi) compute and more than 80% BlockFloat8 deployment on Tenstorrent hardware without measurable audio quality loss, cutting on-prem accelerator cost roughly 4x versus an NVIDIA L40S baseline at matched throughput.
Problem
Inference cost, not training cost, is now the dominant economic factor for production TTS deployments, particularly for latency-sensitive and on-prem serving. Aggressive numerical precision reduction (FP8, BlockFloat8, low-fidelity compute) has already cut LLM inference cost substantially, because token-level objectives tolerate small logit perturbations and autoregressive decoding resets error at each step. TTS models generate continuous waveforms instead of discrete tokens: small numerical perturbations can accumulate across denoising or generation steps and surface as audible artifacts, phase instability, or spectral distortion, so the same precision-reduction playbook cannot simply be transplanted from language models to speech synthesis. The paper asks whether a production-grade TTS system can aggressively reduce numerical precision and compute fidelity while preserving perceptual audio quality, and whether hardware-software co-design can fundamentally reduce inference cost.
Method
The paper studies Lightning V2, an existing production diffusion-based TTS system (acoustic model plus neural vocoder) built by Smallest AI, co-optimized for Tenstorrent’s Tensix core accelerators (P100/P150). It does not propose a new TTS architecture; the contribution is a precision-aware deployment methodology and a hardware-software co-design strategy layered onto that existing system.
On the numerics side, the authors apply two reduced-precision strategies selectively rather than uniformly: LoFi execution (reduced mantissa precision) and BlockFloat8 (BFP8, shared-exponent block floating point). Layer-by-layer tolerance to each format is determined empirically, guided by end-to-end perceptual evaluation, sensitivity to diffusion-step perturbations, and each layer’s dynamic range characteristics; layers with high dynamic range or diffusion-state sensitivity are kept at higher precision. This selection process surfaced a specific failure mode: Pearson Correlation Coefficient (PCC), commonly treated as a reliable proxy for numerical correctness, was found to be a poor predictor of perceptual audio quality (see Key Results).
On the hardware side, the authors exploit three characteristics of Tenstorrent’s dataflow architecture: a packet-based Network-on-Chip (NoC) that supports explicit multicast of frequently reused weights across cores (reducing redundant DRAM fetches), distributed on-chip SRAM with circular buffers that keep intermediate activations resident on-chip across denoising timesteps, and a deterministic, explicitly-orchestrated five-stage core pipeline (reader, unpacker, compute, writer stages) that contrasts with the implicit cache management of CUDA-style GPU execution. Custom kernels were written to preserve numerical stability under reduced precision while exploiting this explicit dataflow model. The result is a co-optimized deployment rather than a new model architecture: gains come from applying selective low-precision execution together with SRAM-aware tiling and NoC-based weight distribution to an already-trained diffusion TTS pipeline.
Key Results
Quality preservation: comparing Lightning V2 output on Tenstorrent P150 against an NVIDIA L40S baseline, DNSMOS dropped only from 3.872 to 3.801 (Δ = -0.071), and normalized WER between the two hardware outputs was 0.009, indicating semantic content is essentially preserved across the precision/hardware change (§4.2, Table 2).
Compute and memory: co-optimization yielded roughly 4x compute reduction in the diffusion acoustic model and 8x in the neural vocoder, alongside a 2x reduction in model size and 1.8x reduction in memory transfer volume (§4.5, §4.6).
Cost: at the single-device level, a Tenstorrent P150 (1,000) deliver 2.6x and 3.6x better cost-normalized throughput than an NVIDIA L40S (100K in NVIDIA L40S accelerators versus roughly 37K in Tenstorrent P100/P150 accelerators, a 3-4x reduction in upfront accelerator cost at equivalent throughput (§4.3, Figure 2). A single unoptimized production layer (~6B MACs) already ran about 2x faster on P150 than on L40S, and the authors project that extending kernel-level optimization to more layers could push the overall gain to 8-12x cost-normalized improvement (§4.4).
Metric mismatch case study: cross-device PCC between identical PyTorch weights run on GPU versus CPU was only ~0.72, yet the resulting audio was perceptually indistinguishable; conversely, one layer with PCC rounding to 1.0 produced audible degradation once executed at reduced precision, taking over a month to isolate (§3.4).
All comparisons are against a single GPU baseline (L40S) and a single proprietary production model; there is no comparison against alternative precision-reduction techniques on the same hardware, nor against other TTS systems.
Novelty Assessment
The contribution is an engineering and systems one, not an architectural one: Lightning V2’s diffusion-plus-vocoder design is treated as a given, and the paper’s value lies in showing how far existing precision-reduction techniques (LoFi, BFP8) can be pushed on a non-standard, dataflow-oriented accelerator through careful per-layer selection and explicit hardware co-design. The most transferable finding is arguably not the specific cost numbers but the empirical demonstration that tensor-level similarity metrics such as PCC can both over- and under-predict perceptual audio degradation, which is a genuine (if narrow) methodological/negative result for anyone validating reduced-precision TTS deployments. The headline 3-4x-8-12x cost figures are specific to one production model, one hardware vendor, and an extrapolated fleet workload rather than a controlled, reproducible benchmark.
Field Significance
moderate — this is a practical, industry-authored systems report rather than a modeling advance: it demonstrates that precision-aware, hardware-co-designed inference can substantially cut TTS serving cost on non-GPU accelerators without perceptible quality loss, and it surfaces a specific, transferable pitfall (PCC’s unreliability as a perceptual-quality proxy) that is relevant to anyone porting continuous-signal generative models to reduced precision. Its scope is narrow: results are tied to a single company’s production model, a single accelerator vendor, and cost projections built on a simplifying fleet-workload assumption.
Claims
- supports: Reduced-precision execution (low-bit mantissa and block-floating-point formats) can be applied to a diffusion-based TTS acoustic model and vocoder without perceptible loss in audio quality, provided precision reduction is selected per layer from empirical sensitivity analysis rather than applied uniformly.
Evidence: Over 95% of layers ran under LoFi and more than 80% under BlockFloat8, yielding ~2x model size reduction while DNSMOS moved only from 3.872 (NVIDIA baseline) to 3.801 (Tenstorrent) and normalized WER stayed at 0.009. (§3.5, §3.6, §4.2, Table 2)
- complicates: Standard tensor-level similarity metrics such as Pearson correlation are not reliable indicators of perceptual audio quality when validating reduced-precision execution of continuous-signal generative speech models.
Evidence: Cross-device PCC of ~0.72 (GPU vs. CPU, identical weights) corresponded to perceptually indistinguishable audio, while a separate layer with PCC rounding to 1.0 produced audible degradation once run at reduced fidelity, taking over a month to isolate. (§3.4)
- supports: Diffusion-based acoustic models accumulate rounding error differently from autoregressive, token-based language models, because their persistent latent state across denoising steps allows small per-step numerical perturbations to compound rather than reset at natural token boundaries.
Evidence: Layers with near-perfect intermediate correlation to higher-precision baselines still produced audible degradation once cumulative error across the multi-step denoising trajectory was accounted for. (§3.2)
- supports: Coordinating low-precision arithmetic with explicit hardware-level data movement optimizations (on-chip weight multicast, SRAM-resident tiling, avoided DRAM round-trips) can substantially lower the accelerator cost of serving real-time TTS at a given throughput, beyond what precision reduction alone would deliver.
Evidence: At a fleet workload sized to sustain 550 concurrent 5-second TTS requests, estimated accelerator cost was ~27K-$37K on Tenstorrent P100/P150, a 3-4x reduction attributed jointly to BFP8/LoFi execution and NoC/SRAM co-design. (§4.3, §5.4, Table 3, Figure 2)
Limitations and Open Questions
The headline fleet-level cost comparison (~ 27K-$37K for 550 concurrent requests) rests on a simplifying assumption (each response is treated as producing a fixed ~5 seconds of audio) and on a P100 latency figure inferred from measured P150 latency rather than independently measured, so it should be read as an extrapolated projection rather than a directly measured production benchmark (§4.3).
Certain layers still exhibit high numerical sensitivity and cannot yet run under LoFi or BlockFloat8 without perceptual degradation, limiting full-model low-precision coverage (§5.1). The authors also state that Tenstorrent compiler/kernel scheduling and memory tiling remain immature, meaning current results likely understate the hardware’s eventual performance ceiling (§5.1, §4.4). Quality evaluation relies on an automated MOS proxy (DNSMOS) and WER rather than human listening tests, and the study evaluates a single proprietary production model on a single accelerator vendor’s hardware, which limits how far the specific cost and quality figures generalize to other TTS architectures or accelerators. No code or trained model release is mentioned.
Wiki Connections
- Diffusion TTS — analyzes a production diffusion-based TTS acoustic model’s numerical fragility under reduced-precision execution, characterizing how rounding errors compound across the iterative denoising trajectory in a way that differs from token-based generation.
- Evaluation Metrics — reports a case study showing Pearson correlation, a common tensor-level similarity metric, can both over- and under-predict perceptual audio degradation when validating hardware/precision changes to a TTS system.