ZONOS3-Mini: Open-Source Real-Time Voice Cloning & Steerable TTS in 100M Active Parameters

By Gabriel Clark, Sofian Mejjoute, George Close, Beren Millidge · November 11, 2026

🤗 Model Weights · 🚀 Hugging Face Space · 🤗 Transformers Repo · 💻 GitHub (MIT) · 🌐 WebGPU Playground · 📊 ZTTS2-Eval · 📝 Original Post

Introducing ZONOS3-Mini, an Apache 2.0 zero-shot voice-cloning Text-To-Speech (TTS) model combining a 15 Hz dual audio codec (ZAC-1), a hybrid Gated DeltaNet / Attention Latent MoE backbone (100M active / 800M total parameters), mechanistic activation steering, and real-time on-device execution across browsers, phones, laptops, and servers.

Introduction & Open-Source Release

We are excited to release ZONOS3-Mini, our next-generation open-source real-time Text-To-Speech (TTS) model for multilingual speech synthesis, zero-shot voice cloning, and fine-grained vocal control across 43 languages. Trained in just 2.5 days on 112 NVIDIA H100 GPUs, ZONOS3-Mini uses a sparse Mixture-of-Experts (MoE) architecture with 100 million active parameters (800 million total) to deliver frontier speech quality and controllability at edge-friendly compute budgets.

With ZONOS2, we showed that sparse MoE architectures can mitigate the classic tradeoff between synthesis quality and inference latency in TTS. However, ZONOS2 relied on an 8B-parameter backbone modeling an 86 Hz audio codec, which constrained local and client-side deployment. ZONOS3-Mini removes those bottlenecks through co-designed efficiency gains across the audio codec, sequence mixer, speaker representation, and post-training pipeline—achieving <570 ms time-to-first-audio (TTFA) in-browser on a Pixel 9, <150 ms on an M3 MacBook Air, and 80 ms median TTFA (457× real-time peak throughput) on a single H100.

Everything needed to test, build on, and deploy ZONOS3-Mini is open-sourced today:

• Apache 2.0 Model Weights on Hugging Face (Zyphra/ZONOS3-Mini & Zyphra/ZONOS3-Mini-hf): Full bf16 PyTorch checkpoints, quantized gguf/ weights (including speech-editing checkpoints).
• Interactive Hugging Face Space (spaces/Zyphra/ZONOS3-Mini): Try zero-shot voice cloning and multilingual synthesis directly in your browser on Hugging Face Spaces, or run 100% client-side on your own GPU via our WebGPU Voice Studio.
• MIT-Licensed Runtimes & Reference Code (github.com/Zyphra/ZONOS3-Mini): Day-0 upstream serving via SGLang-Omni, native Apple Silicon (MLX), standalone C++/ggml (CPU, CUDA, Metal, Vulkan), in-browser WebGPU, and the Zonos Voice Studio UI.
• Open Evaluation Benchmark (ZTTS2-Eval): Expanded multilingual evaluation covering 35 clean and 20 in-the-wild language subsets with an Item Response Theory (IRT) capability index over 420 items.

---

Evaluation & Quantitative Benchmarks (ZTTS2-Eval)

Following our earlier release of ZTTS1-Eval, we are releasing Zyphra TTS Evaluation v2 (ZTTS2-Eval). ZTTS2-Eval expands clean read-speech evaluation (FLEURS-R) to 35 languages and in-the-wild speaker-cloning evaluation (VoxBlink2) to 20 languages, with higher sample counts per language. To avoid single-evaluator bias, Qwen3-ASR is supplemented with Whisper Large v3 in lower-resource languages as well as English (where Qwen3-ASR otherwise advantages Qwen TTS models). Furthermore, the TTSDS2 prosody metrics from ZTTS1-Eval are superseded by dedicated rhythm and pitch distribution matching metrics alongside an adjusted Masked Prosody Model embedding distance with a scrambled-input floor that expands the dynamic range spanned by models.

To aggregate performance rigorously across 55 language subsets, acoustic conditions, and metrics, we introduce the Zyphra TTS Capability Index v1, modeled on the Epoch Capability Index using Item Response Theory (IRT) over 420 items (metric × test set × language), each fitted with its own difficulty and discrimination slope. ZONOS3-Mini ranks first among open-weight models, ahead of Fish S2 Pro and MOSS TTS v1.5.

Pareto frontiers for ZONOS3-Mini against comparable open-source TTS models on a single H100 80GB GPU. Quality combines macro speaker similarity and macro WER across all 55 language subsets of ZTTS2-Eval as ½ z(similarity) − ½ z(WER). Left: median time-to-first-audio at concurrency 8 (80 ms for ZONOS3-Mini). Right: peak throughput at best concurrency (457 s of audio/s for ZONOS3-Mini).

---

Qualitative Comparisons

Listening directly to generated samples remains essential for assessing prosody, naturalness, and speaker fidelity. Below we compare ZONOS3-Mini outputs side-by-side against leading open-weight and proprietary TTS systems (you can also test your own prompts interactively in the ZONOS3-Mini Hugging Face Space).

Trailer Voice

“I must not overfit. Overfitting is the model-killer. Overfitting is the little loss that brings total generalization failure. I will face my loss. I will permit the gradient to pass backward over me and through me. And when it has propagated, I will turn the validation eye to see its path. Where the loss has gone there will be convergence. Only the learned representation will remain.”

Parks and Recreation Guy

“You leak your test set? Straight to jail. No validation, no nothing. Data scientists, we have a special jail for data scientists. You overfit? Jail! You underfit? Also jail! Overfit, underfit. You normalize the training data but not the test data? Jail, right away. You tune hyperparameters on the leaderboard? Believe it or not, jail! You call it AI when it’s just logistic regression? Jail! You use a neural net for a problem solved by a decision tree? Jail! You forget to set a random seed? Jail! You set the seed but don’t log the experiment? Also jail! You say 'the model is 99% accurate' on an imbalanced dataset? Right to jail! You deploy without monitoring drift? Jail, immediately.”

Dwarkesh

“I guess what’s interesting is that it no longer has that synthetic, slop-y cadence you expect from these models. It actually feels like you're talking to a person, right? And maybe the M.O.E thing is actually doing a lot of work here. You get the feel of a much bigger model but with great real-time performance at the same time.”

---

Research: Architecture, Codec & Training Details

ZONOS3-Mini introduces four core improvements across audio representation, sequence modeling, speaker conditioning, and post-training.

1. Audio Representation: Zyphra Audio Codec 1 (ZAC-1)

Descript Audio Codec (DAC), while capable of high reconstruction fidelity, is difficult for autoregressive backbones to model and bottlenecks real-time generation due to its high 86 Hz frame rate. To address both issues, we trained and open-sourced Zyphra Audio Codec 1 (ZAC-1) (HF weights · GitHub): a 15 Hz, 44.1 kHz dual-encoding neural codec (10 × 4,096 codebooks at hop size 2,940 samples: codebook 0 quantizes layer-16 features of a frozen Wav2Vec2-BERT 2.0 teacher, while codebooks 1–9 form an acoustic residual RVQ). Causal transformer stages upsample the 15 Hz codes to 240 Hz features before a multi-resolution iSTFT head synthesizes 44.1 kHz waveforms (full_mr full-size or keep2_plain pruned for ~3× faster decode).

Under matched data constraints, a TTS model trained on ZAC-1 achieves a 10× reduction in Word Error Rate (WER) compared to one trained on DAC while simultaneously improving acoustic similarity. Against modern TTS codecs such as S1-DAC and MOSS v2, ZAC-1 matches quality at higher compression or outperforms at comparable bitrates by exposing a far more modelable latent structure to the autoregressive backbone.

Zyphra Audio Codec 1 (ZAC-1): 15 Hz dual semantic-acoustic tokenization with causal multi-resolution iSTFT decoding to 44.1 kHz audio.

2. Hybrid Gated DeltaNet + Latent MoE Backbone

We bring recent LLM architectural advances directly into speech generation:

• 3:1 Hybrid Sequence Mixer (gdn3_global1): Across 11 layers (d_model = 1,024), ZONOS3-Mini interleaves 9 Gated DeltaNet (GDN) recurrent layers with 2 global GQA attention layers (16 query heads × 32 dims, 4 KV heads) over a 32,768-position context window (up to 10 minutes of continuous audio), improving both quality and long-context throughput.
• Latent MoE, Q-Balancing & Attention Residuals: Every layer uses a sparse Latent MoE (64 experts, top-2 routing from layer 1) alongside Q-balancing and Full Attention Residuals (full_attnres), yielding substantial gains in parameter efficiency at the 100M-active / 800M-total scale.
• Text & Acoustic Conditioning: Input text is normalized with NeMo grammars and tokenized via an 8,192-entry SuperBPE vocabulary, conditioned alongside bucketed loudness (LUFS), bandwidth (f_max), and speaking rate (UTF-8 bytes/s). Under a delay pattern across the 10 codebooks, each completed 15 Hz frame streams immediately into the causal iSTFT head.

Autoregressive streaming pipeline: speaker conditioning, bucketed acoustic controls, and SuperBPE text tokens feed the hybrid GDN/attention Latent MoE backbone, predicting 10 delayed codebooks per 15 Hz frame and decoding immediately to 44.1 kHz audio.

3. Dual-Mode Speaker Representation (MECT-B2 & In-Context Prefix)

We replace the Qwen3-TTS speaker encoder used previously with the MECT-B2 speaker verification model (192-d embedding), improving zero-shot speaker similarity while making linear manipulation in speaker space substantially cleaner. ZONOS3-Mini supports two complementary conditioning modes:

1. In-context reference mode (trained throughout pretraining with a separate reference clip encoded into ZAC-1 tokens in context, up to 60 s), producing highly natural prosody without rigidly copying reference cadence.
2. Direct speaker-embedding mode (finetuned in a short post-pretraining stage on the 192-d MECT-B2 representation), enabling zero-shot cloning, compact voice presets, and continuous identity steering.

4. Preference & RL Post-Training

Following pretraining and speaker-mode finetuning, we apply a two-stage post-training pipeline. First Direct Preference Optimization (DPO) and then Group Relative Policy Optimization (GRPO) to reduce hallucinations, improve pronunciation stability, and harden multi-lingual robustness.

---

Mechanistic Controllability & Steering Demos

Beyond zero-shot intelligibility and speaker similarity, ZONOS3-Mini serves as an open testbed for mechanistic voice and delivery control without retraining:

• Speaker Identity & Accent Vectors: Age, gender presentation, and vocal texture (softness, raspiness, brightness, depth) are steered directly via directions in MECT-B2 speaker-embedding space, while accents use dedicated task vectors and adapters.
• Activation Steering for Delivery: Pace, emotion, projection, and liveliness are controlled at inference time via Activation Steering inside the residual stream.
• Word-Level Streaming Control via Alignment Heads: By monitoring learned cross-lingual alignment attention heads online during autoregressive generation, activation steering vectors can be applied selectively to targeted word spans in real time during streaming.
• Post-Generation Speech Editing: We also release a VoiceCraft-style speech-editing recipe (available in GGUF as zonos3-ptq-edit.gguf, paired with our open-source Zyphra/fusealign word aligner), enabling seamless word/phrase replacement with full delivery control.

Voice Design Examples

Each A/B/C comparison below uses identical text and the same narrator reference voice in embedding mode with neutral delivery. Only the listed design controls change.

1. Same speaker, three accents

“I parked the car near the water, just past the coffee shop. I’ll meet you there after half past four.”

Listen for changes in pronunciation while the voice remains recognizable as the same speaker. England steering is used as an approximation of the Southern English target.

2. Same voice at different ages

“I found the old camera in a box upstairs. There’s still one photograph left, so let’s make it a good one.”

Listen for a progression in perceived age through vocal texture and resonance. The ages are targets, not measured ages.

3. Gender presentation

“The next train leaves in ten minutes. We have just enough time to grab a coffee and find our seats.”

Listen for changes in resonance, texture and pitch, while the base speaker, accent and delivery remain fixed. The intermediate gender setting targets an androgynous presentation.

4. Voice texture

“The wooden box is on the top shelf, beside the green bottle and the little brass clock.”

Listen for moderate texture changes at ordinary speaking volume. Soft texture approximates breathiness; there is no dedicated breathiness control.

5. Compose a cast from multiple controls

“The map ends here. Beyond that ridge, we’re going to have to find our own way.”

Listen for whether age, gender presentation, accent and texture come through together as three distinct casting choices.

Delivery Steering Examples

These 15 samples keep the narrator reference speaker and accent fixed. Whole-line delivery is steered via activation steering unless a specific word span is targeted via attention alignment heads.

6. Whispering on a span

“I didn’t say she took the keys.”

B targets “I didn’t say” and C targets “she took the keys” with local whisper steering (+4). The neutral take uses the same sentence and seed.

7. Same words, three emotional intentions

“You came back. I wasn’t sure you would. I’ve been waiting here all evening.”

Listen for different interpretations of the same situation. These are emotional targets; the controls do not guarantee a specific emotion or fully independent changes in pace and projection.

8. Pace without losing clarity

“The next train leaves from platform four at seven thirty. Please have your ticket ready before boarding.”

Listen for a clear speed range with every word preserved, particularly the platform number and departure time. Pace steering can also affect loudness and pitch.

9. Projection: nearby to across the room

“Can you hear me? I’m right here, beside the gate. Follow my voice, and I’ll wait for you.”

Listen for changes in breath, resonance and vocal effort as well as loudness. The emotional target remains calm across the set.

10. Liveliness: restrained to animated

“Inside the box were a silver key, a folded map, and a tiny doorbell. Naturally, I pressed it.”

Listen for increasing pitch variation, rhythmic interest and expressive phrasing. Liveliness steering can also affect pitch level and duration.

---

Open-Source Inference, Hugging Face Space & Quick Start

ZONOS3-Mini is designed to run across the full spectrum of open-source hardware and runtimes, from multi-GPU servers down to laptops and smartphones:

1. Try on Hugging Face Spaces (Zero Install): Experiment with zero-shot voice cloning and multilingual generation directly on the HuggingFace via the space Zyphra/ZONOS3-Mini Space, or run entirely client-side on in the browser via WebGPU in the Zonos Voice Studio.
2. Serve via SGLang-Omni: Serve Zyphra/ZONOS3-Mini behind an OpenAI-compatible /v1/audio/speech streaming endpoint with zero-shot reference cloning:

sgl-omni serve --model-path Zyphra/ZONOS3-Mini --port 8000

curl -X POST http://localhost:8000/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{"input": "Hello from ZONOS3-Mini on Hugging Face!"}' \
  --output output.wav

3. Local Native Runtimes (MLX, C++/ggml, CUDA) & Plain-PyTorch Reference: Our MIT-licensed GitHub repository provides a unified HTTP server and Voice Studio across CUDA (Mini-SGLang), Apple Silicon (MLX), standalone C++/ggml (CPU, CUDA, Metal, Vulkan—loading quantized checkpoints directly from Zyphra/ZONOS3-Mini/tree/main/gguf), and WebGPU, alongside a readable plain-PyTorch reference implementation (zonos3.reference) for research hacking:

# Download quantized GGUF weights from Hugging Face for MLX or C++/ggml
hf download Zyphra/ZONOS3-Mini --include 'gguf/*' --local-dir servers/ggml

# Or run the plain-PyTorch reference implementation directly against Hugging Face Hub
uv run python -m zonos3.reference \
  --model hf://Zyphra/ZONOS3-Mini/lm/zonos3_mini \
  --text "Hello from the ZONOS3-Mini PyTorch reference." \
  -o ref.wav --device cuda

Supported Languages (43)

Citation

If you build on ZONOS3-Mini, ZAC-1, or ZTTS2-Eval in your research, please cite:

@misc{zyphra2026zonos3mini,
  title        = {{ZONOS3-Mini Technical Report}},
  author       = {Gabriel Clark and Sofian Mejjoute and George Close and Beren Millidge},
  year         = {2026},
  month        = nov,
  howpublished = {\url{https://www.zyphra.com/our-work/zonos3-mini}},
}