mlboydaisuke/VibeVoice-Realtime-0.5B-CoreAI

VibeVoice-Realtime-0.5B — Core AI

microsoft/VibeVoice-Realtime-0.5B (MIT) converted to Apple Core AI — the zoo's first multi-speaker / dialogue (podcast-style) TTS. iPhone (AOT) + Mac, all-fp16.

Not a "first on-device VibeVoice" claim — other CoreML/GGUF ports exist. What this is: the zoo's first multi-speaker TTS, app-integrated, and the other half of a generate -> diarize loop with the zoo's Streaming Sortformer diarizer.

Architecture

Dual Qwen2.5 LM (4-layer text context LM, norm = Identity + 20-layer speech trunk) -> per-frame next-token diffusion (4-layer adaLN head, DDPM cosine, v-prediction, DPMSolver++ 5-step, CFG 1.5) -> causal-conv acoustic VAE decoder (7.5 Hz latent -> 24 kHz, 3200 samples/frame). The LM predicts one latent per frame; the diffusion head denoises it; the VAE decoder renders audio. Multi-speaker output is host turn-switching: each Speaker N: turn is generated from its own voice preset and the turns are concatenated — no multi-speaker prefill, no acoustic encoder.

Contents

path	what
macos/vibevoice_mainlm_fp16_decode_cl512.aimodel	context LM, q=1 decode, KV-stateful, cache 512 (114 MB)
macos/vibevoice_ttslm_fp16_decode_cl512.aimodel	speech trunk, same shape; also drives the CFG-negative stream (569 MB)
macos/vibevoice_diffusion_head_fp16.aimodel	prediction head, (noisy[2,64], t[2]) -> [2,64] (80 MB)
macos/vibevoice_connector_fp16.aimodel	acoustic connector, latent[1,1,64] -> embed[1,1,896] (1.7 MB)
macos/vibevoice_decoder_fp16_t64.aimodel	acoustic VAE decoder, latents[1,64,64] -> audio[1,1,204800] (656 MB)
ios/*.h18p.aimodelc	the same five, AOT-compiled for A19 (h18p), GPU
voices/*.pt	25 upstream voice presets (EN/ZH/…): pre-computed prefill KV, so no acoustic encoder is shipped
coreai_host/voices/<name>/	the same 25 presets as flat fp16 blobs a Swift host reads directly (no torch)
coreai_host/glue/	type embeddings, EOS classifier, DPMSolver++ schedule, Qwen2.5 tokenizer
coreai_host/embed/embed_tokens_fp16.bin	(151936, 896) fp16 embedding table, mmapped for the host token lookup
device_bundle/	compact host inputs + golden.f32 for the on-device self-test

fp16 is required. int8 LMs diverge inside the speech feedback loop (min cos 0.187, early EOS); the diffusion head is fp16-sensitive too (pure-torch fp16 collapses to 0.79 — Core AI keeps the RMSNorm/adaLN reductions in fp32, so the host DDPM reference must run fp32).

Fixed shapes only. Every graph is static (q=1 decode, fixed-T decoder), so the runtime must not be given the expectFrequentReshapes hint on iOS: it makes the runtime skip the AOT specialization and compile on device, which segfaults inside the MPSGraph AICode compiler.

Gates

gate	result
diffusion head / connector, engine vs oracle	cos 0.999999 / 1.000000
acoustic decoder (T=30 / T=64), engine vs non-stream golden	cos 1.000004 / 1.000005
main LM / tts LM decode, engine vs torch	cos 0.999999 / 0.999996
Python E2E on all 5 engines vs the upstream streamed wav	latent min cos 0.999198, wav cos 0.999479
iPhone 17 Pro (A19 Pro, AOT h18p, GPU) vs the golden	cos 0.998308

On device: 6 graph loads in 2.6 s (warm), 24 latents / 3.20 s of audio in 2.3 s = 10.6 tok/s ~ 1.4x real-time.

Use it

import CoreAIKit

let dialogue = try await KitDialogue(catalog: "vibevoice-realtime-0.5b")
let (audio, turns) = try await dialogue.perform("""
    Speaker 1: Did you know this runs entirely on the phone?
    Speaker 2: No cloud at all? That is wild.
    """)

Swift host reference: ondevice/VibeVoiceRunner (Mac) and VibeVoiceSelfTest.swift in the zoo's coreai-audio app — raw Core AI stateful-KV loop + a Swift DPMSolver++ sampler. Python host + conversion recipe: conversion/vibevoice (host_e2e.py = the full generate loop, host_multispeaker.py = the dialogue demo).

Base model: microsoft/VibeVoice-Realtime-0.5B (MIT). EN/ZH. Community port — not an Apple model.