A Closer Look at Phonon TTS. Launching High-Quality On-Device Speech That Tops the Benchmarks.
TL;DR · AI 摘要
Phonon TTS是Hugging Face推出的设备端语音合成模型,无需网络即可生成高质量语音,支持五种语言并提供不同许可证版本。
核心要点
- Phonon TTS模型体积65MB/75MB,支持英法德西葡五种语言
- 采用流匹配技术实现小模型高质量语音生成
- Phonon 31m采用Apache 2.0许可证允许商业使用
结构提纲
按章节快速跳转。
- §引言
介绍Phonon TTS的发布背景及核心优势
分析传统云端TTS的局限性及Phonon的解决方案
- ›技术实现
基于Pocket TTS的流匹配技术实现高质量语音生成
- ·产品特性
介绍模型体积、语言支持及许可证方案
- ›应用场景
展示Apple Watch/iPhone等设备的实际应用案例
- ·未来方向
提及多语言扩展和自定义声音支持计划
思维导图
用一张图看清主题之间的关系。
查看大纲文本(无障碍 / 无 JS 友好)
- Phonon TTS
- 核心技术
- 流匹配技术
- Pocket TTS架构
- 产品特性
- 65MB/75MB模型
- 五种语言支持
- Apache 2.0许可证
- 应用场景
- Apple Watch
- 离线语音助手
- 机器人交互
金句 / Highlights
值得收藏与分享的关键句。
Phonon TTS模型体积仅为65MB/75MB,却能实现媲美云端系统的语音质量
采用流匹配技术使模型参数量减少80%仍保持高质量语音输出
Phonon 31m版本采用Apache 2.0许可证,商业使用无需额外付费
A Closer Look at Phonon TTS. Launching High-Quality On-Device Speech That Tops the Benchmarks.
Back to Articles
[-1
]
[0
Published November 5, 2026
Update on GitHub
Upvote
5
[
Olesia Krizhevich
olesiagradium
Follow
gradium
Laurent Mazare
lmz
Eric Bezzam
bezzam
Eustache Le Bihan
eustlb
Two weeks ago, we opened up Phonon, our small text-to-speech model that runs on your own device. It needs no server and no network, and what you type stays with you.
We've been very happy with the reception so far. So here's the story of the launch: why we built Phonon, what it can do, the demos that came with it, and how it did on Hugging Face's Open TTS Leaderboard.
Why a voice should live on your device
Most voice AI lives in the cloud: every sentence makes a round trip to a server. Phonon runs on your own device instead. Your text stays there and the voice works without a network. We’ve also opened the weights. Phonon 31m is Apache 2.0, so you can use it for free, even in a commercial product. The other models are CC BY-NC, so they’re free for non-commercial projects. If you’re building something commercial with those, pricing is based on your number of users, not how much speech you generate.
Getting a good voice to run on a small device has always been hard. You could make the model small, but then it didn’t sound great. Or you could make it sound great, but then it wouldn’t fit. We addressed both.
We announced Phonon in April and spent the next six months building on it with partners and early users. Their feedback helped us to shape this release. The models got smaller, and we added more languages and support for custom voices through voice cloning and voice design.
Phonon is based on the Pocket TTS lineage and as such uses flow matching to generate the audio which is very different from the modeling techniques used in other TTS systems. This is what allowed us to bring this level of quality to such a small model but it created new challenges while training the models. The training was initially numerically unstable and required several tricks to produce quality results on all five languages.
What we released
Phonon comes in two sizes, at about 65 MB and 75 MB. Together they speak English, French, German, Spanish and Portuguese. There are more than 100 built-in voices, and you can create your own in Gradium Studio . Phonon also handles the important things that most other TTS systems often stumble on, like phone numbers, dates and codes.
It runs on phones, laptops, single-board computers and in a browser tab, so you can hear it right now . Phonon 31m speaks English and is Apache 2.0, so you can build on it freely, including for commercial products. Phonon 40m covers all five languages under a non-commercial license.
Demos and first projects
The launch came with demos from friends and from our own team. On an Apple Watch with Wi-Fi off, Phonon talks you through a coffee brew step by step. On an iPhone with no signal, it counts your reps and keeps you going. A robot called Lamp, from Autonomous, wears a sorting hat and talks in character. In a browser, an offline dino runner becomes a radio show hosted by Rex. There's also a sci-fi game voiced on the player's own machine, and Proust read aloud page by page.
The Hugging Face team built the Transformers integration, so developers could use Phonon in Transformers from day one.
Developers have started building on it too. An ExecuTorch conversion runs Phonon on a visitor's GPU in the browser, Micdrop brings it to Node.js apps, and Spaces like smallest-s2s build on it.
Phonon was downloaded more than 200,000 times in its first two weeks, and it’s still trending on Hugging Face.
How Phonon did on the Hugging Face Open TTS Leaderboard
The Open TTS Leaderboard is a very new project from Hugging Face, and we love it. With more than 9,000 TTS models on the Hugging Face Hub , the leaderboard gives developers one place to compare open models and hear them side by side. Across 23 languages, models are evaluated on intelligibility ( word error rate of the outputs ), speed ( inverse real-time factor and time-to-first audio ), and naturalness ( UTMOS for English and KOMOS for Korean ). Fixed-voice and voice cloning are evaluated, with an additional speaker similarity metric for the latter. The leaderboard also has a Listen tab, where you can conveniently compare TTS outputs yourself and vote for the ones you like the most.
On intelligibility, Phonon ranks first in German and Spanish , and in the top three in every language we evaluated, next to models up to 100 times its size.
Speed is where we see Phonon truly shine, as the models top the CPU streaming benchmark on time-to-first audio.
Inference
We built a new inference framework, ptts , specifically for Phonon. The goal of ptts is to get the best performance on a wide variety of devices and as such supports more than 10 backends: CPU (Intel, Arm, WebAssembly), GPU (CUDA, Vulkan, Metal, WebGPU), and NPU (Apple Core ML, Snapdragon®).
Phonon is also available in 🤗 Transformers, from v5.21, as its reference PyTorch implementation. Where ptts runs a single stream on device, Transformers targets GPU throughput: batched generation, full-graph torch.compile , e.g., for synthesizing speech datasets at scale. Being native PyTorch, it is a base to build upon, whether to integrate Phonon into your own stack or to study the model itself.
It belongs to the Pocket TTS lineage, so we first added Pocket TTS to Transformers. Phonon sits on top as a modular model: about 200 lines describing only what differs.
Phonon generates audio latents autoregressively: each step outputs a latent that is fed back as the next input, and the training loss is computed on them. Latents are the model's inputs, outputs and targets, as text tokens are for a language model, so the codec mapping audio to latents plays the role of a tokenizer. Transformers ships it as such: an audio tokenizer , held by the processor next to the text tokenizer:
from
transformers
import
AutoProcessor, GradiumPhononTTSForConditionalGeneration
model_id =
"gradium/phonon-31m-en"
processor = AutoProcessor.from_pretrained(model_id, device_map=
"auto"
)
model = GradiumPhononTTSForConditionalGeneration.from_pretrained(model_id, device_map=
"auto"
)
print
(
f"
{model.num_parameters():,}
"
)
# 30,788,354
print
(
f"
{processor.audio_tokenizer.num_parameters():,}
"
)
# 10,305,313A voice is not a recording but a tensor of speaker latents. Phonon never encodes audio directly with its audio tokenizer, which ships decoder-only, and voices come precomputed: gradium/phonon-voices .
from
huggingface_hub
import
hf_hub_download
from
safetensors.torch
import
load_file
path = hf_hub_download(
"gradium/phonon-voices"
,
"voices/LMDIYE0Oa377Qh6Z.safetensors"
)
voice = load_file(path)[
"speaker_wavs"
][
0
].T
# (num_frames, 512)The prompt layout (voice positions, text, then a token opening the audio) ships with the checkpoint as a chat template, as for other multimodal models in Transformers. You write a standard conversation, with an empty audio item where the voice goes, and your code stays the same across checkpoints. The processor then expands the voice placeholder to one position per frame, normalizes and tokenizes the text, and batches the inputs:
conversation = [
{
"role"
:
"user"
,
"content"
: [{
"type"
:
"audio"
}, {
"type"
:
"text"
,
"text"
:
"Hello from Phonon."
}]}
]
inputs = processor.apply_chat_template(
conversation,
tokenize=
True
,
return_dict=
True
,
add_generation_prompt=
True
,
processor_kwargs={
"audio_prompt_latents"
: [voice]},
).to(model.device)
# prompt: <|audio_prompt|> × num_frames, "Hello from Phonon.", <|audio_bos|>generate returns latents, which the processor decodes to audio:
outputs = model.generate(**inputs)
# audio_latents: (batch_size, num_frames, 32)
audio = processor.decode(outputs.audio_latents, outputs.audio_latents_mask)
processor.save_audio(audio,
"phonon.wav"
)See the docs 🤗 for more!
What's next
Phonon can speak with any voice. Clone one from a short recording with the speaker’s consent, or describe a new one in plain text in Gradium Studio . Then download the voice file, load it in Phonon, and it runs fully on the device.
We also train custom Phonon models with design partners. If your memory budget is tight, we can make the model smaller. If you need Korean, Arabic or Japanese, we can train for that too. And if your product has its own vocabulary, we can fine-tune on it. Get in touch and tell us what you need.
Thank you to the Hugging Face team, and to everyone who tried Phonon in its first two weeks.
Phonon is our first open release. Try it out and tell us what you build.
Models mentioned in this article 3
Datasets mentioned in this article 1
Spaces mentioned in this article 3
More Articles from our Blog
audio
speech
leaderboard
Open TTS Leaderboard: Scalable Evaluation for Multilingual Text-to-Speech and Voice Cloning
79
September 30, 2026
The Open ASR Leaderboard Adds Its First Global South Language
- +6
61
August 28, 2026
Community
Edit
Preview
Upload images, audio, and videos by dragging in the text input, pasting, or
clicking here
.
Tap or paste here to upload images
Comment
· Sign up or log in to comment