Tencent AuK: An Open-Source Voice Model Trained on 1.95 Million Hours (2609.08936)

The AuK Technical Report (arXiv 2609.08936, submitted 8 September 2026, first author Ziyang Ma) describes a 1.5B-parameter open model that generates, edits, enhances and separates speech from plain-language instructions. The authors report about 3.03 billion instruction-audio instances and 1.95 million hours of effective supervision. Their headline result is an average Seed-TTS-Eval error of 2.65% against 3.07% for Qwen3-TTS. The code sits in Tencent's Hunyuan GitHub organization under an MIT License.

Every number below is the authors' own, measured on baselines they chose. No independent replication has been published as of 15 October 2026.

We read: the full paper (arXiv HTML version), plus the GitHub README. No key claim here rests on the abstract alone.

What problem does the AuK paper solve?

It targets the fragmentation of speech tooling: separate models for voice cloning, style-controlled TTS, word-level edits, emotion changes, denoising and speaker separation. The authors argue that separate task-specific models fragment the user experience and duplicate modeling effort.

Unifying them is hard for three reasons the paper lists. Output constraints differ (generation makes new speech, editing must preserve the rest). The conditioning interface differs (text only versus text plus audio). Supervision and evaluation are heterogeneous, because open-ended edits depend on subjective judgments that no single automatic metric captures.

AuK's answer is one interface: a natural-language instruction plus optional audio goes in, a waveform comes out. The five task families are speech generation, content editing, paralinguistic editing, acoustic editing, and enhancement and separation.

How does AuK work?

AuK combines a frozen multimodal language model for meaning, a frozen audio VAE for acoustics, and a trained rectified-flow transformer that predicts the output latent. Only the transformer and a layer-fusion module are updated during training.

Architecture

• Semantic encoder: Qwen2.5-Omni reads the instruction and any reference audio. A learnable weighted sum of its layer-wise hidden states forms the condition, on the argument that different depths carry different linguistic and acoustic cues.
• Audio VAE: it maps 24 kHz mono audio to 64-dimensional latents at 50 Hz. The authors trained it on roughly 3 million hours of speech, music and general audio, sampled 6:3:1.
• Backbone: 30 layers, made of 10 dual-stream MMDiT blocks followed by 20 single-stream DiT blocks. Hidden size is 1536 with 24 heads, which gives about 1.5 billion parameters. The design follows the FLUX-style hybrid.

The 1.5B figure counts the backbone only. According to the GitHub README, the Qwen2.5-Omni-3B encoder is a separate download, and the VAE loads from its own file. Running AuK means loading all three.

Training recipe

1. Warm-up: 50k updates on generation tasks only, to stabilise text-to-speech alignment.
2. Joint pre-training: 600k updates on generation and editing together, on 256 GPUs. Each optimizer step covers about 14 hours of audio. Sampling probabilities per batch are 28.10% generation, 23.17% enhancement and separation, 23.02% content editing, 21.75% paralinguistic editing and 3.96% acoustic editing.
3. Editing preference optimization: annotators rated candidate edits as failure, partial or success. After filtering, the dataset held 818 informative groups and 9,080 rated candidates, used with a flow-based DPO objective.
4. Generation reinforcement learning: Flow-GRPO with automatic rewards for content correctness (ASR), speaker similarity and style consistency. The pool is 10,000 zero-shot prompts and 5,000 instruction prompts. For background on reward-driven training, see our DeepSeek-R1 explainer and the InstructGPT and RLHF explainer.
5. Distillation: consistency initialization, then task-routed Decoupled DMD. The result is AuK-Flash, which runs 4 steps without classifier-free guidance.

One data idea is worth noting. For zero-shot TTS the authors pair two different utterances from the same speaker, one as the acoustic prompt and one as the target. A speaker with n utterances gives n×(n−1) training instances, and the prompt transcript is never supplied. The authors say this improves expressiveness and speaker similarity, but the report does not show an ablation table for it.

This is also why "1.95 million hours" needs care. The paper calls it effective supervision, and a single recording can appear in many pairs. The report does not give a unique-hours figure.

What are the results?

AuK leads most of the tables the authors chose to show, but not all of them. The table below uses the paper's Table 3 and appendix tables, with the strongest listed prior system per row.

All figures are lab-reported. The Seed-TTS-Eval average mixes English word error and Chinese character error, and the authors ran each evaluation three times and report the mean. The paper does not report variance across those runs in the summary tables.

On SpeechEditBench, AuK reports joint success of 91.83% on content editing against 76.46% for Ming-UniAudio, and 71.33% on prosody against 26.50%. The authors also report that AuK-VAE beats four other audio VAEs on PESQ, STOI and two spectral distances across speech, general audio and music.

On the speed claim, the authors report that AuK-Flash gives a 4.5× wall-clock speedup over the 32-step teacher under matched conditions. Third-party measurements are not yet available.

Does the evidence hold up?

Partly. The gains over Ming-UniAudio on semantic editing are large, but several details soften the "leading performance" headline.

Where the headline overreaches or needs context

• Not a clean sweep on generation. On Seed-TTS-Eval test-zh, the paper's own Table 5 lists VoxCPM 2 at 0.97 against AuK at 1.02. On InstructTTSEval, AuK scores 75.57 on English APS against 82.90 for Qwen3-TTS-VD and 83.00 for Hume AI, and 68.23 on Chinese role-play against 74.00 for MOSS-VoiceGenerator.
• Baseline selection. Ming-UniAudio is compared on editing but absent from AuK's zero-shot TTS table. Its own abstract reports a Chinese Seed-TTS-WER of 0.95, against AuK's 1.02 on test-zh. The two numbers may not come from identical settings, so this is a gap in the comparison, not proof of a loss.
• Absolute edit success is low. On MMAE-Speech, the share of samples that satisfy every instruction and consistency rubric is 12.44% for AuK and 13.85% for AuK-Flash. SpeechEditBench emotion editing is 9.94% for AuK. Relative gains are real, but most whole-sample edits still miss at least one rubric.
• Restoration margins are thin. The authors themselves describe results on enhancement and separation as "competitive", not leading. On Libri2Mix, AuK's 9.12% WER sits next to 9.34% for MossFormer2-SS.
• No unified-versus-specialist ablation. The report does not show that one unified model beats specialist models trained on the same data, which is the central premise.
• Emergent behaviours are anecdotal. The authors observe whispered speech from text and English de-accenting despite Chinese-only de-accent supervision. They state these observations are not substitutes for comprehensive benchmark evaluation.
• Data provenance is vague. The bulk corpus is described as large-scale and bilingual. Some data are in-house, including a de-accent corpus covering 13 Chinese dialect and accent categories. The paper names no dataset licences for the main corpus.
• Misuse is not discussed. The report text has no section on consent, watermarking or misuse of voice cloning and editing.

Reproducibility. The authors released source code and weights. The GitHub README lists command-line inference, a Gradio demo, ComfyUI nodes, a Prompt Enhancer, a Python API and fine-tuning instructions. That is a stronger reproducibility position than most lab reports, though the training corpus is not released.

How does AuK compare with related work?

AuK sits between open speech-editing models and strong TTS systems. The closest comparators are named in its own tables.

A real difference is scope. Ming-UniAudio and Step-Audio-EditX focus on editing, and Qwen3-TTS on synthesis. AuK also takes on enhancement and separation, which makes its "one model" claim broader, and its restoration results thinner.

Architecturally, AuK follows the diffusion-transformer line rather than the autoregressive token approach. For the transformer background, see our Attention Is All You Need explainer, and for the denoising background, the diffusion models explainer.

What does a working researcher take from this?

Four practical points stand out.

1. Treat the Prompt Enhancer as part of the system. The authors say reaching the model's ceiling still requires the Prompt Enhancer to identify the task, normalise parameters and rewrite the request. An agent-built set of free-form editing instructions did not generalise well to arbitrary requests. Free-form control is a pipeline, not a model property.
2. Cross-utterance prompting is cheap to copy. Pairing different utterances from one speaker, with no prompt transcript, is a data recipe you can test in your own TTS setup.
3. Routing in distillation matters. Uniform Decoupled DMD degraded multi-speaker and vocal separation in the authors' experiments, so they routed separation examples to a supervised objective. If you distil a multi-task generator, check each task separately.
4. Check the licence chain. The repo says MIT, but the encoder is downloaded separately. A commercial deployment needs the terms of every component, plus a policy for cloned voices.

For evaluation work, note that edit success is still low in absolute terms, so a benchmark that reports only averaged instruction following can hide how often an edit fully succeeds.

What's still unclear?

Several open questions remain, and a few source conflicts.

• Independent results. No third-party benchmark of AuK or AuK-Flash has been published as of 15 October 2026. The authors' choice of baselines is the main caveat on the "leading" claims.
• Unique audio hours. The 1.95 million figure is effective supervision. The Tencent repository describes the data only as millions of hours of diverse audio.
• Release date. Sources differ slightly. The Hugging Face card, as described by OrcaRouter, dates the open-source news 7 September 2026. The GitHub README news line is dated 9 September 2026, and the arXiv paper was submitted on 8 September 2026. OrcaRouter's 9 September check also reported the GitHub repository as unreachable, while the repository was public on 10 September.
• Affiliations. The code lives under Tencent-Hunyuan. ComfyUI Wiki describes the project as Tencent Hunyuan with Shanghai Jiao Tong University and the Shanghai Innovation Institute. Individual author affiliations were not confirmed.
• Speed in practice. The 4.5× figure is a lab-reported ratio against the 32-step teacher. Absolute latency, memory needs and quality at 4 steps on hardware you control are not published in the report summary tables.
• Safety. With zero-shot voice cloning and accent, emotion and content editing in one open MIT-licensed release, the report offers no analysis of misuse.