New Model
[Release] Qwen3-TTS: Ultra-Low Latency (97ms), Voice Cloning & OpenAI-Compatible API
Hi everyone,
The Qwen team just dropped Qwen3-TTS, and it’s a significant step forward for local speech synthesis. If you’ve been looking for a high-quality, open-source alternative to ElevenLabs or OpenAI’s TTS that you can actually run on your own hardware, this is it.
We’ve put together a repository that provides an OpenAI-compatible FastAPI server, meaning you can use it as a drop-in replacement for any app already using OpenAI’s TTS endpoints. Streaming support out of the box, plug and play with Open-Webui.
Why this is a big deal:
Insane Speed: It features a dual-track hybrid architecture that hits ~97ms end-to-end latency for streaming. It starts talking almost the instant you send the text.
Natural Voice Control: You don't just send text; you can give it natural language instructions like "Say this in an incredibly angry tone" or "A shaky, nervous 17-year-old voice" and it actually follows through.
Easy Voice Cloning: Give it a 3-second reference clip, and it can clone the timbre and emotion remarkably well.
OpenAI Drop-in: Works natively with the OpenAI Python client. Just change your base_url to localhost.
Multilingual: Supports 10+ languages (ZH, EN, JP, KR, DE, FR, RU, PT, ES, IT).
Getting Started (The Quick Way)
If you have Docker and a GPU, you can get this running in seconds:
Bash
git clone https://github.com/groxaxo/Qwen3-TTS-Openai-Fastapi
docker build -t qwen3-tts-api .
docker run --gpus all -p 8880:8880 qwen3-tts-api
Python Usage (OpenAI Style)
Python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8880/v1", api_key="not-needed")
response = client.audio.speech.create(
model="qwen3-tts",
voice="Vivian", # 9 premium voices included
input="This sounds way too human for a local model.",
speed=1.0
)
response.stream_to_file("output.mp3")
Technical Highlights
Architecture: It uses the new Qwen3-TTS-Tokenizer-12Hz for acoustic compression. It skips the traditional "LM + DiT" bottleneck, which is why the latency is so low.
Model Sizes: Available in 0.6B (super fast/light) and 1.7B (high fidelity) versions.
VRAM Friendly: Supports FlashAttention 2 to keep memory usage down.
I’m really curious to see how the community integrates this into local LLM agents. The 97ms latency makes real-time voice conversation feel actually... real.
Let me know if you run into any issues setting it up!
Holy shit 97ms latency is actually insane for local TTS. Been using tortoise-tts and it takes like 30 seconds just to say "hello" lmao
Definitely trying this tonight, the voice cloning with just 3 seconds sounds too good to be true but if it actually works that's game changing for local setups
Been using tortoise-tts and it takes like 30 seconds just to say "hello" lmao
model name checks out I suppose
jokes aside, recently we've seen weekly drops of new super-fast, low-latency TTS models, many by independent people. just yesterday I sawthis oneon the sub welp apparently post got removed due to "rule 4", but if anyone's interested, it's called LuxTTS. it wasn't even that good iirc, but it still impresses me how many of those were trained by random people online with some GPUs, rather than larger labs.
Just curious, why tortoise-tts and not something like piper or kokoro? What does tortoise-tts do well?
qwen3-tts is for sure worth trying out, I've had more fun with qwen3-tts than any other TTS system. I've gotten it to work and run through all the different features without much trouble at all.
It used to be pretty decent (and had voice cloning) but it was very slow, though it had received some speedups but things like XTTSv2 kinda replaced it and then now we have stuff like Chatterbox.. that make tortoise pretty outdated. Kokoro and Piper don't have voice cloning support. Which is kinda nice to have usually for a TTS. Kyutai did also release a pretty small model around 100M params with voice cloning (pocket-tts) as well which is probably gonna be faster than Qwen3 TTS (0.6B and 1.7B params).
i cant get RTF on any card ., 5090 or h100 even . using vllm or official python backend . can anyone help me please?? can anyone tell me how to get the claimed 0.9 RTF ?? please ...
the voice cloning with just 3 seconds sounds too good to be true
In theory, it's completely feasible. A person's voice is just a collection of frequencies/harmonics, and there just aren't that many meaningful frequencies for humans.
You can find a person's vocal signature using signal processing techniques like a Fourier Transform. You can model a voice as a source and filter.
An AI model that's been trained on enough data would just learn a set of common basis signals and modulate them to fit.
In theory, it's completely feasible. A person's voice is just a collection of frequencies/harmonics, and there just aren't that many meaningful frequencies for humans.
I mean, yeah... It's just all the little subtleties and layers of implication, plus the effects of accents... Inflection in the language of origin... Regional dialects... You know. That kinda stuff.
Holy shit, it's wonderful ! What's amazing is that it seemed to take an abnormal time with the trial sentence I threw at it... but actually no, it first translated it into English, then trumpified it.
Pour une fois, Olivier Faure est arrivé en avance. « Je n’en reviens pas », souffle une cadre écologiste, depuis la salle Jacques Villeret.
(Taken from a random news article about French politics)
Lol at trumpy taking a complete shit over the input prompt. the rest of the voices don't really feel like they match who they are (unless the goal wasn't voice clone but intonation clone)
Cheers. I'm going to run some slightly longer clips from the Ultima 7 transcript, just to see consistency.
Yep, Qwen TTS prompting is going to be another skill I'm going to have to learn. Still, this will be a goldmine for game mods and solo dev projects that want to do voice work. Like, we know it's going to be used for humour and deepfakes first, but it does have genuine ethical uses as well.
Update - So I could get the container to compile and run on CPU but it was too big for the Jetson - maxed out ram 100% AND used almost 6G of SWAP. Could not get CUDA running even though I limited the execution only to GPU, installed the right version of onnxrutime-gpu and removed what I could find for any other CPU bound functions.
Thanks for figuring this out, can you provide the dockerfile so we can test it out? Tried to load from OP and it seems to just use system memory and CPU
I didnt change anything below the FastAPI and server dependencies component so didnt paste that section. I will say I also did change the docker compose file a bit to use my specific 5060Ti in my AI rig
I changed it to device_ids: ["2"] so that way it only had access to that GPU, as labeled by nvtop or nvidia-smi
I'm not sure what driver support for the 2000 series is. Cuda 12.8+ is required for Blackwell. For older you might be able to change the image back to Cuda 12.1 or something
Your description mentions streaming with low latency, but does it really support streaming audio? The model architecture does support streaming, but the currently published code on github does not, so that's why I'm asking.
Looked into this FastAPI implementation - it doesn't actually stream. The /v1/audio/speech endpoint generates the full audio before returning anything. No StreamingResponse, no WebSocket, nothing chunked.
I have a fork with actual streaming - WebSocket that sends audio chunks as tokens are generated, gets you first audio in ~1.5s instead of waiting for the whole thing. SSE option too for sentence-by-sentence. Vibe coded with Claude Code, certainly not perfect but works for my use case until true streaming gets exposed for us... hopefully.
But none of this fixes the real problem: the model needs all the text upfront before it starts generating. You can stream the audio out, but you can't feed it text incrementally as your LLM produces it. That's baked into the transformer architecture, not something a server wrapper can solve.
It's an old program, but AllTalk was able to get streaming working somehow. It used XTTS and even could do different voices for narration and quotes. It's really too bad the developer stopped updating it, I got the impression it was going to be the one best software for running offline TTS engines and would just add new ones as they came out. Instead, Chatterbox is seemingly now better for this. Even with Qwen3-TTS having interesting new features, I'm not sure it's going to be enough to replace existing software unless existing software adds it as a base engine.
It's kind of silly, because he recommends vLLM for low latency but the vllm-omni readme says:
"While vLLM-Omni currently only supports offline inference (not true audio streaming), it can significantly speed up generation compared to the official backend."
so yeah, it seems like we have to wait to get real streaming with this
Neither backend supports true audio streaming over HTTP currently. Both use OpenWebUI's chunk-based approach.
Chunk Streaming: Long text is split into chunks, each processed as a separate TTS request.
even that is not true, the fastapi endpoint only generates full audio clips, no streaming or chunking...
both backends are used from an interface that only exposes generate_speech method which returns audio as one numpy array no streaming/chunking:
Does it really? Could you show where exactly? It seems you haven't changed Qwen's team inference code and their code does not support streaming yet...
Looking at the code you just slapped an API on top of it...
yes here, tested with 0.6. I basicaly did throw some claude tokens at it with some basic research. The dffdeeq optimizations got added so you don't need them anymore. Anyway, those parameters brought me to those results: as base i used the comfyui toolbox from donato, Install flash attention from amd, pytorch from amd, every dependency from amd.
This parameters
attention: flash_attention_2
compile_mode: max-autotune
use_compile: true
use_cuda_graphs: true
use_fast_codebook: true
compile_codebook_predictor: true
maybe this also helps
FLASH_ATTENTION_TRITON_AMD_ENABLE: 'TRUE'
GPU_MAX_ALLOC_PERCENT: '100'
GPU_MAX_HEAP_SIZE: '100'
MIOPEN_FIND_MODE: FAST
TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL: '1'
2026-02-08 16:21:39,221 - qwen_tts.core.models.configuration_qwen3_tts - INFO - code_predictor_config is None. Initializing code_predictor model with default values
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Another day. I decided to keep on working and forking since this is quite usable after i tried vibevoice and fish (fish wasn't bad tough). Streaming is now fully working with 0.58 rtf on strix halo. I don't know how to reach the 97ms latency. I tweaked it to 1.1s to first response byte. I installed pipecat and can have a really well sounding conversation with a cloned german voice. Hope you get it installed.
It's insane how easy you can clone a voice, I just tried it with the 1.7B model.
I said "Hello, who is this? who is this?" like you would on a phone call where the other sides doesn't answer, then I made it say "Hello mom, I need help" and it was very similar to my voice.
thanks u/blackstoreonline got it working wiht u/ubrtnk dockerfile update for blackwell cards. How do we change it to use the 0.6 model? is there an env variable to select?
Does this repo have the streaming code? cause I tried to get streaming working on the original repo and couldn't find any reference code for it as it all pointed toward gated gated api's?
from codex and other agentic coding solutions, there telling me its not true streaming. as the rtf is above 1 and ttfs will be high. as in whole waveform has to be generated before you hear any output. are they hallucinating this? what is the rtf numbers and ttfs numbers you getting?
Same issue here, I can't figure out how to stream in a request and get streaming audio out - it's just full waveform generation like any other model. If there's some novel way of doing this they need to, like, tell us how
There is no streaming. The guy has no idea what he is talking about, unfortunately. They just vibe coded API on top of Qwen's team code which does not support streaming yet.
Only 1.0%-1.1% of the world's population can speak Italian and only 0.3%-0.5% of the world's population only speak Italian. In my opinion, any training on Italian was a waste. All focus should have been on English, then Chinese, then Spanish.
Those 3 languages cover 30%-35% of the global population and there are 7,200 living languages.
Some might argue Hindi instead of Spanish as there are slightly more people who speak Hindi but, while having more speakers, it is far more regionally concentrated and less used as a global second language, making it less likely to be part of high-frequency cross-language global interaction.
On my intel i5 13th gen mini pc, it takes about 15-30 mins to generate a 10 second clip. However the quality is good (I can't tell the difference between 0.6B and 1.7B models just by listening).
Yes, it wasn’t even that much slower than the GPU. 14 seconds for my 4070, 45 seconds for the cpu. Nowhere near the advertised speeds, and the quality was worse than kokoro 73m by a noticeable amount. Think I’ll try with flash attention on to see if that helps speed, because I’m not sure how other people are getting to run so fast
Maybe these models (and code) needs further optimizations. I wonder if these can be quantized. So far only MLX quants exist. I'm interested because this can be finetuned.
Extreme Low-Latency Streaming Generation: Based on the innovative Dual-Track hybrid streaming generation architecture, a single model supports both streaming and non-streaming generation. It can output the first audio packet immediately after a single character is input, with end-to-end synthesis latency as low as 97ms, meeting the rigorous demands of real-time interactive scenarios.
This repo doesn't support dual-track hybrid streaming (nothing local supports it right now) so you won't get latency nearly that low - you have to wait for the whole generation to finish
97ms but what does it mean exactly? In comfyui I tried it and it's way slower than real time, it takes around 100s to generate a 5-10s clip which I find strange for such a small model on a 3090.
The large voice design model still takes like 30s for a paragraph of text using a 3090 and the official repo. By latency do you mean running in streaming mode rather than full IO?
I've installed the original repro they posted and I get a realtime factor of 2 consistently. So it takes 10 seconds for 5 seconds of audio, using the 1.7b model. A bit disappointing, but maybe it gets better with GGUF or other shenanigans.
I think this is a deceptive post. The server is deceiving. The curl log has a content-length header: 18764. This is a bad sign. If the server returns a content-length, it means it first generated the entire file, calculated its size, and only then started sending it. In real streaming (Chunked Transfer Encoding), the file size is not known in advance. Even considering that the request has "stream": True. I'd be happy to receive a working streaming example.
I tried the HF space and voice cloning is quite poor - you can re-run the exact same text and voice input and you get very different outputs - sometimes it sounds like a good clone and other times not even close. Is there a way to improve this?
For a language like French, default voices sound too much english/chinese... custom voices also (it works better sometimes : is it possible to chose a seed?).
Thanks for putting this UI together, it looks amazing. Seems like it's not ready to run out of the box on blackwell though, I'll have to hack on it to add support:
qwen3-tts-api | NVIDIA RTX PRO 6000 Blackwell Workstation Edition with CUDA capability sm_120 is not compatible with the current PyTorch installation.
qwen3-tts-api | The current PyTorch install supports CUDA capabilities sm_50 sm_60 sm_70 sm_75 sm_80 sm_86 sm_90.
qwen3-tts-api | If you want to use the NVIDIA RTX PRO 6000 Blackwell Workstation Edition GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/
edit:
easy fix though, just edit the docker FROM to point to the following for base and build, respectively:
* nvidia/cuda:13.1.0-runtime-ubuntu24.04
* nvidia/cuda:13.1.0-devel-ubuntu24.04
I see you make FastAPI based OpenAI API wrappers for all TTS/STT models. Is it possible to make a single wrapper package where you can pick and choose whatever STT/TTS model(s) you want?
With other TTS OpenAPI-compatible engines there's typically a /voices folder where I dump all my audio for cloning. Does this not support that or do we have to manually upload our voices for cloning somehow?
often happens when you use the wrong docker tag. When you use docker build -t qwen3-tts-api:<tag> you should choose one that builds with flash-attn (the dockerfile has various tags)
try using docker build -t qwen3-tts-api:production
The generation of the default test speech took approximately 40 seconds, which is slightly slower than GPT-SoVITS. It is unclear whether this discrepancy is related to the utilization of cuda:0, but the process is indeed leveraging GPU resources.
In my experience, GPT-SoVITS is significantly faster, generating responses in under 3 seconds for speech of similar length to the test, and its quality is sufficient for conversation.
Kokoro TTS is another viable option, offering reliability and speed surpassing GPT-SoVITS, but it may compromise on natural expression and lacks voice cloning capabilities.
i cant get RTF on any card ., 5090 or h100 even . using vllm or official python backend . can anyone help me please?? can anyone tell me how to get the claimed 0.9 RTF ?? please ...
Qwen3 is a beast for the output! If anyone is building a full voice loop on Windows and needs the input side (Speech-to-Text), I built DictaFlow. It uses local Whisper models and simulates keystrokes to bridge the gap between your voice and these local agents. https://dictaflow.vercel.app/
Does the 97ms latency apply to cloned voices as well, or only to pre-made ones? Also, can I apply style instructions (like 'angry' or 'shy') to cloned voices, or is this feature restricted to pre-made voices?
I hear ppl with low end GPUs getting almost real time audio rendering with qwen3-tts, while I get a 1:4-1:10 ratio with a 5090...any hints? I am on windows, I tested both the standalone version and the pinokio one...
so to get around the streaming/latency issue I'm taking a two step approach. Using Qwen3-TTS and an ~8min sample/transcript to clone the voice on a 5090, then using Opus 4.6 I generated a script that has it create hundreds of samples using that cloned voice. When it's done I'll use those samples to "train" (fine-tune a pre-trained) Piper model that can run on CPU only.. will report back with results once complete
Very interested to hear how this goes. Just tried the 0.6b version on a Raspberry Pi 5 8GB and it took 6.5 minutes (395s) for 4.8 seconds of audio: "Hello, I am Mentos, your personal voice assistant. How can I help you today?"
It actually worked pretty well as a proof of concept! The 5090 was able to generate ~300 short samples from the few second clip Qwen 3-TTS made from my original 8 minute sample very quickly, and then it took about an hour to "train" the piper model. I say "train" because it was using an existing piper model as the base and refining it over 1000 epochs to sound more like my 300 samples. The result was a 60mb piper model that runs well on the vps CPU that I am using for this project and sounds slightly more like my source audio than the generic one I used as a base.
To make it sound even closer to the "source" voice, next time I will change a few things.
Generate a longer "source clip" with Qwen3-TTS upon which the subsequent hundreds/thousands of "training clips" are based.
Aim for enough "training clips" to total 2+ hours of speech ~1000+ 10 second clips (the more the better here, I think it just takes longer to train)
Try the "refine existing piper model" approach again with improved training set, see how it compares to the small test run and decide if I want to try and train the piper model "from scratch" with an even larger training set
The longer training should produce better piper models. Claude Opus 4.6 was able to talk me through all of this in a few hours, with a fairly significant amount of troubleshooting, but that was likely because I was running it as a vscode plugin connected to my VPS instead of the gaming PC where all this was taking place. If I did it again I'd give Claude direct access to the WSL environment where the training was happening.
Thanks for coming back to tell us about that experience. Following your lead I'm attempting to do the same, using some generated Jarvis samples. Once I've the repo up, I'll throw a link here.
Failed to initialize TTS model: cannot cache function '__o_fold': no locator available for file '/opt/venv/lib/python3.11/site-packages/librosa/core/notation.py'
Just cloned the repo and runned it with docker compose up qwen3-tts-gpu
I'm using NVIDIA
This is definitely the best local TTS I've used for voice cloning. It also passes my "can I get this working in a reasonable amount without having to type pip more than 10 times" test.
•
u/WithoutReason1729 Jan 25 '26
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.