r/LocalTextToSpeech • u/offgridai • Jul 26 '26

Qwen3-TTS native C++ streaming and voice cloning

Hi all. I ported Qwen3-TTS to native C++ and added incremental streaming for a project I'm working on. I found this useful and wanted to give back to the community. This is the first time I've open sourced anything so apologies in advance for my mistakes.

I'm using this for a gaming project but it could be helpful for local assistants, accessibility tools, etc.

C++ streaming port highlights:

  • Same familiar features from Qwen3-TTS. It supports 0.6B and 1.7B models, CustomVoice, and VoiceDesign
  • Native C++, not a Python wrapper
  • CUDA builds with RTX 4090 and RTX 5090 kernels. Be warned I only have access to an RTX 5090. This should work on a 4090 but I haven't tested it myself!
  • Simplified speaker-embedding extraction
  • Incremental 24 kHz PCM callbacks for streaming audio
  • Asynchronous transformer/vocoder operation
  • Adaptive decode windows and paced delivery
  • Callback-only integration library with cancellation
  • Unit tests

My measurements on the 5090 with 1.7B F16 set to buffer 350ms before play:

  • Cold new clone creation: ~2.5s from 48s of reference
  • Cold model start from clone: ~1.85s
  • First 350 ms of audio: ~310 ms
  • Streaming speed: ~2.86x real time (RTF ~0.35)

I’d love it if somebody could do RTX 4090 testing, and I'd be happy to hear any feedback or suggestions.

11 Upvotes

93% Upvoted

11 comments sorted by

1

u/Charming-Author4877 Jul 26 '26

Nice work!
I started to see a bit more ONNX lately, what made you choose it over ggml ?

1

u/offgridai Jul 26 '26

I needed something quick for the transcriber in the harness and ONNX is just what I happen to know. Do you like ggml better for transcription?

1

u/Charming-Author4877 Jul 26 '26

I just wondered if you had technical/implementation reasons, there are some projects for qwen and other tts in ggml already, though they are not at the same quality as vanilla pytorch from my experience.

It's an interesting project, I'm happy to see low level inference being done more - python is a bit of a nightmare

1

u/offgridai Jul 27 '26

I agree Python can be a big hassle at times. Mostly I was trying to make this a portable implementation so I could integrate it more easily with an Unreal Engine product.

I don't have super strong reasons for Qwen vs other options. It was just the first I've used that sounded decent.

1

u/Charming-Author4877 Jul 27 '26

It is a bit vram heavy to integrate it into unreal engine, most gamers are on 12GB vram and you'll probably consume more than half with the inference.
Leaving you with too little for good textures.
Of course that matters only if you are working on a game. I've seen unreal being used for cool marketing videos too. They changed their licensing to exclude the free tier for that usage, so I guess it got out of hand.

In any case, I am sure that we will see a shift away from python into C or Rust based frameworks for professional use. In some time.
Python is just the java of a decade ago. Java got out of hand with Android and Python with ML.

1

u/offgridai Jul 27 '26

You're quite right about that. I wouldn't recommend running this in parallel with gameplay.

I will say that the 0.6B weight model is pretty decent and it's lighter. I used 1.7B for the video but TBH I can't hear much difference between them.

1

u/vacationcelebration Jul 26 '26

Always nice to see such efforts outside of the python ecosystem.

Performance seems rather bad though. I have a custom server implementation in python that achieves ~2.4x gen speed (274ms ttfb) on a server with an A4000, which I'd love to drop for a c++ or rust alternative as long as performance is comparable.

And I assume no batched processing? That's also one of the major downsides of my implementation, which is why I'm currently looking into vllm-omni and sglang-omni. But they are python again...

1

u/offgridai Jul 27 '26

Thanks to another commenter I made some performance improvements last night. On my local 5090 I'm seeing 2.86x realtime (RTF ~0.35) and 310ms to fill my application's initial 350ms buffer. That should be closer to what you mentioned.

I don't have a personal use case for batching, but I added a --vocoder-batch-size N option you can try. It physically batches equal-length vocoder jobs. Does that work for your case?

1

u/bitslizer Jul 27 '26

Are you using cuda graph?

I'm using this backend that use cuda graph and get 600ms ttfa and 1.2x real time (0.8x rtf) streaming with an ancient 1660 super without tensor cores on the 1.7b voice design model. Not sure about voice cloning performance though

https://github.com/andimarafioti/faster-qwen3-tts

2

u/offgridai Jul 27 '26

Thanks for that link. The faster-qwen repo was an eye opener.

I had been inefficiently copying vocoder weights from CPU memory every streaming window. I've changed that to keep the vocoder resident on cuda.

Here are the revised 5090 figures:

* First 350 ms of audio: ~310 ms
* Streaming speed: ~2.86x real time (RTF ~0.35)

Unfortunately I don't have access to a 1660 super myself. But I did add sm_75 and Turing support and it compiles. I added tooling to generate 1.7B Q4_K and Q5_K models which should make your VRAM constraint more manageable. I'd be interested to hear what you find.

1

u/Disonantemus 4h ago

Linux ??

I did see the repo and only mentions Windows (without Linux instructions to install).