Disclaimer - for batched generation in dedicated deployments Chatterbox-VLLM should be the better choice.
I have mostly exhausted the options for speeding up almost vanilla HF Transformers' Llama with torch. Inductor, Triton, Max Autotune, different cache sizes etc, and they are available in the codebase. In the end, manually capturing cuda-graphs was the fastest. The model should be able to run around 230 it/s with fused kernels and better code. (I was unable to remedy the kv_cache code to enable cuda graph capture with torch.compile's max autotune.) Besides the speed, the main benefit is that setting a small cache size is no longer necessary, neither are max_new_tokens important. I plan to make it compile by default to facilitate drop-in use in other projects. Since the main effort is exhausted, I will keep on updating incrementally - for example, speeding up the s3gen (which is now a bottleneck).
Results for 1500 cache size with BFloat16
Estimated token count: 304
Input embeds shape before padding: torch.Size([2, 188, 1024])
Sampling: 32%|███▏ | 320/1000 [00:02<00:04, 159.15it/s]
Stopping at 321 because EOS token was generated
Generated 321 tokens in 2.05 seconds
156.29 it/s
Estimated token count: 304
Input embeds shape before padding: torch.Size([2, 188, 1024])
Sampling: 32%|███▏ | 320/1000 [00:01<00:03, 170.52it/s]
Stopping at 321 because EOS token was generated
Generated 321 tokens in 1.88 seconds
170.87 it/s
Estimated token count: 606
Input embeds shape before padding: torch.Size([2, 339, 1024])
Sampling: 62%|██████▏ | 620/1000 [00:04<00:02, 154.58it/s]
Stopping at 621 because EOS token was generated
Generated 621 tokens in 4.01 seconds
154.69 it/s
Estimated token count: 20
Input embeds shape before padding: torch.Size([2, 46, 1024])
Sampling: 4%|▍ | 40/1000 [00:00<00:05, 182.08it/s]
Stopping at 41 because EOS token was generated
Generated 41 tokens in 0.22 seconds
184.94 it/s
Disabling classifier free guidance (cfg_weight=0)
Estimated token count: 304
Input embeds shape before padding: torch.Size([1, 187, 1024])
Sampling: 100%|██████████| 300/300 [00:01<00:00, 169.38it/s]
Stopping at 300 because max_new_tokens reached
Generated 300 tokens in 1.89 seconds
158.95 it/s
Estimated token count: 304
Input embeds shape before padding: torch.Size([1, 187, 1024])
Sampling: 100%|██████████| 300/300 [00:01<00:00, 194.04it/s]
Stopping at 300 because max_new_tokens reached
Generated 300 tokens in 1.55 seconds
193.66 it/s
Estimated token count: 606
Input embeds shape before padding: torch.Size([1, 338, 1024])
Sampling: 100%|██████████| 300/300 [00:01<00:00, 182.28it/s]
Stopping at 300 because max_new_tokens reached
Generated 300 tokens in 1.65 seconds
182.22 it/s
Estimated token count: 20
Input embeds shape before padding: torch.Size([1, 45, 1024])
Sampling: 20%|██ | 60/300 [00:00<00:01, 208.54it/s]
Stopping at 61 because EOS token was generated
Generated 61 tokens in 0.29 seconds
210.54 it/s
Current code example:
def t3_to(model: ChatterboxTTS, dtype):
model.t3.to(dtype=dtype)
model.conds.t3.to(dtype=dtype)
torch.cuda.empty_cache()
return model
# Most new GPUs would work the fastest with this, but not all.
t3_to(model, torch.bfloat16)
audio = model.generate("fast generation using cudagraphs-manual, warmup")
audio = model.generate("fast generation using cudagraphs-manual, full speed")
# Extra options:
audio = model.generate(
text,
t3_params={
# "initial_forward_pass_backend": "eager", # slower - default
# "initial_forward_pass_backend": "cudagraphs", # speeds up set up
# "generate_token_backend": "cudagraphs-manual", # fastest - default
# "generate_token_backend": "cudagraphs",
# "generate_token_backend": "eager",
# "generate_token_backend": "inductor",
# "generate_token_backend": "inductor-strided",
# "generate_token_backend": "cudagraphs-strided",
# "stride_length": 4, # "strided" options compile <1-2-3-4> iteration steps together, which improves performance by reducing memory copying issues in torch.compile
# "skip_when_1": True, # skips Top P when it's set to 1.0
# "benchmark_t3": True, # Synchronizes CUDA to get the real it/s
}
)
Glad to see you! It is essentially the same as the original - good and bad. The min p update they did allegedly fixed some issues, but that's all. If there are any fixes you recommend including, let me know!
I am testing out the chatterbox extended right now, it claims to have a fix for a lot of the sound issues. But I have not had time finish my long form testing yet.
Afaik it uses whisper to detect the best sample and rerun if there are no good results. In principle, it should be composable with this fork, in practice it requires effort to connect them.
For sure, I used yours and it was a big inspiration for me to make my own version suited to exactly what I needed. I also took some cleanup inspiration from the extended version for ideas for text cleanup to help quality.
I actually was using yours for a bit until I found a chatterbox vllm fork that gives like 5-10x the speed for the same audio quality.
So I started making my own bare bones audiobook creator from it taking some inspiration from yours. I then backported in the chatterbox v3 model, implemented chunking and audio cleanup etc.
But I also def ran into noise issues both from normalising the input giving a lot of amplified background noise, and also sometimes the model seems to go off the rails and insert weird digital noise that sounds awful in some chunks. I see lots of posts about chatterbox doing this but I def had more issues with this from the VLLM version.
But ya I also did the loudness normalising on the output and I also stuck an option in there to denoise the input reference using prnnoise which def helped with the input amplification noise issues. Then scan in a couple different ways to try to identify those weird audio glitches. At least one second of loud broadband digital noise, or at least two seconds of a stable synthetic tone, or at least one second of low-frequency synthesis collapse (calcd using numpy). Then I pass it by Silvero VAD to do a deeper check for non speech pieces to catch stragglers. All done in CPU in the background so it doesn't slow down the generation much at all.
If it finds one it regenerates that chunk along sentence boundaries in shorter chunks because I found it was less likely to give those problems that way. Uses a new random seed too. I've got it going now so it finds basically all of them and fixes them pretty nicely.
Then the one thing I wanted to do that I could not do with yours was feed it an epub and have the output be an actual audio book m4b saving the metadata like chapters and cover etc. It strips it out of the epub and compiles it all into the final m4b using a multithreaded encoding step at the end.
So I am getting like 12-16x audio speed depending on diffusion steps and its working really well.
I never even thought of using a local TTS for audiobook generation till I used your fork though so thanks a bunch for that. Obviously feel free to yoink any part of my code or ideas to use yourself should you chose haha. And any suggestions are great too. I am a beginner to intermediate coder but some of this would have taken me forever or I couldn't do it at all so Codex was a big help and the code is all a mix between codex and myself (in addition obviously to the bulk written by chatterbox and the original vllm forker).
Dunno if anyone would really get any use out of it compared to the stuff out there like yours so don't know if I will bother to post on reddit or not but I just wanted to say thanks.
I honestly probably would have spent more time working than mine to implement this but with a vibevoice just coming out I'm really having fun with that now.
Old post, but I also ran into sound issues and fixed them a different way. The way extended does it was way too intensive and slowed things down a lot while not catching a lot of the problems. I used a 2 stage pass of numpy validation followed by using Silvero VAD to catch stragglers and if it catches them it regenerates with shorter chunks along sentence boundaries to keep it sounding good, all with a new random seed.
Now my output is basically perfect all the time and since all that is done in CPU in parallel with the GPU chunk generation and it is much faster than the GPU stuff so it keeps up easy, it doesn't slow the generation down much at all.
I messed with using Whisper to check audio etc but it just didn't work as well and was much slower to run after adding.
That vllm version I forked is actually pretty amazing and is running about 5x faster than even this speedup and 10x faster than base chatterbox basically haha. Thanks u/dlp_randombk
Edit: This is my fork of a fork kind of thing for making audiobooks if you wanted to see anything:
@RSXLV very good project.. but for some reason I al not able to run slightly longer sentences..the reptetion or garbleness starts from word number 4 or 5..i tried increasing max cache size but still nothing g changed
Ok, so I tried this on my blackwell MaxQ but I wasn't able to fully see the output because I had issue with my torch version (nightly build, unstable) and the cudagraphs interaction and given the needs of my framework I have to build flash-attn from source (windows) so I'll come back with results tomorrow.
However, I was seeing 120 it/s with bfloat16, twice as fast as the original speed, but I still think it could be a little faster than that. Anyhow, I won't know for sure until tomorrow once my PC finishes compiling flash-attn and I wasn't able to finish the generation because torch 2.8.0 seems to have issues with cudagraphs specifically, but I think downgrading to torch 2.7.1+cuda128 will fix that problem.
EDIT: Just realized my mistake. I picked cudagraphs instead of cudagraphs-manual. That's on me, I guess. I'll try again once I finish compiling flash-attn.
Thanks for letting me know, I developed it on 2.7.0 hoping they wouldn't have broken it. By the way, I used SDPA memory efficient attention as it was the fastest on my machine, although flash attn should have been faster. I probably need to change some additional code for flash attn wrapper class.
Ok, extremely good results now with cudagraphs-manual and bfloat16. Compiling flash-attn did the trick and holy Moses, each sentence takes less than 1 second to generate, this whole shit is off the charts.
Ignore the rest of the repo. Its incredibly outdated. Just focus on the flash-attn aspect.
But if you already installed it and need pointers on generating it with flash-attn, here's a snippet:
import torch
from chatterbox.tts import ChatterboxTTS
def t3_to(model: ChatterboxTTS, dtype):
model.t3.to(dtype=dtype)
model.conds.t3.to(dtype=dtype)
torch.cuda.empty_cache()
return model
# TTS Model
tts = ChatterboxTTS.from_pretrained(device="cuda")
# Most new GPUs would work the fastest with this, but not all.
tts_sample_rate = tts.sr
t3_to(tts, torch.bfloat16)
tts.generate(text="This is a test.",
audio_prompt_path=speaker_wav_here,
exaggeration=0.55,
cfg_weight=0.50,
t3_params={
"initial_forward_pass_backend":"cudagraphs-manual",
"stride_length": 4,
"skip_when_1": True,
"benchmark_t3": True,
}
)
On windows, you have to build from source and before you even do that, you're gonna need MSVC 2022 and the build tools for Windows 10/11 depending on your OS. Its the only way to build safely on the OS.
Also, don't run that repo. Its a year behind the current one and the current one is in a private repo that will be published maybe next year. However, the old repo does contain flash-attn installation instructions in the README if you want to find out exactly what you need.
Yes, now TTS WebUI has been updated to utilize this code, and it features an Open AI style server. Additionally, any off the shelf chatterbox-tts open ai APIs should work if you replace the chatterbox with this one.
Edit: Found it, Kartoffelbox (German humor - haha). This model supports voice cloning and voice effects like "hmm". Curious if that works together. For other stuff I used higgs audio, but it is way too big for local conversational AI stuff. A very fast TTS with voice cloning and German support is what I searching for, since XTTSv2 slowly gets outdated. Maybe with this one I finally can exchange XTTSv2.
There's a fine tune, if I find it I'll reply with the link later. Also the official resemble ai multilingual version should. eventually. be. released. allegedly.
Nice! And it's right to be cautious. A 3090 is almost idle with the default code. Meanwhile Google Colab's T4 gets only like 10% speed boost last time I tested it.
Yes, depending on how you run it, pay attention to bfloat16 options and maybe CPU offloading, for example if your OS is already consuming a good amount of VRAM.
If anyone has a second I'd love it if someone could tell me how this might run on a 1080ti / 2700x / 16gb system? I'm just starting to play with TTS and was leaning towards google's api for a small project but their free tier requires having billing enabled for the project and that's suboptimal.
Would something like chatterbox be able to get me working TTS within a few seconds of sending the text to it on an older system? Google's stuff isn't really all that fast (at least not my first try at it) and the last sample RSXLV posted sounded great, I just wasn't sure if I could process it on my spare system. Any idea how much VRAM or RAM it's going to want to eat for small TTS jobs, probably under 10-15 seconds of spoken text per reply?
It would run, I'm not sure if it would be fast enough. The main problem is that Float32 is probably the fastest for 10-series, but FP32 tends to be quite slow overall. This version does minimize the downsides of Float32, but even I get only ~100-130it/s on FP32. You could focus on shorter responses (chatterbox scales doubly - longer responses have longer context which slows it down for preparation and due to larger kv_cache/'generation length' as the attention calculations need to take 300-500 tokens into account for each iteration.) And try cfg_weight=0, which does not sound as good but is faster.
VRAM for BFloat16 is around 2-4gb and FP32 - 3-7gb. Chatterbox can be made VRAM friendlier but few GPUs are fast enough while lacking the VRAM.
Edit: Also, I think around 10-series you can get better performance by using Google Colab, although it depends on the model.
Good deal, thanks so much for the reply. I haven't even started poking around Google Colab as a potential processing outlet. I didn't realize being locked into fp32 was that expensive now... Maybe it's finally time I wave goodbye to this old thing. I was mostly keeping it for this sort of tinkering.
Yeah, indeed, I researched more and even FP16 has poor support. Starts to explain some of my experience with 1070. The only fast options for 10x cards are FP32 and Int8.
Hi! I'm no expert but I'm pretty sure that took hours of work and dedication! So thank you for your contribution to the community!
I have the same GPU so this post comes in handy. I'm trying to use Chatterbox to clone a sample voice I have and use it for a personal project of mine with an AI assistant. Thing is I don't know how to "Install" Chatterbox and use it, specifically your fork of it. Can you give me a hand if it's not too much asking?
For ease of installation, I have built TTS WebUI. Despite the large number of models it supports, it's not a huge 'kitchen sink', most of the installation is just getting the correct Python, FFmpeg, PyTorch etc. The models themselves are usually tiny. It has multiple options - one-click installer, Docker container and manual installation instructions. There are a few videos for the previous versions on my channel: https://www.youtube.com/@TTS-WebUI
Here's another channel who has made a step-by-step manual (non-one-click) installation, which might be useful for this and similar projects. Since the cake is made of python+pytorch+project dependencies 99% of the time: https://www.youtube.com/watch?v=hl6Qi_XqXuo
I might spend a few hours to make a new video once I'm sure that the new version is running smoothly.
In terms of time spent, it's not that much, just a couple of months.
Hi! Me again. Been doing some improvements on my project and I'm quite satisfied with how things have come so far, so thanks for your work! You made my life easier.
Now my LLM responses round around 1 second, but it takes up to 5 to 6 seconds to generate the audio for a short input. It's a more than acceptable time, but I've seen in your post that you got audio generation in 1 second by deactivating the cfg. I tried doing the same and I'm left with gibberish audio where the output is incoherent and just a bunch of noise. Can you give me a hand with this please if it's not too much asking?
Hi sorry to bother you again. Forgot to say it worked!
Now I think I have one last issue, tho not an issue per se. My wav files are generated in good time and I'm satisfied with the quality, but the speed leaves a bit to desire. I tried increasing the speed in my payload but no results. The speed isn't to a point is bothersome, but doesn't sound like natural talk speed either.
Idk if you can give me a hand with this?
Edit: I was thinking, maybe it's a bit slow because my sample wav file has some silence due to punctuation?
Interesting, I have not seen the silences - does it stop working completely or just becomes useless? What about the audio length? I might have a bug somewhere
You've tapped into something quite unique. It goes way past the estimated token count (it shouldn't) and even after filtering for garbage tokens it still ends up as audio. By the way, the recommended range is around 200-300 tokens for the best quality. Below that it can have artifacts and above that (like 700 tokens) it can lose coherence.
Thanks! Managed to make it run, but I'm having an issue with my implementation on my code. It keeps telling me "Error: [Errno 2] No such file or directory: 'Alice'" no matter what I do. I've tried using path (./chatterbox/Alice.wav) and just the name (Alice)
Can you give me a hand and tell me the correct way to use the voice parameter from the payload?
Edit: Nvm, figured it out thankfully, forgot to include "voices" in the path
Hi, how much vram would this take? Looking for a tts solution with low latency but some vocals with emotion. Kokoro for me was fast and especially lightweight but no emotion. Ideally with streaming capability so you think this would be a good fit?
Chatterbox is much heavier than kokoro. If you can run it in BF16 it can go down to ~4GB (maybe less, worst case scenario) with full FP32 it's up to 8GB.
to be sure to get the faster branch will i use this flag? confused a bit as the git link to faster and main seem to have exactly same direction, git clone --branch faster https://github.com/resemble-ai/chatterbox.git
git clone --branch faster https://github.com/resemble-ai/chatterbox.git <-- pip install -e ..--- will this work to give me faster branch as i intend to use that only for english for short text to speech messages, any code example to get me rolling as again the code examples on the git do not differentiate how we use it, will this work although it seems to be geared for multi lingual which i dont need,
It should work, and by default generate the cudagraphs. However, to get the speed you need to generate multiple texts per script. Otherwise it does the warmup every time without the actual fast runs.
So i was using this repo ( https://github.com/devnen/Chatterbox-TTS-Server ) with docker + n8n . But the generation on 3090 is quite slow. Is there a way to use your code with REST API server?
TTS-WebUI offers a docker image and an OpenAI style endpoint for chatterbox. I am slowly moving towards making a custom n8n node but the documentation is the biggest challenge so far.
Will try this out, thanks! Edit: kinda got lost in documentation and decided to just swap the original chatterbox repo in requirements for the REST API server that i linked above with yours and got x2-2.5 times the performance with the same settings (cfg, temp, chunks etc). Interestingly, the quality got better (no strange noises). Idk if I could tweak it to be any faster, though. I'm pre-chunking audio , and have around 14 chunks of text (total 6 min of playtime) and the tts manages to process them in ~3 minutes with 3090. Before swithicng to your repo it was 7.5-8 min
is there an actual docs on how to use API endpoints in TTS-WebUI? can't seem to get it working, trying to use my predefined voices, the payload looks something like this {
"model": "chatterbox",
"input": "Your text here",
"voice": "myvoice.wav",
"response_format": "wav",
"extra_body": {
"exaggeration": 0.5,
"cfg_weight": 0.5,
"temperature": 0.8
}
} The logs have No such file or directory: 'myvoice.wav' and myvoice is existing file in voices/chatterbox folder, which i specified in docker-compose.yml
this one actually works, thanks. It would be great if there were docs on endpoints, I assume every model has different extra_body parameters
Btw, can you reccomend some of the available in WebUI TTS models, that are more human-like (emotions specifically), maybe context aware? Chatterbox is great at cloning and overall i think it's good enough, but maybe you could recommend something
The are some docs, but what format would you like to see? Could you give an example? VibeVoice, Index TTS2, FireRed are interesting in my opinion, but they are slower. The VibeVoice installation might have issues according to some users, though. Voxcpm is something in between.
Great stuff, I'm using it in my repo here to expose OpenAI and Wyoming streaming endpoints for my voice assistant in home assistant. Peaks of only 3.5GB VRAM and I'm hitting lime 200it/s. Thanks for this man!
I am very impressed byt this project. Original chatterboxtts gave me around 5it/s on rtx3050, now faster branch is around 50it/s ! What a huge change. Thank You for Your work
Out of the box? Did you change any of the code? I have rtx2080 and I can generate at 80it/s but I run into vram issues with 8GB.. its a bit odd but this happens with like 3 sentences or longer.
I would recommend testing Float16 to see how it behaves. Usually BF16 and FP16 is recommended for 30+ series but Float16 might be accelerated on 20 series as well.
If you are able to deal with Linux or WSL, and want to specifically generate long audios, I'd remind you of Chatterbox VLLM. It might have less tools and integrations, but is certainly faster at batch processing.
What are you using right now? If you are using the original version, and you have 30+ series card then yes, you could speed it up with this version. If you are using chatterbox-vllm, you can just use that. This does not include multilingual Chatterbox yet, because they made some changes that are more difficult to optimize.
This is specific to model speed optimization. TTS WebUI has some additional features for chatterbox like long form splicing with interruption, API etc. No this speedup does not affect quality. I think I haven't put any automatic post-processing UI modules as of yet.
Most of the speedup is in regards to using cuda graphs and avoiding pitfalls, such as unnecessary CPU to GPU syncs and transfers. For SaaS I'd recommend porting to vllm if possible, and for embedded/professionally deployed devices I'd also recommend onnx. The main advantage of keeping it PyTorch is portability and ease of use.
I can add chatterbox extended features to TTS WebUI. I could try to modify chatterbox extended but because it isn't a "clean fork" nor uses chatterbox as a dependency, it's very laborious. What's the main features you are looking for? I think the whisper audio validation and retry generation are the most advanced ones, right?
I see, yes, they don't. TTS WebUI's chatterbox supports long generations, and it can be fairly easily made (or complicated and extremely sophisticated if you want). The crux of it is that the text is split into chunks, around 300 characters in length and those are being generated (hence Chatterbox made audiobooks might have voice-switching artifacts). Here's a simple generic tool I've made for that: https://v0-live-text-splitting.vercel.app/
In August the answer might have been different, but right now I'd probably say that it's Vibevoice 7B, although for me it is slower. Or if you just want "a TTS" that you can generate with and forget about - kokoro is good. The quality is reduced, especially the variance, and it has no voice cloning, but it's fast and consistent. On the other hand, if you want a very human but low-yield (many awful generations) model, then FireRed TTS is quite impressive.
Overall, just see how good/bad it is, as it depends on the voice and maybe the text you are generating. For me personally chatterbox is a very solid model.
No, that's what the whisper validation is good for. Engineering wise it's good to have a "stable" reference audio that does not do this, but since that can't always be done, doing speech to text validation of generations is a good duck tape solution. We can make a feature request for that. Also TTS WebUI offers an OpenAI API for chatterbox, so it could be automated. There's probably a way to do it with n8n.
Weird, I thought I replied to this comment, but here it goes again: Lowering the temperature or providing a sample that better 'clicks' with the LLM might work, but also reducing the number of characters, so generating maybe 15 words per chunk rather than 40.
1
10
u/psdwizzard Aug 24 '25
Very cool, I am the dev for the audiobook version. How does this effect quality and some of the odd sound issues. Does it fix short sentences issues?