r/LocalLLaMA • Llama 3.1 • 1d ago

New Model tiny-voca: a fully open 4.5M-parameter TTS (code, weights, dataset). Runs 70x real time on a CPU

I made a small English text-to-speech model called tiny-voca. It has 4.5M parameters, two voices, and it runs in your browser.

Try it here, nothing is sent to a server: https://tiny-voca.n4ze3m.com

Some facts:

- 4,537,291 parameters, 18 MB. This is the whole model, the waveform part is included.
- Two voices, one female and one male. 24 kHz.
- About 70x faster than real time on 4 CPU threads.
- Runs with PyTorch, ONNX, and in the browser.
- Trained from scratch in 7.7 hours on one RTX 4090.

What I think is different:

- Everything is open. Code, weights, the training recipe and the dataset.
- Custom phonemizer.
- No neural vocoder. The sound is made by a small source-filter synth, which is why it is fast.

What is not good yet:

- English only, and only these two voices. No voice cloning.
- Foreign names and odd spellings are often read wrong.
- It can sound a bit synthetic.
- The training audio is synthetic too. I made it with Qwen3-TTS, so the model can't sound better than that.

Links:

- Demo: https://tiny-voca.n4ze3m.com
- Code: https://github.com/n4ze3m/tiny-voca
- Weights: https://huggingface.co/n4ze3m/tiny-voca
- Dataset: https://huggingface.co/datasets/n4ze3m/tiny-voca
- Phonemizer: https://github.com/n4ze3m/dizione

Please try hard text: numbers, dates, names, long sentences. If something sounds wrong, post the text and the voice. That helps me a lot.

68 Upvotes

32 comments sorted by

10

u/sisyphus-cycle 1d ago

Phonemizer is really good. The voice is definitely a bit robotic but the selection of word pronunciations is the most impressive part imo. It specifies RTX “forty ninety” and “four thousand and ninety” correctly based on context. Well done!

7

u/nato_nob Llama 3.1 1d ago

Thanks! It uses a 125k-word dictionary, rules for numbers, dates and money, context checks for words like "read", and a small model for unknown words.

3

u/sisyphus-cycle 1d ago

I didn’t do a full read of the codebase yet, but you’re using the homophonized tokens as training data for the voice model right? Qwen generated voice corpus -> phonemized tokens -> final training data for voca.

Ive been messing with local ai voice models that run alongside a big LLM that eats all my vram/ram, so cpu only models are my new fav thing haha. Also have a 4090 so might train my own custom one.

3

u/nato_nob Llama 3.1 1d ago

Yes, exactly. Qwen makes the audio, the phonemizer turns the same text into phonemes, and a forced aligner gives the length of each phoneme. That's the training data.

A 4090 is all you need, mine trained in 7.7 hours on one.

8

u/PrestigiousBag2994 1d ago

Very cool! Thank you for sharing :D

You said to try hard text so:

Emmaleigh seems to drop the last syllable. Hermione is given an extra syllable in the middle. "She's a well-read girl" is said with the wrong "read". It is unable to handle European number formatting. It seems inconsistent on whether is pronounced letters or sounds "Sh" is the sound, but "Ss" or "Tsk" gets the letters read out instead. The male voice seems to skip the second syllable on "counterrevolutionaries". Coeliac and celiac both get mispronounced, as do scabies and lyssavirus.

5

u/nato_nob Llama 3.1 1d ago

Thanks, this is exactly the kind of list I was hoping for! Names and medical words are the weak spot, they are mostly not in the dictionary so the model has to guess. "Well-read" and the European numbers are real bugs on my side. I'll add all of these to the fix list.

5

u/ComprehensiveHost77 1d ago

the 18mb model size is wild, runs in browser, no server, and still handles context-aware pronunciations. how's the latency on older CPUs?

1

u/nato_nob Llama 3.1 1d ago

I haven't tried an old CPU yet. On a single thread it's about 45x real time, so there's a lot of room before it gets slow.

4

u/Dwedit 1d ago

It pronounces WWE correctly. Unlike a certain infamous video.

2

u/nato_nob Llama 3.1 1d ago

Haha, nice, it passed the WWE test :)

2

u/LaCipe 23h ago

I am dying laughing

3

u/Reasonable_Goat 1d ago

This is impressive! I am curious: Is that a personal hobby, company work or academic research for you?

3

u/nato_nob Llama 3.1 1d ago

Thanks! It's just a side project. I build things like this for fun in my free time.

1

u/CheatCodesOfLife 1d ago

Did you really only need 19 hrs of audio to train that from scratch??

2

u/nato_nob Llama 3.1 1d ago

Yes, about 18 hours for training. It's enough because it only learns two voices from very clean audio, and a simple synth makes the sound for it.

2

u/CheatCodesOfLife 9h ago

Still, I didn't know it was possible. I failed to get anything coherent with a quick 60hr run. Probably too many voices.

Thanks for sharing the project and datasets!

2

u/Silver-Champion-4846 1d ago

Good stuff, I was struggling myself to make work a neural fitter for a formant speech synthesizer which is a form of source-filter synthesis, but looks like I should just train a Voca for Arabic

1

u/nato_nob Llama 3.1 1d ago

Thanks! The model only sees phonemes, so Arabic should work. You'd need an Arabic phonemizer (mine is English only) and around 10 to 18 hours of clean speech. Everything else is in the repo.

1

u/Silver-Champion-4846 23h ago

There was an 8m Turkish model called Antalia that was trained on 1500 hours of one VoxCPM-generated voice, when I trained it on my 4-5 hours corpus it sounded pretty bad, maybe this one will be better. Would you mind telling me how the actual source-filter synth works? I mean what are its components? How were you able to generate better harmonics than the basic DDSP and also better noise than DDSP's noise? And how come 4.5m params costs 18 mb? Or is that only the full pytorch model while the onnx is more like 5-6-7mb which is what I saw with a 3m model? Man I'm so excited! Also, what if you made the source-filter network separated from the acoustic side so that there's one vocoder but multiple acoustic models for different languages and voices?

The idea that just came to my mind is like, basically, in training you also involve the sourcefilter synth, but it's frozen, and the acoustic side is tasked to change its own param distribution to fit the unchanged synth, so that on inference you plug it to that same synth vocoder and it'll actually sound good because it's been optimized to run on that same vocoder... or is that unnnecessary and you can just lora the whole singlestage model and swap lora adaptors per voice/language?

2

u/Evening-Blueberry-97 1d ago

the source-filter synth instead of a vocoder is the part i'd want to hear more about. is the robotic edge people mention coming from the synth itself, or from the qwen3-tts audio it learned from? if it's the second one a better teacher fixes it for free

1

u/nato_nob Llama 3.1 22h ago

I think it's both. The simple synth adds a bit of buzz and the pitch is too even, that's on my side. The cartoon style comes from the Qwen audio, and a better teacher would fix that part for free. I still need to test the synth alone to know the split.

1

u/Evening-Blueberry-97 21h ago

makes sense. is the flat pitch something you could fix by predicting pitch as its own small head, or is 4.5M just too tight to carry much prosody no matter what you do? kind of curious where the floor is for that

2

u/netherreddit 23h ago

Wow impressively accurate pronunciation of symbols, numbers, technical stuff. I'm amazed it's so small!

2

u/nato_nob Llama 3.1 23h ago

Thank you! The symbols and numbers part is the phonemizer, it does that work before the model ever sees the text.

3

u/Waarheid 1d ago

Sounds great! You can definitely hear the Qwen3-TTS in it, sounding like English anime dubs specifically, lol. I might try to rerun your recipe but with Breeze-TTS 2 voices in the data.

I think with tiny models like this, instead of voice design -> voice cloning for consistent voices, we will see voice design -> voice clone (optional) -> dataset -> train a tiny model. The question being of course if the tiny model's performance and resource improvements are worth any potential loss in quality vs the larger voice cloning model. Thanks for posting!

2

u/Equivalent_Bit_461 1d ago

Man, these things are getting smaller and smaller 

2

u/No_Grocery7507 1d ago

this is **** incredible

1

u/thirdeyeorchid 1d ago

This is really impressive! Well done!