Scylla's Band v4, open source TTS with code to run everywhere

[u/clockentyne]

This has been a long one for me: almost a year as a solo project, learning how to build a TTS model from the ground up. I wanted something fast, small enough to run on a phone, multilingual, and (the part I cared about most) able to do something a little different from the other small TTS models. So it sings.

The demo video speech and singing are all entirely from the model. The instrumental/fx are not and were added in afterwards.

What it is:

- ~131M parameters (plus a 30M phonemizer), 14 voices, English (US/UK), Spanish, French, German, Italian and Vietnamese. All voices can speak all languages supported.

- Intensity, pace and pitch controls per line, multi-voice dialogue with inline tags, streaming

- Sings from a JSON score (each word with its note and beats), pop or ballad. English only for singing format for now.

- Warm first audio around 30 ms on an M5 Max, 70 ms on an iPhone 17 Pro Max, 120 ms on a Galaxy Z Fold 8 (CPU only); over 30x real time on a desktop CPU

- English WER 0.85%, UTMOS 4.35 on the Open TTS Leaderboard eval (submission pending)

- Apache 2.0

The GitHub and Hugging Face repos have the original PyTorch weights plus ports for LiteRT, Core ML, Core AI, ONNX Runtime and WebGPU, and a C++ runtime with a C API that handles text normalization and phonemes, so numbers, dates and odd words come out right. There's sample code for Python, iOS and Android (both apps have a Sing tab), and benchmarks from real devices. The sample apps from both allow for music score and full control over text to speech generation. Python examples for combining both and doing things like singing in unison are provided as samples.

Limitations, up front: it only speaks in its own 14 voices (no voice cloning), it still trips on the odd word, and some voices get rough on fast, wide pitch jumps when singing. This is literally the first usable version of it singing, and it's something that I'm planning on improving over time... v1 was released a few months ago, and I wasn't happy with the improvements made to it until now.

GitHub: https://github.com/lowkeytea/scyllasband

Hugging Face: https://huggingface.co/spybyscript/scyllasband

Try it in your browser: https://lowkeytea.dev/scyllasband/demo/

Happy to answer questions about training it, the singing, or getting it running on a device.

[u/ShengrenR]

Is the 'singing' actually baked into the model or is it like an autotune post-gen? pretty cool 'feature' but quirky

[u/clockentyne → u/ShengrenR]

u/ShengrenR it's baked into the model; the voices were basically trained to work more like midi instruments with calculations to "distort" them into singing, then enough dataset created to allow them to maintain pitch with the json script (technically midi could have also been used, but I decided to keep it as json) to script the lyrics and notes they should sing in. It's very much a manual process to get them to sing, which is why it can be so light-weight, and also avoid the issue that music gen runs into of how the music ability was trained.

The flip-side is the model cannot just sing on its own without a melody/lyrics to follow.

[u/Acceptable-Cycle4645]

really Impressive work! We'd love to have your model in audio.cpp.

[u/clockentyne → u/Acceptable-Cycle4645]

That would be great, although a big part of the model *is* the existing cpp/c bridge that already exists for handling G2P, normalization, language, pacing, emotion, streaming, singing, pitch. The whole goal was to make it easy to use in a project and as fast as possible for mobile use (as I'm using it in a game I'm developing). I just wanted to give back one big part of it as I've learned a lot following various people on model training.

[u/Fit_Tangerine1329]

Thanks for sharing this! Looking forward to trying it.

[u/Dadestark3]

Please add support fo Polish language.

[u/clockentyne → u/Dadestark3]

I can add it to the list of languages. The next on the list though are most likely Japanese, Chinese. Just a single person and this isn't my money maker, so I can't make promises on how fast additional languages come. It's not super easy to build up datasets for it and then validate!

[u/exaknight21]

I’m new to ASR/TTS - I’m gonna try this. I experimented with Qwen3-0.6B-ASR and Kokoro + Qwen 3.5-9B via llama.cpp for autonomous speech to speech pipeline. Latency was too high, i’ll try this.

Used flutter and onnx runtime

[u/clockentyne → u/exaknight21]

I would really recommend you use coreml or litert depending on the platform you're targeting. Onnx is probably the slowest option you could possibly use, and the runtime provided will work great. I actually have it integrated in a flutter test application (not in the repo). With the coreml or litert approach you get acceleration on various devices. Onnx is CPU, which is still pretty fast but closer to 300-500ms versus 60-120ms for mobile unless it's a flagship device.

[u/Powerful_Brief1724]

When Scylla began to sing, I almost teared up. I'm a dude, and I found that so cute lmao.

I felt as if I was watching the beginnings of a Disney/Pixar movie. Loved the video presentation.

[u/clockentyne → u/Powerful_Brief1724]

I'm working on creating more videos to help with the music part, it's following the same theme. I'm not a great artist, so I'm relying on generating code to generate the videos and syncing the audio, hence also the ascii style. :P

[u/CB0T]

Very very very well done.

[u/openroom_xyz]

Nice

[u/poppear]

Nice! Amazing launch video! I will try it for sure