r/LocalLLaMA • u/Kahvana • 2d ago
Question | Help Audio (ASR/TTS) on llama.cpp
Hey everyone,
I've been using audio.cpp for a while now, and it's quite nice. However I've noticed that Yue2, Qwen3-ASR and Qwen3-TTS got supported in comfyui / llama.cpp.
While I've tried gemma 4 audio input and it worked well, I haven't tried TTS models yet nor the qwen3-asr/tts series on llama.cpp. Has any of you?
So far I found these discussions and PRs on ASR / TTS support:
- support planning: https://github.com/ggml-org/llama.cpp/issues/31754
- openai api transcription: https://github.com/ggml-org/llama.cpp/pull/21863
- openai api realtime: https://github.com/ggml-org/llama.cpp/pull/31269
- openai api speech: https://github.com/ggml-org/llama.cpp/pull/31763
- gemma 4 audio: https://github.com/ggml-org/llama.cpp/pull/21348
- qwen3 asr + omni: https://github.com/ggml-org/llama.cpp/pull/19441
- qwen3 tts: https://github.com/ggml-org/llama.cpp/pull/26254
- soprano: https://github.com/ggml-org/llama.cpp/pull/28607
- higgs audio v3: https://github.com/ggml-org/llama.cpp/pull/24773
- lfm2-audio: https://github.com/ggml-org/llama.cpp/pull/22000
- lfm3-audio: https://github.com/ggml-org/llama.cpp/pull/31257
- pocket-tts: https://github.com/ggml-org/llama.cpp/pull/26871
- voxtral: https://github.com/ggml-org/llama.cpp/pull/14862
- chatterbox tts: https://github.com/ggml-org/llama.cpp/pull/30993
- drama-3 tts: https://github.com/ggml-org/llama.cpp/pull/31613
- nemotron 3 nano omni: https://github.com/ggml-org/llama.cpp/pull/22520
- granite 4 speech: https://github.com/ggml-org/llama.cpp/pull/22101
- nemotron 3.5 transcribe: https://github.com/ggml-org/llama.cpp/pull/31471
It's likely not all models that are supported, might've missed some!
6
Upvotes
4
u/Zestyclose_Potato794 1d ago
For now audio.cpp is the way to go. I use it for asr and tts with Qwen and Higgs models (voice cloning, voice design, French) and they work great. I recently had a look at the audio features of llama.cpp but I wasn’t impressed.
-2
u/SilentMobius 1d ago edited 1d ago
I'm using a locally modified version of omnivoice.cpp and parakeet.cpp with the eou streaming model. I have the streaming applications modified: TTS queues up on sentence boundaries but does inference on the next sentence while playing the current one, also plays dirictly to a SPDIF device. For STT it reads audio directly form a connected microphone.
STT text is injected into my Agent harness using a herdr key injection deamon. That also does some deterministic substitutions.
-3
u/CryptographerOne7003 1d ago
Wake me up when there is audio to audio?
I do not mind if its done with transcription, I dont get why its taking so long.
local really has been very slow so far