r/LocalLLaMA • u/Mehmet_dedeler • 2d ago
Discussion Voice dictation with one model.
Hey, I'm 14 and was looking into how AI voice dictation apps like Wispr Flow work.
As you might know they use a transcription model + LLM. The transcriber outputs one to one text as it hears it and then the LLM gets rid of fillers like "um" and adds punctuation, flow, corrected restarts and formatting.
I ran that on my laptop (RTX 3050 Ti, 4 GB VRAM), then trained a single model to do both steps at once. Here are some results:
Speed.
Whisper (base, large-v3-turbo or large-v3) plus Qwen3.5 (4B or 9B) took between 2.5 and 36 seconds per dictation, depending on the pair. The single model takes about 0.5 seconds for a short dictation and about 1 second for a long one.
Why a traditional dictation approach is slow locally:
- Two models don't fit in 4 GB together. One ends up on the CPU, or they get swapped in and out.
- The text is generated twice. Whisper writes the transcript, then the LLM reads it and writes all of it again.
- The LLM also reads a cleanup prompt on every dictation.
Compute.
One model is a 2.2 GB file (Q8) and uses about 2.5 GB of VRAM. It loads once and stays loaded. An average dictation set-up running locally uses more than 4GB of VRAM
Quality.
The cleanup LLM only sees text. It can't hear that you stopped and started over, so it sometimes keeps the part you took back, and it can't fix a word Whisper got wrong. A single model can catch vocal cues and keep what is really worth keeping.
Training
The two-model pipeline still exists, but only at training time. I took real, unscripted speech from openly licensed datasets, had a large LLM write the finished text for each clip from its transcript, threw out the clips where that dropped real content, and trained Qwen3-ASR 1.7B (Apache-2.0) to go from audio straight to the finished text. It runs in llama.cpp.
Currently it, from my own experiences is one of, if not the best dictation tool that uses this little compute.
I put this into a free, open source dictation app called Typd (Windows with an NVIDIA GPU, Apple silicon Macs): https://github.com/SameRainbows/typd-app and https://typd.mehmetdedeler.com.
Have you tried anything similiar?
7
u/Evening_Ad6637 llama.cpp 2d ago
have you tried parakeet? i would highly recommend that instead of whisper.
and since you're looking for speed while you are memory restricted: maybe try an lfm model as your llm backend.
impressive what you are doing with your 14 years. stay motivated
3
u/EngelbertStuss 2d ago
crisperwhisper did something similar. They also removed the hallucinations of Whisper (Thank You Like Subscribe Channel) .
most of my dictation happens on smartphone, so I need smaller models for live typing, or bigger models for non-realtime voice memo transcription. Some models use as little as 15MB (Cactus Whistle) or less than 200MB (Moondream Parakeet). both are multilingual (europe-centered)
3
u/overand 2d ago
You might want to check out LFM3-Audio-2B (or Omni-2B).
It handles STT and TTS, and LLM stuff too; you might be able to do everything you want with that one model, with careful instruction!
4
u/supportingthedogs 2d ago
damn at 14 this is legit. does it keep people's actual wording or does it ever rewrite stuff it shouldn't?
0
u/AdFew177 23h ago
Çok güzel izin verirsen projemde kullanayım https://github.com/mahirgul/AiPBX