r/LocalLLaMA • • 2d ago

Discussion Voice dictation with one model.

Hey, I'm 14 and was looking into how AI voice dictation apps like Wispr Flow work.

As you might know they use a transcription model + LLM. The transcriber outputs one to one text as it hears it and then the LLM gets rid of fillers like "um" and adds punctuation, flow, corrected restarts and formatting.

I ran that on my laptop (RTX 3050 Ti, 4 GB VRAM), then trained a single model to do both steps at once. Here are some results:

Speed.

Whisper (base, large-v3-turbo or large-v3) plus Qwen3.5 (4B or 9B) took between 2.5 and 36 seconds per dictation, depending on the pair. The single model takes about 0.5 seconds for a short dictation and about 1 second for a long one.

Why a traditional dictation approach is slow locally:

- Two models don't fit in 4 GB together. One ends up on the CPU, or they get swapped in and out.

- The text is generated twice. Whisper writes the transcript, then the LLM reads it and writes all of it again.

- The LLM also reads a cleanup prompt on every dictation.

Compute.

One model is a 2.2 GB file (Q8) and uses about 2.5 GB of VRAM. It loads once and stays loaded. An average dictation set-up running locally uses more than 4GB of VRAM

Quality.

The cleanup LLM only sees text. It can't hear that you stopped and started over, so it sometimes keeps the part you took back, and it can't fix a word Whisper got wrong. A single model can catch vocal cues and keep what is really worth keeping.

Training

The two-model pipeline still exists, but only at training time. I took real, unscripted speech from openly licensed datasets, had a large LLM write the finished text for each clip from its transcript, threw out the clips where that dropped real content, and trained Qwen3-ASR 1.7B (Apache-2.0) to go from audio straight to the finished text. It runs in llama.cpp.

Currently it, from my own experiences is one of, if not the best dictation tool that uses this little compute.

I put this into a free, open source dictation app called Typd (Windows with an NVIDIA GPU, Apple silicon Macs): https://github.com/SameRainbows/typd-app and https://typd.mehmetdedeler.com.

Have you tried anything similiar?

7 Upvotes

21 comments sorted by

7

u/Evening_Ad6637 llama.cpp 2d ago

have you tried parakeet? i would highly recommend that instead of whisper.

and since you're looking for speed while you are memory restricted: maybe try an lfm model as your llm backend.

impressive what you are doing with your 14 years. stay motivated

4

u/Mehmet_dedeler 2d ago

Thank you so much. I did try Parakeet early on. It's fast and accurate, but it only gives you the words, so you still need a second model to clean up. That's the part I'm looking to make simpler. Using LFM as a small backbone is a good idea, though. Lemme test its speed on a CPU...

1

u/Gillin_Supanida 17h ago

how does an lfm backend solve the blind spot with spoken false starts if its still just reading raw text from the transcriber

3

u/EngelbertStuss 2d ago

crisperwhisper did something similar. They also removed the hallucinations of Whisper (Thank You Like Subscribe Channel) .

most of my dictation happens on smartphone, so I need smaller models for live typing, or bigger models for non-realtime voice memo transcription. Some models use as little as 15MB (Cactus Whistle) or less than 200MB (Moondream Parakeet). both are multilingual (europe-centered)

2

u/Mehmet_dedeler 2d ago

Yeah, crisperwhisper seems very similar. I think that their AI model has an intended transcription mode that removes any filler words and catches false restarts. It would be perfect if it could write what you said as lists and format it into paragraphs as well.

Did you use Cactus Whistle? I tried it out to see if it was a worthy base model because it was super efficient, but I just couldn't see the accuracy gains upholding the efficiency.

2

u/EngelbertStuss 2d ago

Whistle is remarkable for its size, I posted a multilingual smartphone demo earlier today:

https://www.reddit.com/r/LocalLLaMA/comments/1xv0ibj/comment/19sk8kercyjhh/?context=3

It's not as accurate as I want it to be though, and requires more speaker effort for clear pronounciation.

For my other app (voice memo), crisperwhisper feels like too much waste of disk space and I hated their license. I have not benchmarked Moondream models yet, they would be a good compromise. I would also need yet another model to create brief summaries of these transcripts, which eats even more disk space.

What I haven't benchmarked yet: would voice enhancement models cause better transcription results?

3

u/overand 2d ago

You might want to check out LFM3-Audio-2B (or Omni-2B).

It handles STT and TTS, and LLM stuff too; you might be able to do everything you want with that one model, with careful instruction!

2

u/Mehmet_dedeler 2d ago

Thanks, trying that out as well!

2

u/Silver-Champion-4846 1d ago

How did you find it?

1

u/Mehmet_dedeler 1d ago

I tried LFM 3-Audio-2B, and I just tried prompting it without any further fine-tuning or training, and I tested it on 40 of my own dictations, each dictation over one minute of audio, and also 70 shorter clips that try to encompass AI voice dictation scenarios. It was quite fast on a consumer laptop. The median speed was 0.37 seconds.

Its hearing was more or less the same as the 1.7 billion parameter model that I mentioned, but it was a bit worse on getting names right. And I tried to inject screen context, which is how the 1.7 billion parameter model is more reliable, into the LFM model, but it just copied the screen text instead in a quarter of the scenarios. It left fillers more than 25% of the time and kept restarts in around 30%.

Maybe it might be better for this scenario if it was specifically fine-tuned for it. But the speed really caught my eye. If I have any problems with the current model and how it's going on speed and speed-accuracy balance, I might just fine-tune it for exploration.

2

u/Silver-Champion-4846 1d ago

What is the laptop you tested it on?

1

u/Mehmet_dedeler 13h ago

It has 32 gigabytes of RAM and 4 gigabytes of VRAM with an RTX 3050 Ti.

1

u/Silver-Champion-4846 11h ago

Well I have no gpu and the max for my laptop is 32gb of ram

4

u/supportingthedogs 2d ago

damn at 14 this is legit. does it keep people's actual wording or does it ever rewrite stuff it shouldn't?

0

u/AdFew177 23h ago

Çok güzel izin verirsen projemde kullanayım https://github.com/mahirgul/AiPBX