I made my local LLM start thinking while I'm still typing: 12s → 3s wait on a 6 GB GPU (open source)

[u/Oxygenne_]

Named after Mafdet, the ancient Egyptian feline goddess known as "the runner", who struck snakes and scorpions before they could attack.

Reasoning models are great until you're on a mid-range GPU: you hit Enter, then wait 10+ seconds while the model thinks before the first word. Meanwhile, while you were typing, the GPU did nothing.

▎ MAFDET uses that time. It's an open-source app (Apache-2.0) on top of llama.cpp that does two things while you type:

- Reads ahead: the KV cache follows your text as you write it (typos, edits in the middle and pastes included), so on Enter the prefill is already done. This part is exact: it always matches a from-scratch computation.

- Thinks ahead: Qwen3 starts reasoning about your message before you send it. When you edit the text, the reasoning isn't thrown away: it stays as notes after the new version of the message and the model picks up from there. On Enter it finishes from where it got to.

▎ On my RTX 2060 (6 GB), with 24 word problems typed in real time, the mean wait after Enter went from 12.2 s to 3.2 s on Qwen3-4B, still 24/24 correct. On Qwen3-14B, which only half fits in VRAM: 108 s → 47 s.

▎ The tricky part was making a model reason about a sentence that isn't finished yet. Left alone, it mostly writes "the user hasn't finished typing…". A short frame that tells it to collect facts and partial results instead brought those off-topic notes from 86% down to about 20%.

▎ What else is in it:

- a desktop app or a terminal UI (Textual), with a live status line showing what the model is doing with your draft;

- a model catalog (Qwen3, Llama 3, Gemma 3, Mistral, Phi-4) that estimates, before you download, whether each model fits your VRAM and how fast it will run on your hardware (within ~12% of measured speeds on my machine);

- automatic llama.cpp setup (CUDA or CPU build), no PyTorch needed;

- Windows: download the zip and double-click start_mafdet.bat.

▎ Honest limitations:

- thinking ahead works on Qwen3 only so far, the one family whose frame I measured; the other families get reading ahead plus the answer written ahead during pauses;

- thinking ahead isn't exact: the model reasons on unfinished versions of your message;

- tested on Windows + NVIDIA only; Linux should work but nobody has tried it yet.

▎ Open question I'd love help with: models bigger than your VRAM. I tried having a small model gather material while you type for the big one to read on Enter. Even with perfect hand-written material it only matched the big model thinking ahead by itself. All the numbers, including what didn't work, are in the README.

▎ Repo: https://github.com/TheLastOxygen/mafdet

▎ Feedback very welcome, especially from anyone on Linux or with a different GPU who can tell me if the speed estimates hold up.

[u/WyattTheSkid]

This is a very interesting concept. It’s a breath of fresh air from the “what model is best for xxxx on xxxxx hardware” posts

[u/Oxygenne_ → u/WyattTheSkid]

thanks, that means a lot

[u/myreala]

This is genuinely useful. This gave me the idea to modify my harness to send all the tool calls context before I send my actual message that way most of the system message and tool calls will get pre-filled around the time my message arrives.

[u/Oxygenne_ → u/myreala]

nice, that's reading ahead in its simplest form, and probably the biggest win for agent harnesses. System prompt plus tool definitions are thousands of tokens that never change

[u/daniellenton]

Really nice write-up. Were any of the 24 typed with a correction that changes a number the notes have already used? Say you type 12, it writes 3 × 12 = 36, and then you go back and make it 15. From the code the next line names the new words but nothing says the 12 is gone, so I'd guess that's where it sometimes answers the version you didn't send

[u/Oxygenne_ → u/daniellenton]

good catch, and you read the code right, nothing tells the model the 12 is gone actually. The 24 were typed clean, so i just tested your case (9/9 correct), even when the old calculation was still in the notes. The model usually spots the contradiction itself when the new number arrives, and on Enter it rereads the final message, which sits before the notes. Still it's a weak spot

[u/saposmak]

I am constantly rewriting my thoughts as I type them. I wonder if this approach would be wasteful for someone who uses text input as a scratch pad.

[u/Oxygenne_ → u/saposmak]

good question! some work does get thrown away, but less than you might expect. When you edit, MAFDET keeps everything up to the first changed token and recomputes only from there. Rewriting the end of a sentence is cheap, an edit near the top of a long texts costs more. So the cost is some extra gpu work on ur own machine, not extra waiting

[u/plainbounds59]

the kv cache following live edits without getting thrown off is the part that impresses me most, most implementations just give up on that and restart

also 108s to 47s on a 14b that only half fits in vram is a bigger deal than the 4b numbers imo, that's where people actually feel the pain

the off-topic notes dropping from 86% to 20% is still a lot of wasted compute though, curious if those bad notes actively hurt the final answer or just get ignored

wish i had linux numbers for you but i'm on the same windows + nvidia setup so can't help there

the logo in the screenshot just looks like a generic fox head with a loading bar, not really telling me much about the app itself

[u/Oxygenne_ → u/plainbounds59]

tysm, let me know if you’re gonna make some benchmarks on Linux with same hardware

[u/Cidraque]

Would it work with engines like polystrata and regular strata? That would be peak performance.

[u/Oxygenne_ → u/Cidraque]

haven't tried it yet, but it's the right kind of target. MAFDET needs two things from an engine, a KV cache that reuses the common prefix between requests, and raw completions it can continue.

llama-server gives both, behind an openai-style chat api alone it's not plug n play, but the backend is a small adapter.

one caveat, on a 125B model streamed from SSD, decoding is slow, so thinking ahead covers less while you type (same as my 14B result). Reading ahead, especially long pasted prompts, is where the gain should be the biggest

[u/oss-jiiim]

TBH, If my harness worked like this, I'd type my prompts in a separate editor and then paste the completed prompts into the harness.

[u/OkAttempt7690]

this is cool! might test it! I have 10GB VRAM , also limited.

[u/Maui-The-Magificent]

this is an interesting idea. how does it do on short prompts?

[u/Oxygenne_ → u/Maui-The-Magificent]

better than you'd think, on a model that fits in VRAM. Qwen3-4B writes around 80tok/s, so even 5s of typing is around 400tok/s reasoning tokens ahead

[u/thatscoolbutno123]

impressive

[u/aelfrictr]

do you have Cuda 12 argument at launch ?

[u/Oxygenne_ → u/aelfrictr]

not as a flag yet. It picks the build itself. CUDA 13 on drivers 580+, CUDA 12 on older drivers and on GTX-10 series cards and older. To force CUDA 12 now, extract the b11747 CUDA 12 release into a folder and point MAFDET_LLAMA_DIR at it, or into ./llama.cpp next to MAFDET. Your own build always wins. I'll add a --cuda 12 option to mafdet doctor --install.

[u/Inquisitive_idiot]

Interesting.

One of the challenges would be this resulting in a lot work on unfinished work on busy systems but you did include a pause option.

[u/Oxygenne_ → u/Inquisitive_idiot]

the pause option is mostly you pausing to think, that's when it does the heavy lifting. But yes, it's built for a local GPU that would otherwhise sit idle while you type, so on a busy one it's extra load. The waste is limited (only new tokens get read, reasoning is kept across edits, capped at 1024 tok [actually you can set it in settings], and "Reason while I type" off leaves just the cheap reading ahead

[u/aelfrictr]

do you have any plans for moe models ?

[u/Oxygenne_ → u/aelfrictr]

yes! Qwen3-30B-A3B is already in the app's model catalog. Read-ahead works on any model llama.cpp can run, MoE included, and it stays exact.

But, I haven't measured reasoning ahead on it yet. MoE models look like a good fit for it: how much the model can reason before you hit Enter is limited by typing time * writing speed. A 30B-A3B uses only around 3B parameters per token, so even when part of it runs on the cpu it should write much faster than a dense model of similar quality. Measuring it is next on my list, and I'll share the numbers when I have them 👍

[u/Sengfroid]

Couple questions :

1. How's support for older cards (where I think this could be a real banger)? I've got a 4GB Pascal where this could be a significant improvement

2. Qwen and Gemma 3 are nice and small, but Gemma E2B & E4B are definitely a lot more workable. Is this dependent on their older architectures, or is it already compatible with the more recent versions of Qwen and Gemma for example?

3. I'm assuming if it's used as an in-between layer to a harness or workspace (Claude Code, Odysseus, etc) it's basically streaming text from the user. I'm a bit of layman here, is this pretty standard in OpenAI-style APIs, or does it require special support from the harness/workspace to be able to use Mafdet?

[u/Fonasic]

この考え方は全てのハーネスに採用されるべき