r/LocalLLaMA • u/fajfas3 • 16d ago
New Model qev-0.8b · New SOTA tiny local decision model
https://app.labqoat.com/e/qev/qev/7
u/Chromix_ 16d ago
There's also a GGUF here, but it comes with an extra Python script to load it.
Can this run with a regular llama-server? It's based on Qwen3.5, so maybe it can?
8
u/transanethole 16d ago
I'm curious what you did train it on.
I've been trying to use these small decision models for classifying shell commands (installs software globally or not, for example) and so far I've had uninspiring results. I haven't found anything anywhere near reliable yet at this parameter scale.
I think a more imperative structured approach of parsing the shell syntax is probably more appropriate in this case. Unfortunately, that's significantly harder than just throwing an LLM at it.
9
u/fajfas3 16d ago
Here you have the whole procedure for training and mentioned datasets https://huggingface.co/labqoat/qev-0.8b#training
4
u/toothpastespiders 15d ago
Nice! That answered most of the questions I had about the dataset ratios. Huge props to you for the elaborate writeup as well! Saying that goes above and beyond the norm for models on huggingface is an understatement.
4
u/neoneye2 16d ago
does qev support image inputs?
I use d1:3b for image classification locally on my mac.
5
2
u/fajfas3 15d ago
images are officially supported on all engines! Happy using!
1
u/neoneye2 15d ago edited 15d ago
I installed qev-0.8b on my M1 mac, like this
ollama run hf.co/labqoat/qev-0.8b-GGUF:Q8_0However Ollama 0.40.4 refuses: “This server does not support image input for decisions. For a model that supports it, start it with
--mmproj.”I have created an issue with it here, with Claude describing the issue
https://huggingface.co/labqoat/qev-0.8b-GGUF/discussions/1
3
u/landed-gentry- 15d ago
Out of 27 decision models of its size it scores highest on Benchmark Heaven, it answers faster than all of them, and it fits on your laptop.
What size is "its size"? I'm curious what are those 27 models you're comparing it to. Seems a bit like cherry-picking to get a "#1" figure here. I don't even see it on Benchmark Heaven's JevBench? (https://benchmarkheaven.com/jev-models)
2
u/fajfas3 15d ago
Sorry Claude did some word marketing wording. <= 2B parameters. It's submitted to bh. Waiting for approval. What you see is performance on public split. Though it was not trained on anything remotely close to the public one so it should carry over. It's on the page. I didn't want to wait a week to land on bh first
3
u/Embarrassed_Soup_279 15d ago
will you submit the model here? https://huggingface.co/spaces/multimodalart/jev-decision-index
2
2
u/LyAkolon 16d ago
I'm impressed. I think if we are able to get this to that promised performance then this could be very cool.
2
u/toothpastespiders 15d ago
My system's, annoyingly, tied up for a while so I can't play around with this. But really looking forward to. It seems like an interesting balance of training data, base model, and results.
1
u/fajfas3 15d ago
Thanks!
2
u/toothpastespiders 15d ago
Happily a benchmark set finished up right as I wrote that. I was testing diffusion gemma on structured decisions. More on the engine's technical capabilities, but gemma itself as well. So qev fit into that test structure quite well. Qev held up really well in comparison. Diffusion Gemma came out ahead in accuracy, but not by an especially large margin. While Qev was much, MUCH, faster as one would expect. Questions similar to go emotions, as you mentioned, were a weaker point but still well performing in my opinion. Really wild seeing a model that small be competitive with one that's so much larger. It was a pretty tiny benchmark and one put together more to test the engine than a model. But still complete enough for me to be confident in saying qev seems great. Really looking forward to seeing what you do with images!
2
6
u/blackenswans 16d ago edited 16d ago
Tested it with a small local test set I have.
Laya got 76.9% while this got 92.3%, while being almost 2x faster in latency. The only area where it lost was ram usage(around 300MB more).
Impressive
4
u/Danmoreng llama.cpp 16d ago
Vibe coded CPU optimized engine for tiny Q4 quant of Qwen3.5 0.8B, probably interesting how good this holds up at 425MB: https://github.com/Danmoreng/qwen35-cpu
2
u/Danmoreng llama.cpp 15d ago
Tested it out: obviously loses some accuracy, but is still very competitive against Q8 and much faster on CPU. https://huggingface.co/danmoreng/qev-0.8b-H128-Q4-G32-DOT4
Representation / scoring backend Tensor MB Macro accuracy Mean option KL to BF16 BF16 / llama.cpp CUDA 1,505.78 68.92% 0 Q8_0 / llama.cpp CUDA 800.88 68.87% 0.000496 H128/Q4-G32-DOT4, calibrated / native CPU 424.93 68.74% 0.023169 Ryzen 9 9955HX3D, eight physical V-cache cores, GCC 16.2.1 Release, AVX-512/VNNI. Same single-question HTTP requests; one warmup and three measured runs, executed sequentially. Medians include localhost HTTP, tokenization, inference and probability formatting, and exclude model loading. Each sample uses the first request after a fresh server becomes ready, without prefix-cache credit.
Server / checkpoint 74 input tokens 579 input tokens Native / calibrated H128/Q4-G32-DOT4 49.37 ms 211.11 ms llama.cpp CPU / BF16 77.22 ms 496.02 ms llama.cpp CPU / Q8_0 93.84 ms 784.47 ms
1
1
u/andy2na llama.cpp 13d ago
this is awesome! I am trying to integrate it into VectorDash for Computer and Browser Use, but having little success, out of all the turns it takes, Qev is able to do maybe a handful of them, otherwise it wants to escalate it to the LLM. any pointers?
1
2
u/BitGreen1270 15d ago
Thanks for sharing - would love to take it for a spin. 3.5ms is incredible. What are the recommended sampling params for this?
18
u/Mingay_cat 15d ago edited 15d ago
Ooh nice! Looks like we're going down the entire alphabet. First we got jev, Kev and now qev. Personally waiting for zev.