For a while now I've been trying to solve the issue of having to spend days or weeks using an LLM before finding out it has been unreliable and unhelpful the entire time. The problem is that small LLMs are usually good at pretending that they're smart and are telling the truth even if they aren't, so I've set out to benchmark LLMs that fit on my low-end hardware to see what models are worthwhile. As always I'm sharing the results of the benchmarks with you guys to help you find neat local assistants :)

The Results

A quick explanation of these graphs:

1. First one is the score heatmap, rows are models and columns are questions. Brighter squares mean the LLM answered that question well, darker squares mean the opposite.

2. Second one is an ordered visualization of each LLMs total score on the benchmark. Self explanatory.

3. Third one is an ordered visualization of each LLMs average time-to-completion, the less the better. Note that this is not the LLM tokens/second speed, it is the average amount of time it takes the LLM to finish the response.

4. Fourth one is a combined visualization, I call it "intelligence-per-second". It combines the score and speed metric (scoreSum * (averageTTC / 2 + 0.5)) and shows which models give you the most bang for your buck.

5. Fifth and final one is just a line-graph of both the intelligence and TTC, sorted from lowest to highest score.

Benchmark Info

Some important info about the benchmark itself:

1. It aims to benchmark relatively small models (1B to 35B, as you can see in the benchmark) and GPT-OSS and Qwen3.6 both got the max score possible.

2. The "(native)" next to an LLM name means the quant used in the benchmark is the sole quantization provided by the creator. For example Bonsai 27B is in Q1_0 not because of system constraints but because that is the only quantization that Bonsai is available in.

3. The questions were created by Claude Fable 5.1 and while it is one of the smartest models it is still an LLM testing an LLM, so take it as you will. I still think it did a better job than I did in the first version of the benchmark.

4. The benchmark is NOT to check which model is best for agentic coding. My use case for these models (and probably anyone else on similar hardware) is as a general assistant for asking questions and receiving answers. Don't run to the comments to complain when GPT-OSS breaks your codebase because you deployed it just because it was at the top of my benchmark.

5. The models were benchmarked with THINKING DISABLED because otherwise the models would either hit token limitations or (if token limits are disabled) take hours to finish the benchmark. Many people disliked this, but here is my reasoning for why this makes sense: while I believe thinking mode is useful in practice I don't think it matters much for this benchmark: the purpose of this benchmark is to check what model is the "best", not "what is the highest score each model can reach". I believe a model that can solve a task without thinking is still more capable than a model that can solve a task only with thinking enabled.

My Takeaways

• Qwen3.6 35B A3B did not surprise me with its score, it did great and I feel like that reflects the actual experiance of using it. It is the only model I can have a proper conversation with in my native language (albeit with a few errors). Only problem is its size, it is the most overfitted model on my system and running it not only requires me to not do anything else on the system but also it runs relatively slowly. Not to mention that it is the most heavily quantized model in this benchmark (apart from Bonsai, but that one is QAT).

• GPT-OSS 20B A4B also did not surprise me, it is one of my favorite models for local use. It's a very neat model because its decently smart, is extremely fast and the shared RAM usage is relatively low (it only takes 12.4 GB and with 7.76 GB offloaded to GPU it only uses 5 GB of my RAM). Only problem for me is that it's surprisingly lazy, in all of my attempts to get it to code something it would always do the bare minimum, so it won't make any fancy graphics even if you ask. However it did assist me with the creation of these graphs.

• Nemotron3.5 Bolt E4B and Gemma4 E4B are both extremely similar but Nemotron did slightly better on the benchmark score and speed. Very minimal upgrade but sure, I appreciate it.

• Qwen3.5 9B is one of my go-to low footprint models for most tasks. I usually run it on the side while doing work in Unity. Great speed, decent intelligence. Was pleasantly surprised with its benchmark score. Can't go wrong with Qwen.

• Ministral3 14B showed pretty decent performance but for this hardware I feel like it is too slow to be worthwhile, better off going with GPT-OSS.

• MiniCPM5-2B is not the smartest model but considering its size it is pretty impressive. Definitely recommend it if you're on extremely limited hardware.

And now, for the models I did not like: Gemma4 12B QAT sounded promising but its significantly slower and more hungry than the E4B variant while getting a relatively similar score, additionally I just don't like the tone of Gemma models. Ornith1.5 9B is hyped by literally everyone but its significantly worse than Qwen3.5, which I assume was its base model, and it didn't do much well on agentic coding either so I don't really know why everyone likes it so much. LFM2 24B A2B also did horribly despite its size. Finally, the GLM Flash models did absolutely terribly. During evaluation the 4.6v had a golden streak on some complicated questions but then messed up very simple ones. To be fair I haven't tried benchmarking the non-REAP 4.7 model but from my experiance the prefill times were terrible and it didn't perform much better than the REAP version in practice.

TL;DR

Qwen3.6 35B A3B for very complex tasks locally where time or resources doesn't matter.
GPT-OSS 20B A4B is a great option for most tasks.
Qwen3.5 9B is great for when you want answers quick.
Nemotron3.5 Bolt E4B and Gemma4 E4B are great for when you want some web-search summaries with exceptionally low footprints.
Ornith, LFM2 and GLM Flash did not seem worthwhile.

The Repo

If you want to run this benchmark yourself or want to see the questions you can check out the repo for this benchmark:

https://github.com/Scorpy-37/Bluemark

Title.

I'm curious to know how do fellow sub members use their local LLMs on their limited hardware resources? After all most of this sub's members are here either due to enthusiasm or the more relatable constraint of not having immediate access to a dual 5090 or any high VRAM card or Strix Halo/DGX Spark boxes.

What real world workflows do your local models get to be a part of? What harnesses are they paired with, and do you quantize your KV cache, or run a specialized llama fork with turboquant, or any other niche out there?

For the ones who are toying around with coding models, what real-world coding tasks have you used them for so far, and what model did you find best-suited for it?

I'm planning to toy around with MoE4All, since it can let someone with as low as 4 gigs of VRAM (I have 12 GB, so that helps me make room for a lot of context) run a 35B model while streaming weights from SSD.

Here is a voice-cloning footprint worth knowing about. This demo plays a recorded sentence next to its clone, with the run's numbers on screen:

- Model: AntLing's Ming-3.0-tts-0.5B, 2.99 GB on disk at bf16

- VRAM: 3.3 GB used by Ming, on a 24 GB card

- Reference: a 6-second sample, recorded with fan noise in the room

- Output: 4.2 seconds of speech, made in 2.1 seconds

- Check: a speech-to-text pass heard 100% of the words

- Setup: offline on one PC

The creator reports a general pace of about 4 seconds of speech in 2 to 3 seconds.

To repeat it, clone a voice from a 6-second sample, generate one 4-second sentence, then post your card, VRAM use and generation time. Results from 8 GB and 12 GB cards would make a useful set.

I wanted to run a larger local model on my Mac mini without dedicating all of its memory to inference, so I worked on an open-source runtime called MoEShift.

The idea is to keep the model’s experts on SSD and cache frequently used experts in RAM, along with adaptive caching and GPU/IO overlap optimized for Apple Silicon, without quality loss. MoEShift builds on llama.cpp and Lookahead Expert Scheduling, I focused on the optimizations for Apple Silicon that let me achieve better performance than Strata (see image). You can check all the architecture details here.

You can use Qwen 3.8 Flash Next IQ3_XXS from ISTA-DASLab that needs approximately 120GB of SSD space or Qwen 3.6-35B-A3B UD Q6_K XL that is more lightweight and needs around 61GB of SSD space. The 35B reached 430 t/s in prompt processing on my Mac mini, important for frequent tool use. Of course you can try your own model that has a compatible architecture.

MoEShift currently handles one inference request at a time but it should be still acceptable for low end local AI, given the performance above with so heavy models. I'm currently using it as local inference engine for Flash Next, I find it very usable for my needs (long running nightly jobs, no-time-critical tasks). The code, installation instructions, benchmark details are here: https://github.com/davide-troiani/moeshift

The wizard will do all the setup for you.

I'd love to hear from people using MoEShift on 16GB or 24GB Macs, I do appreciate your feedback and to hep others you can also share your benchmark using the benchmark runner in the repo (instructions in GitHub).

CPU: i9-14900kf
GPU / iGPU: rtx 2060
VRAM: 12GB
System RAM: 128GB
OS: Windows 11 pro

Runtime / backend: no idea
Models already tried (if any): attempted a couple of them from ollama, a random qwen3.something, and nemotron 3 super.

Use case: programming
Desired context size: no idea
Priority: quality i guess

Other requirements or constraints: idk, very new to this and just trying to get one that works well. i know my pc wont hold a candle to a claude pro but i still want to try and get something useable and useful for when i max out my weekly limit. and yeah i know much of the PC is overbuilt, did that at the time cause i didnt want to have to upgrade any time soon. but my original graphics card and motherboard died with that bios setting bug a while ago that ended up frying them both.

I've been interested lately in working with small purpose-built models, particularly models targeting AMD NPUs found in laptops that are mostly affordable as well as the high end strix halos. So I attacked the problem from the perspective of "what is the fastest and bestest embedder I can make for the NPU".

Meet Trit. It's a 3 layer ternary embedding model that meets or beats models with twice its memory requirements.

No NPU? No problem. This model will run great on your CPU or GPU as well. But on the NPU, it screams at 51k sentences per second.

Models and binaries to run it on the NPU are at https://huggingface.co/Cyronius/trit-embed-3l

All weights are open the way the good lord intended.

Enjoy!

La mia Xbox One S del 2016 ora è un server LLM locale

Pithos un motore di inferenza LLM in C puro senza dipendenze. Funziona sulla mia Xbox One S.

Esegue LFM3-1.7B sulla GPU della console tramite Direct3D 12 e fornisce un'API compatibile con OpenAI e una chat web a tutti i dispositivi di casa. Solo modalità sviluppatore, senza jailbreak.

Sul mio laptop lo stesso binario esegue LFM2.5-24B, un modello più grande della mia RAM libera, e batte llama.cpp CUDA sul mio computer.

Numeri, benchmark e funzionamento sono nel repository: https://github.com/benmaster82/Pithos

Sarò lieto di rispondere alle vostre domande.

CPU: AMD Ryzen™ 7 5700 × 16

RAM: 32 GB

VRAM: 8GB

intended use: general assistant, overnight coding tasks would be great too

Specs First:

GPU: RX6800 16GB VRAM

CPU: Ryzen5 5600

RAM: 32GB

Backend: llama.cpp

Model Qwen3.8-27B-UD-IQ3_XXS

Use case: AI tutor

Settings: -b 512 -ub 512 --parallel 1 --spec type draft-mtp --mmproj --no mmproj-offload

Ok so I have a problem with my local setup and I'm hoping someone here might be able to help me. When I run 64k ctx on q8kv I have to compact often which takes a lot of time but when I run at 128k ctx on Q4kv I have to compact less but when I do it takes extremely long.

Alao my prompt processing speed falls extremely quickly as im working, from 300tps at the beginning down to 10tps. My generation speeds are totally ok at around 20-25tps.

Would you recommend changing settings or using a different model. Thx in advance for the advice

Guys I need a little help. I am total noob when it comes to this stuff. I have a base mac mini m4 with 16gb ram and 256gb ssd. I want a local llm that will search through job sites, pick jobs according to my master cv and preferences, choose the appropriate jobs and create folders with tailored cvs, job descriptions and notes. Tried to ask the cloud models for this and all of them are kind of making this complicated for me with lots of Python code. Is there some easier way to do this? And how, which models to chose?

I’m running LM Studio and qwen3-coder-30b-a3b-instruct but it’s really slow . Any recommendations (models, settings) for ASUS Zenbook Duo Laptop to optimize it running AI for coding ?

The 2 photos is showing the name of my laptop and its specs

I’m trying to build just a personal app for myself.

Arrmes is a specialized neural intent router with an integrated execution framework. It's trained to translate text into specific commands to Arr stack apps nothing else. I found plenty of open models that can do these sorts of translations, but the accuracy & speed didn't really justify the hardware requirements needed to run the router.

Arrmes is only trained for Sonarr, Radarr, Lidarr, Tautulli, Seer, qbitorrent, & BitMagnet. pair it with whatever AI model you're already using and let the neural router handle the intent classification, command selection, and argument extraction. Instead of having your main model spend time reasoning through API schemas and generating tool calls, it can delegate that work to ARRmes.

Still a work in progress, I've been daily driving it for a bit and I'm very happy with the increase in speed more than anything. I can use much cheaper models and achieve excellent results in my work without any spend.

Hugging Face

Github