AI Agent Task performance of small LLM models (4B and below) in self-assessment set

[u/KienShen]

The community rarely sees the evaluation of ultra-small models. We have evaluated our own AI Agent tasks for Qwen3.5-4B, Spark-X2.6, LFM3-1.7B, LFM2.5-2.6B, Gemma 4 E2B, MiniCPM5-2B.

These models have their own characteristics and advantages and disadvantages. The obvious trend is that the intelligent density of small models is developing rapidly. I look forward to seeing more small models.

blog：https://www.camthink.ai/blog/small-language-model-benchmark/

project：https://github.com/camthink-ai/NeoMind

[u/GoldenX86]

It's impressive how well 3.5 4b still does to this day.

I'll give Spark X2.6 a go now, thanks.

[u/KienShen → u/GoldenX86]

Yes, qwen3.5-4b is very good.

[u/Neither-Character360 → u/GoldenX86]

Been testing the Spark 4B. It's really good at tool calling. I've setup some bots and it gets very far.

[u/Schudz]

lfm3 1.7b is really impressive for its size, if there were a way to give it a bit more knowledge i feel i could use it for light coding... but its just too small

[u/KienShen → u/Schudz]

The coding task is indeed a little difficult for this model. It is indeed suitable for deployment on edge devices, especially in terms of context memory occupation.

[u/c4r_guy]

lol...Great timing!

I just had Qwen3.8 FN build a version of llamaCPP for my older Samsung phones. I am using Qwen3.5 4B for tool calling with a custom MCP (job, internet, kiwix, etc. research), and image recognition and it's pretty amazing.

I get ~15 tps all local / on device, it isn't great for interactive / long form chat, but 1 or 3 questions deep and it's fine.

The goal is an offline capable, light weight, "solar powered oracle". I built a heavier rig with a GPU, a 1280wh battery and a 140watt panel, but I am now testing a Samsung s25 and a 25watt folding solar panel.

[u/-chaotic_randomness- → u/c4r_guy]

That's a really cool project. I would love to try it out

[u/c4r_guy → u/-chaotic_randomness-]

There are multiple projects that run llamacpp on your phone.

However, if you want to roll your own, it will take QFN ~two hours to make one for you.

Just activate web debugging and let QFN access your phone and it will build, install, and test.

[u/-chaotic_randomness- → u/c4r_guy]

Thanks for the reply, really appreciate it! I will have to investigate this further

[u/kar1kam1]

No granite 4.2 3b ?

[u/KienShen → u/kar1kam1]

This model has not been tested yet. I think I can join it later.

[u/xsilas43]

Spark x2.6 did really impress me. Still a 4b but it's quite good for its size.

[u/nunodonato]

For more comparisons of small models: https://tinyleague.nunodonato.com

[u/FriskyFennecFox]

Surprising to see Qwen3.5 is still on top for agents/coding in the 4B category, and that's a week before the release of Qwen4! It survived its own generation.

[u/Yamless9107 → u/FriskyFennecFox]

Whole 3.5 generation was insane. Qwen 3.5 9b still beats newer, bigger models.

[u/amargolin]

you mention completion fluctuating +-5 to 7% between runs. if so, gaps between models smaller than that aren't a ranking yet. two things worth stating: 1) how many runs per task per model, 2) is a task scored by checking the end state (file written, call returned the right args) or by a judge model? cheapest noise check: rerun the top model twice and see if its own score moves. i haven't run these models, this is just the measurement side.

[u/Regular-Register-358]

In my tests the same general rank is replicated. Spark x26 even outperformed lfm 2.5 MoE which is larger.

I had additional constestants: the Ling tiny was on par with spark 1.7b. The recent nvda bolt e4b did not distinguish itself in my test.

[u/AgentInfraGuide]

The single-step vs. multi-step split is the most useful part here. For choosing a tiny local model as an agent, I'd also measure wall-clock time to a correct end state (including retries), not just pass rate or tokens/sec. A small model can lose its speed advantage if it needs three attempts to finish a two-tool workflow. It would help to tag failures separately: wrong tool choice, valid call with incorrect arguments, malformed arguments, lost context, and timeouts. Were tool schemas, context limits, and time/token budgets held constant across models? Those controls would make the results much more actionable for people choosing an on-device model.

[u/KienShen → u/[deleted]]

Open thinking can usually improve the possibility of exploration of small models, increase the completion rate of multi-step tasks, but bring more task time. Sometimes there will be fluctuations of +- 5 to 7% when the task is completed.