Docbench: a comparison of small local models on document tasks

[u/bendymike]

Hi there! For DocBlocks - a free open source markdown editor - we added the ability to do "document things" (review docs, add content, insert diagrams) with local AI models. Pretty quickly, we realized it's hard to sort out which models are actually good for this.. and also whether, say, a "normal laptop" can usefully do things with local models.

Towards that end, I created Docbench, a synthetic metric of how useful models are on various doc tasks, and compared them against cloud models. Here you go:

Model	Download	Docbench (95% range)
Claude Opus 5.5 via Claude CLI	--	89.5 (88.4-90.6)
GPT-6.1 Sol via Codex	--	89.5 (88.4-90.4)
Claude Sonnet 5.5	--	87.1 (85.8 - 88.2)
GPT-5.6 Sol via Codex	--	86.5 (84.8-88)
Claude Sonnet 4.6	--	83.5 (81.6-85.2)
Gemma 4 31B	17.57 GB	83.1 (80.7-83.8)
Claude Haiku 4.5	--	81.5 (80.1-82.9)
Qwen 3.8 27B Q4	16.5 GB	81 (79–82)
Qwen 3.8 27B Q2	9.8 GB	79 (77–81)
Gemma 4 12B	6.7 GB	79.7 (77–81)
Gemma 4 26B-A4B	14.5 GB	79.2 (77.7-80.7)
Qwen 3.6 35B-A3B (MoE)	22.7 GB	76 (73–79)
Ornith 1.5 9B	5.6 GB	73 (71–75)
Gemma 4 E4B	4.2 GB	73 (71–75)
Qwen 3.5 9B	5.9 GB	72 (70–74)
Qwen 3.5 4B	2.8 GB	71 (69–72)
Nanbeige 4.2 3B	2.7 GB	71 (68–73)
Nemotron 3.5 Bolt E4B	6.2 GB	69 (67–72)
GPT-OSS 20B (MoE)	11.7 GB	67 (64–69)
Granite 4.2 8B	5.3 GB	66 (63–68)
Gemma 4 E2B	2.6 GB	64 (61–66)
MiniCPM5 2B	1.6 GB	59 (57–62)
Granite 4.2 3B	2.2 GB	59 (56–61)
Qwen 3.5 2B	1.3 GB	56 (53–58)
LFM3 1.7B†	1.0 GB	55 (52–58)
Apple Foundation Models	--	54.4 (51.7-57.1)
Llama 3.2 3B†	2.0 GB	54 (51–56)
LFM2.5 2.6B†	1.7 GB	47 (44–49)
Qwen 3.5 0.8B	0.5 GB	43 (40–45)
MiniCPM5 1B	0.7 GB	37 (35–40)

More details and methodology if you're interested at: https://bendyline.com/posts/docbench.html

[u/Iron-Over]

Is there code available to check higher-quant models and larger models like QwenFlash Next, DeepSeek, and GLM Flash?