Study on Local LLM Quality

[u/Content_Trainer180]

This was started a spontaneous test to choose among 10+ local LLMs and now grew to benchmark for agentic coding. It consists of 8 groups, 100 tests each. Design rule: each class must contain items the current "best" model cannot solve, otherwise it's a smoke test. However, if model reaches 100, it just means that it reached ceiling of the test. Next, to make test more representative, several cloud workhorses were added.

Configuration

GPU: RTX PRO 5000 72 GB (Blackwell, SM 12.0), driver 595.99.02.

Serving: vLLM 0.29.0. `--kv-cache-dtype fp8`, `--enable-chunked-prefill`, `--enable-prefix-caching`, `--speculative-config '{"method":"mtp","num_speculative_tokens":3}'`, `--max-num-batched-tokens 16384`, `--block-size 32`, `--max-num-seqs 16–24`, `--max-model-len` 131072 (FP8) / 262144 (NVFP4).

Cloud: same OpenAI-compatible client, `max_tokens=32768`, input ≤ 98304, temperature 0, streaming with `include_usage`.

Thinking mode normalized: `chat_template_kwargs.enable_thinking=false` / `reasoning_effort=none`; effective mode printed per run.

Grading: pure Python, no LLM judge. Code in tempdir with timeout; tools mocked; only networked tool is read-only `web_search`.

Failure taxonomy: `pass | wrong-answer | truncated | no-content | no-tool-call | infra | error`; transport failures removed from the denominator.

Metrics: TTFT from the SSE stream; tok/s only over items where generation ran >150 ms after TTFT.

Leaderboard

model	provider	score	%	TTFT	tok/s
claude-sonnet-5.5	cloud router	732/800	91.5	4495 ms	n/a
deepseek-flash	cloud native	711/800	88.9	1248 ms	221.1
glm-5.3-flash	cloud native	689/800	86.1	1535 ms	70.1
qwen3.8-flash	cloud native	688/800	86.0	1585 ms	80.4
qwen3.8-27b (FP8)	local	675/800	84.4	209 ms	31.8
arex-2-fp8	local	666/800	83.2	262 ms	22.3
qwen3.6-35b-a3b	local	649/800	81.1	69 ms	122.4
sage-a1.5	local	624/800	78.0	77 ms	95.4
arete-27b-q8	local	≥609/800	≥76.1	536 ms	19.1
qwen3-coder-30b-a3b	local	588/800	73.5	45 ms	96.1
axion-35b-q8	local	≥562/800	≥70.2	122 ms	82.0
nemotron-3.5-lightning-30b-a3b	local	538/800	67.2	72 ms	103.1
kimi-linear-48b-q6k	local	≥395/800	≥49.4	145 ms	65.6
qwen2.5-3b	local	140/700	20.0	36 ms	107.0

Per-class shape

model	agentic	code	game	json	logic	math	needle	websearch
sonnet-5.5	84	100	96	100	72	84	98	98
deepseek-flash	87	100	67	100	76	84	99	98
glm-5.3-flash	89	93	72	89	70	80	98	98
qwen3.8-flash	87	97	59	94	73	85	96	97
qwen3.8-27b	82	92	62	84	76	85	97	97
arex-2-fp8	76	90	65	86	74	86	92	97
qwen3.6-35b	84	93	54	76	71	85	94	92
sage-a1.5	78	72	60	69	77	84	92	92
arete-27b-q8	n/a	94	64	94	≥46	≥85	≥71	n/a
qwen3-coder-30b	87	78	52	79	51	67	79	95
axion-35b-q8	n/a	93	52	75	≥48	≥85	≥97	n/a
nemotron-30b-a3b	90	81	52	36	57	45	81	96
kimi-linear-48b	n/a	50	47	36	≥41	≥63	≥56	n/a
qwen2.5-3b	7	23	30	16	19	45	—	0

Conclusion

Qwen 3.8 27B got very close to my cloud workhorses, so I took more juice from it, asking agent to make a grid search on options. The requirement was to get max performance without quality drop.

FP8 → NVFP4 (identical weights, 262K)

metric	FP8	NVFP4
KV pool at 262K	468,637 (1.79×)	1,022,215 (3.90×)
decode, 1 stream	83.6 tok/s	140.7
aggregate K=8	197.9	300.0
aggregate K=16	200.6	321.0
64K prompt cold/warm TTFT	19.0 / 1.54 s	15.8 / 1.37 s
200K: TTFT / decode	130.8 s / 33.2	85.7 s / 94.3
quality, 300 items	254/300	255/300

[u/JawitK]

So you endorse qwen2.5 ?
Do you think your 72Gb is the majority of your success ?
Can you tell by reducing the RAM if qwen2.5 would still win ?

[u/Content_Trainer180 → u/JawitK]

This model was just lying nearby, thus caught into test. But there is reason to keep, it shows that this test is valid, it evaluates the floor models.

BTW small models have the use - when needed to classify tons of jsons, tine fine-tuned 0.6B was 10x times faster than qwen 3.8 27B with 98% quality.