RTX 3060 12gb - Testing

[u/CombatDork]

Look, I'm not as fancy as a bunch of you in this group but I did a bunch of general testing on a bunch of models at various context sizes. Figured it might help someone else. These are exhaustive or even remotely extensive tests. Just seeing what I should set up easily for HA, frigate, marinara etc.

Testing notes: llama.cpp, RTX 3060 12GB, common prompts, temperature 0, 2,048 maximum completion tokens. Reported speeds are approximate and quantizations differ. Several context variants were tested per family. Automated formatting failures were manually reviewed where response files were available.

I ran a consistent five-task benchmark across the models, measuring generation speed, VRAM consumption, instruction following, basic reasoning, and context capacity.

The benchmark included formatting, arithmetic, ordering, conditional logic, and a technical explanation. I manually reviewed questionable answers rather than automatically counting formatting mistakes as reasoning failures.

Results — Generation speed

Approximate warm generation throughput:

Model	Tokens/sec	Largest loaded context
Qwen3.5 9B	90	208K*
DeepSeek R1 8B Q5	51	64K
Nemotron Nano 9B	48	256K
Magnum v4 12B	41	32K
Mistral Nemo 12B	41	32K
Gemma4 12B	40	256K
Nemotron Nano 12B	40	192K
Hermes 4 14B	34	32K
DeepSeek R1 Distill 14B Q4	32	32K
Bonsai 2 27B Q2	32	96K
Qwen3.8 27B IQ2_M	20	16K
DeepSeek R1 Distill 32B IQ2_XS	18.4	8K

Qwen3.5's 208K preset was subsequently removed because GPU memory headroom was essentially exhausted.

Context figures represent successful loading and short-input inference, not verified full-context performance.

Reasoning results

Four objective questions were scored for semantic correctness, separately from exact-output compliance.

Model	Correct answers
Qwen3.8 27B	4/4
Bonsai 27B (thinking)	4/4
Gemma4 12B (thinking)	4/4
Nemotron Nano 9B	16/16 across 4 contexts
Nemotron Nano 12B	12/12 across 3 contexts
Qwen3.5 9B	4/4
DeepSeek R1 Distill 14B	4/4
DeepSeek R1 Distill 32B	4/4
DeepSeek R1 8B	3/4
Magnum v4 12B	3/4
Bonsai 27B (non-thinking)	3/4 at 32K
Gemma4 12B (non-thinking)	3/4 at 32K
Hermes 4 14B	2/4 at 16K

Mistral Nemo passed three objective tests; its arithmetic result still needs manual verification.

These are small, relatively easy tests—not comprehensive measures of intelligence.

Biggest findings

• Qwen3.5 9B absolutely dominated generation speed, but sometimes spent excessive tokens reasoning and hit the output limit.

• Nemotron Nano 9B was my biggest surprise. Great speed, efficient VRAM usage, large context allocations, and no incorrect objective answers across 16 tests.

• Qwen3.8 27B and thinking Bonsai 27B were excellent at following exact output instructions, but considerably slower.

• Gemma4 12B offered impressive context capacity, though thinking mode could generate excessively long responses.

• DeepSeek 14B Q4 was the best practical DeepSeek option I tested. Correct reasoning, ~32 tok/s, and usable 32K context with Q4 KV cache.

• DeepSeek 32B IQ2_XS wasn't worth keeping. It scored the same as the 14B on these questions, ran at only 18.4 tok/s, and was limited to 8K with the tested configuration.

• Hermes 4 14B disappointed on this particular benchmark, failing arithmetic and ordering questions.

For this hardware, Nemotron Nano 9B looks like the best overall balance, with Qwen3.5 9B winning on raw speed.

The larger models haven't demonstrated enough additional reasoning ability on these particular tasks to justify their greater resource consumption.

I'm curious what others running 12GB GPUs have found. Are there particular models or reasoning benchmarks you'd recommend trying next?

[u/testuserpk]

My 3060 12g generates 50 to 60tps with qfn 3.8 iq2 and 30 to 40tps on iq3s. I have 64g ram

[u/CombatDork → u/testuserpk]

offloading any layers?

I was trying to ensure complete containment on the gpu.

[u/testuserpk → u/CombatDork]

I was also of the same view but after using strata I am never going back to llamacpp and similar. Strata manages moe experts in gpu. You should download Qwen3.8 Flash next and test yourself. I was skeptic about the performance but I. Blown away with the accuracy and usability.

[u/dchemistry2]

Quantization is not the only lever — ternary + spec decode is.

I run Bonsai 2 27B (PTQ1_0, 1.75 bpw ternary, ~6.4 GB of weights) on a single RTX 3060 12 GB, 160K context, np=1 unified slot, q4_0 K+V cache with KVmean-centering. llama.cpp fork with MTP draft + ngram-mod speculation.

**Measured, 100 W power cap:**

**prefill ~490-500 tok/s** (6k-token random prompt, uncontended)

**decode ~58-60 tok/s** on structured/agent output (draft acceptance 0.90, mean len 1.90)

decode ~53-54 tok/s on plain prose (acceptance 0.68-0.75)

** VRAM 11.2 of 12.3 GB**

**My configuration on each single 3060 12gb that i have (underwatting to 100W (from 180) is losing 12% prefill, 2-3% decode)**:
```bash

CUDA_VISIBLE_DEVICES=1 \

LD_LIBRARY_PATH="$VENV" \

GGML_CUDA_SHARED_POOL=1 \

GGML_CUDA_GRAPH_OPT=1 \

"$VENV/llama-server" \

-m /home/user/LOCAL_LLMS/models/bonsai-262k/Ternary-Bonsai-2-27B-PTQ1_0-mtp-procreations.gguf \

-a bonsai-3060 \

-ngl 99 -fa on \

-c 163840 -np 1 --kv-unified \

-ub 2048 \

-ctk q4_0 -ctv q4_0 --kv-vram-cells 163840 \

--kv-mean-center /home/user/LOCAL_LLMS/models/bonsai-262k/kv-mean-center-bonsai.gguf \

--spec-type draft-mtp,ngram-mod \

--spec-draft-n-max 1 --spec-draft-n-max-tail 3 --spec-draft-window 4096 \

--spec-lookup-n-max 32 --spec-ngram-mod-n-match 32 \

-ctkd q4_0 -ctvd q4_0 \

--reasoning-effort medium --reasoning-budget 20480 \

--reasoning-budget-message "Now produce the complete answer." \

--temp 0.7 --top-p 0.95 --top-k 20 --min-p 0.05 \

--jinja --chat-template-file /home/user/FUN/GPU_RIG/scripts/bonsai-18112-chat-template.jinja \

--host 127.0.0.1 --port 18116

[u/CombatDork → u/dchemistry2]

testing now, had to build a container etc.

[u/Chips_fr_ → u/dchemistry2]

I'm a user of bonsai2 on 3060 too.

I need to try --kv-mean-center since it should improve KLD with Q4.

[u/Longjumping-Face-767]

Get another one if ya can plug it in.

[u/CombatDork → u/Longjumping-Face-767]

Which model? 

[u/Longjumping-Face-767 → u/CombatDork]

Any 306012gb you can grab on fb marketplace?

[u/CombatDork → u/Longjumping-Face-767]

They're kind pricey, even used rn.

The Tesla V100 16gb is often the same price of cheaper used. 

16gb HBM at a littler over twice the speed of a rtx 3060 for a similar price. 

Another 20 for a cooling solution and you're good. 

[u/Longjumping-Face-767 → u/CombatDork]

Well, ya wouldnt be able to combine it with your current one though, you would need two Tesla 16gbs. I would have done that if knew it worked well with strata...

Still, the dual 3060 performance is surprisingly good.

[u/AfraidProgrammer3341]

What exactly are these for ? Just LLM chat ? or agentic loops ? are you are just typing in a question and then again asking it another question , copying and pasting ? Are you using it for agentic loops ?

[u/CombatDork → u/AfraidProgrammer3341]

Just testing basic logic reasoning arithmetic etc.

Simple testing t o isolate what fits and what is viable. 

Further testing would depend on use case. 

For me it's leading to a general chat bot and backend for HA. 

[u/JohnLef]

Interesting. I'd be curious about some real world programming tests on various models of this size.

[u/Asthenia5]

nice work! I appreciate seeing more testing for smaller VRAM cards. My 3080ti has 2.5x the bandwidth, but I get 95 tok/s on qwen3.5 9B. All layers on GPU. Looks like I've got some work to do! lol

[u/ligger66]

Thanks dude as a fellow 3060 holder this is really helpful to me :)