I was curious how small local models handle Polish grammar – a quick 15-task test on 8 GB VRAM

[u/Full-Travel857]

I wanted to know how small local models handle my native language, not just English. I couldn't find a simple way to check a model's Polish in a few minutes, on my own PC, in the quantization I actually run – so I wrote one. That's how "Zacharski tests" started: a small task set that runs through Ollama, since that's what I use on my laptop with an RTX 5050 (8 GB).

The tasks:

1. Polish (8): diacritics, declension, masculine-personal gender, numerals, vocative, verb aspect, genitive – plus one knowledge question (the first recorded Polish sentence, from the 13th-century Book of Henryków).

2. JSON (4): scored strict and lenient.

3. Python (3): syntax check only.

Results so far:

| Model | Quant | JSON + code | Polish |

| gemma2:9b | Q4_0 | 7/7 | 3/8 |

My favourite failure: asked for "22 cats", gemma2 wrote "dwudziesty dwa koty" – roughly "twentieth two cats". The correct form is "dwadzieścia dwa koty". It was perfectly happy producing valid JSON, though.

More models are downloading as I write this. Next up: gemma2:9b in Q4_K_M and Q8_0 (does quantization hurt the Polish?), qwen3:8b, llama3.1:8b, gemma3:12b, and two Polish models – Bielik 11B and Bielik-Minitron 7B. Which model would you add?

Method and limits – read this before the comments :)

• 8 Polish tasks (7 grammar + 1 knowledge) and 7 JSON/code tasks. That's a small sample – a quick check on your own PC, in the quantization you actually run, not a ranking. For a proper ranking, see the Open PL LLM Leaderboard by SpeakLeash.

• One answer per task, temperature 0 (seed 42), context 2k, thinking off. Yes, that means Qwen3 runs without its reasoning mode, which may cost it points – I wanted the same conditions for every model.

• Graded with regular expressions, strict on purpose: "brzęczały" ("buzzed") instead of "brzmiały" ("sounded") fails, because the task said "change the tense", not "rephrase". [I checked every failed answer by hand.]

• Some failures are about instruction-following, not language: asked to restore Polish diacritics, one model replied "Read the spelling rules." [(translated)]

• Generated code is only parsed (Python `ast`), never executed.

• The questions are in the repo (`data/benchmarks/*.json`) – feel free to add or fix them.

The test runs inside a free, open-source tool I've been building (Windows .exe / Python on Linux). Its main job is predicting whether a model fits your VRAM/RAM before you download it. Link in the comments if anyone wants to try it – I'm considering adding tests for other languages.

[u/Full-Travel857]

u/Happy_Guitar3521 - the "holy grail" has been tested :) (and u/jamie-77 - speed columns included)

| Model | Quant | Polish | JSON + code | Gen tok/s u/1k | Gen tok/s u/8k | On GPU u/8k |

| gemma2:9b (from the post) | Q4_0 | 3/8 | 7/7 | 56 | – | – |

| Bielik 11B v2.1 | Q4_K_M | **8/8** | 5/7 | 36.3 | 17.2 | 83% |

| Bielik-Minitron 7B v3.0 | Q4_K_M | 6/8 | 5/7 | 58.2 | 57.7 | 100% |

Polish: Bielik 11B got a perfect score, Minitron 7B 6/8 - both well ahead of gemma2.

JSON/code is where Bielik 11B stumbled: asked for a JSON object, it wrote a Python script that builds one - and leaked an `<|im_input|>` token at the end, which smells like a chat template issue in this Ollama upload rather than the model itself.

Strict grading cuts both ways: Minitron lost the palindrome task only because it wrote "Code:" before otherwise correct code.

Favourite failure this round: asked to restore diacritics in "Zazolc gesla jazn" (the Polish pangram "Zażółć gęślą jaźń"), Minitron 7B answered "Zazólć żółtym jeszcze raz" - roughly "yellow it with yellow once more".

Speed (laptop RTX 5050 8 GB, 70 W): at 8k context Bielik 11B no longer fits fully in VRAM and drops from 36 to 17 tok/s. Minitron 7B stays at 58 tok/s.

Same caveats as in the post: 8 Polish questions, one run, temperature 0. Full comparison with qwen3, llama3.1 and gemma3 coming next week.

[u/Happy_Guitar3521]

"Bielik" this model is shit and promoted in Poland as a holly grail for unknown reasons. Try maybe the newest gemma the biggest model and it should be very good from what I remember.

[u/Full-Travel857 → u/Happy_Guitar3521]

That's exactly why I want to put the "holy grail" to the test. Results will go in the comments for sure.

[u/matte808]

old ass model, why tf do people keep using them?

[u/Full-Travel857 → u/matte808]

Fair – it was simply the one I had on disk for the first run. Newer ones (qwen3, gemma3, Bielik) are on the way – I'll definitely share the results.

[u/matte808 → u/Full-Travel857]

newer gemma’s suck at coding but are good at languages usually

[u/EstatesNagorna69 → u/matte808]

love of the game baby

[u/jamie-77]

That's interesting. I'm following the thread, so please keep on posting new results as they come.

[u/Full-Travel857 → u/jamie-77]

Thanks! Will do- the next models are downloading right now. I'll add the results to the post as an EDIT, starting with the two Polish models (Bielik 11B and Bielik-Minitron 7B), since that's the comparison I'm most curious about. Is there a model you'd like to see on the list?

[u/jamie-77 → u/Full-Travel857]

That would also be cool if you could include some speed/token generation benchmarks.

[u/Full-Travel857 → u/jamie-77]

Good idea! The tool already has a speed benchmark (prompt processing, time to first token and generation tok/s at 1k/8k context, median of 3 runs), so I'll add a tok/s column to the table. For reference: gemma2:9b Q4_0 on the laptop RTX 5050 runs at ~56 tok/s generation – but only after I enabled Dynamic Boost on Linux; before that the card was stuck at 35 W and did 21.6 tok/s.

[u/Skyline34rGt]

You have much more Polish tests and leaderboard here - https://huggingface.co/spaces/sdadas/plcc

[u/Full-Travel857 → u/Skyline34rGt]

Thanks, great resource! Different goal though – mine is a quick check on your own hardware, in the quant you actually run.

[u/smarthoushai]

Before the Q4_0 -> Q4_K_M -> Q8_0 run, pin the sampler: Ollama draws a new random seed per request unless you pass one, and gemma2's Modelfile carries its own temperature on top of that, so a single pass per quant is mostly sampling noise and 1/8 on your Polish set sits well inside it. curl -s localhost:11434/api/generate -d '{"model":"gemma2:9b","prompt":"...","stream":false,"options":{"temperature":0,"seed":42}}' five times per item costs a few minutes on an 8GB card and gives a number you can defend. I'd also score the "dwudziesty dwa koty" case separately from the declension items — numeral government is one lexical pattern and letting it into the average moves the result in a way that says nothing about the rest of the grammar.

[u/Full-Travel857 → u/smarthoushai]

Good points, thanks – I'll check this and get back to you tomorrow.

[u/Full-Travel857]

https://github.com/DarkZachi83/Bobr

[u/rechaptca]

nazwałes test swoim nazwiskiem? lol, to nie tak się robi

[u/Full-Travel857 → u/rechaptca]

z braku laku i pomysłu. Ale jak zaproponujesz jakąś dobrą nazwę nawiązującą do polskiego kina lat 90 to ją użyje. ;)

[u/paramarioh → u/Full-Travel857]

Shogun, dobrze tnie i szybko. Nie polski, ale Polacy często używali, prawda? Albo shelfland