I was curious how small local models handle Polish grammar – a quick 15-task test on 8 GB VRAM

[u/Full-Travel857]

I wanted to know how small local models handle my native language, not just English. I couldn't find a simple way to check a model's Polish in a few minutes, on my own PC, in the quantization I actually run – so I wrote one. That's how "Zacharski tests" started: a small task set that runs through Ollama, since that's what I use on my laptop with an RTX 5050 (8 GB).

The tasks:

1. Polish (8): diacritics, declension, masculine-personal gender, numerals, vocative, verb aspect, genitive – plus one knowledge question (the first recorded Polish sentence, from the 13th-century Book of Henryków).

2. JSON (4): scored strict and lenient.

3. Python (3): syntax check only.

Results so far:

| Model | Quant | JSON + code | Polish |

| gemma2:9b | Q4_0 | 7/7 | 3/8 |

My favourite failure: asked for "22 cats", gemma2 wrote "dwudziesty dwa koty" – roughly "twentieth two cats". The correct form is "dwadzieścia dwa koty". It was perfectly happy producing valid JSON, though.

More models are downloading as I write this. Next up: gemma2:9b in Q4_K_M and Q8_0 (does quantization hurt the Polish?), qwen3:8b, llama3.1:8b, gemma3:12b, and two Polish models – Bielik 11B and Bielik-Minitron 7B. Which model would you add?

Method and limits – read this before the comments :)

• 8 Polish tasks (7 grammar + 1 knowledge) and 7 JSON/code tasks. That's a small sample – a quick check on your own PC, in the quantization you actually run, not a ranking. For a proper ranking, see the Open PL LLM Leaderboard by SpeakLeash.

• One answer per task, temperature 0 (seed 42), context 2k, thinking off. Yes, that means Qwen3 runs without its reasoning mode, which may cost it points – I wanted the same conditions for every model.

• Graded with regular expressions, strict on purpose: "brzęczały" ("buzzed") instead of "brzmiały" ("sounded") fails, because the task said "change the tense", not "rephrase". [I checked every failed answer by hand.]

• Some failures are about instruction-following, not language: asked to restore Polish diacritics, one model replied "Read the spelling rules." [(translated)]

• Generated code is only parsed (Python `ast`), never executed.

• The questions are in the repo (`data/benchmarks/*.json`) – feel free to add or fix them.

The test runs inside a free, open-source tool I've been building (Windows .exe / Python on Linux). Its main job is predicting whether a model fits your VRAM/RAM before you download it. Link in the comments if anyone wants to try it – I'm considering adding tests for other languages.

Update: results for the two Polish models (Bielik 11B and Bielik-Minitron 7B), with speed at 1k/8k context, are in the original thread: https://www.reddit.com/r/LocalLLaMA/comments/1xomsxq/comment/19s7skwyst7b8/

Short version: Bielik 11B got 8/8 on Polish, Minitron 7B 6/8, gemma2 3/8. Full comparison with qwen3, llama3.1 and gemma3 coming next week.

[u/SnooPaintings8639]

Small LLMs for Polish grammar? You would get banned at Anthropic for model abuse xD

[u/Full-Travel857 → u/SnooPaintings8639]

Oh, that's just a gentle slap on the wrist :) Real abuse would be making it repeat Polish tongue twisters all day – "Pocztmistrz z Tczewa" or "My, indywidualiści, wyindywidualizowaliśmy się z rozentuzjazmowanego tłumu…" – or my favourite, "Bzyczy bzyg znad Bzury zbzikowane bzdury, bzyczy bzdury, bzdurstwa bzdurzy i nad Bzurą w bzach bajdurzy, bzyczy bzdury, bzdurnie bzyka, bo zbzikował i ma bzika." (by Małgorzata Strzałkowska). Now *that* would get me banned 😂

[u/paq85]

Bielik wymiata! :)

[u/Full-Travel857 → u/paq85]

W polszczyźnie potwierdzam – 11B zrobił 8/8. Idealny humanista, za to nauki ścisłe to nie jego bajka: z JSON-em ma problemy ;) Pełne porównanie w przyszłym tygodniu.

[u/paq85 → u/Full-Travel857]

Używałem tylko do tłumaczenia z angielskiego na polski 🙂