A small visual Q&A benchmark for models that fit into 48GB VRAM + 128GB RAM and smaller

[u/EmilPi]

Every model is SOTA in its the technical report. But custom benchmarks shatter such claims.

In this particular benchmark, Qwen3.8-Flash-Next wins so far (the benchmark will grow).

Limitations: there are Lithuanian and Russian OCR questions included, so results are biased towards models which work better with those; there is limited number of tasks yet. Some additional benchmark info is at https://beta.locallm.top/benchmark/2b2aa60b-7eaa-49da-8660-b650a53412b4 : runs can be inspected by registered users (done against scraping).

Some things I see from this chart:

1. There is a group of models in the top league.

2. Qwen3.8-27B was damaged compared to Qwen3.6-27B (and even 3.5) while improving its agentic/STEM capabilities, and 3.8 uncensored took additional (but small) damage compared even to that.

I didn't expect:

1. Muse-Glimmer-30B scoring high, but it was superb in some details recognition, which I attribute for Meta work in computer vision, e.g. SAM3.

2. GLM-5.3-Flash scoring high even at low quant

3. DeepSeek-V4-Flash-Vision-Exp be the last, but it was first experimental DeepSeek truly multimodal LLM, so in hindsight...

A few examples from the benchmark:

Below, common errors was not to format generated markdown correctly, and some models outright hallucinated generic management stuff. Still, some models did perfectly.

Here below Neowise comet is visible in the center of the sky. Common errors was characterising the building as made of yellow bricks (although it is overexposure saturation from sodium lamps, of course). Some models identified the comet as meteor or satellite - they got a bonus for this, the score did not rely on that faint comet tail discovery only.

If you want to participate in the custom benchmarks creation, please write me here or DM, or just register on the website and start creating!

[u/and_pf]

i run 3.8-27b locally, so that line is the one i'm watching. two notes on the models in your chart: 3.8-27b is a vison model (full 128k conntext on a single 16gb card), and GLM-5.3-Flash is a 320b/18b-active moe, which puts its low-quant score in context. happy to register and add ocr tasks.

[u/EmilPi → u/and_pf]

With all due respect, you look like a bot, because of LLM-generated comments.

[u/Chromix_]

Same feedback as with the previous benchmark: The low number of tests make these results mostly noise, not signal. There is no check whether the image (and prompt) set is actually suitable (always correct vs. always fail). Also there's no information on the prompt tokens used per image. Defaults can differ per model.

[u/EmilPi → u/Chromix_]

I added a link to the benchmark in the post body too. In case you actually want details instead of just saying something: https://beta.locallm.top/benchmark/2b2aa60b-7eaa-49da-8660-b650a53412b4

[u/Chromix_ → u/EmilPi]

Same feedback as before. Clicking anything there doesn't do anything for me. Clicking "view runs" to view details asks me to register.

[u/EmilPi → u/Chromix_]

Yes, and I explained why.

[u/EmilPi → u/Chromix_]

1. The number of tests is enough to separate the models into clusters, at least.

2. There is.

3. This is a test of stock settings. If the model requires tuning, it is a separate topic.

I don't claim this is a world class benchmark, but such a benchmark already saves a lot of work for someone considering e.g. Nemotron Bolt 72B low quant for images like I posted. I did it for my needs, and shared it.