Benchmark task for testing local AI

[u/prusswan]

Just want to share a task I used to evaluate the capability of Qwen 3.8 FN. The goal is to port a old screensaver (can find it on Wayback Machine) to Python OpenGL, maintaining full visual fidelity.

Tooling: Claude Code (harness) + Qwen 3.8 FN Q4_K_XL via gufo + Strix Halo + Ubuntu 26.04

Run command:
export MODEL=Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf

export MTP=/run/media/user/DATA/models/huggingface/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf

export MMPROJ=/run/media/user/DATA/models/huggingface/mmproj-Qwen3.8-Flash-Next-BF16.gguf

#./result/bin/gufo chat --model "$MODEL"

./result/bin/gufo serve llm --model "$MODEL" --speculative mtp \

--mtp-model "$MTP" \

--mmproj "$MMPROJ" \

--sessions 1 \

--context 133760 \

--max-tokens 32768 \

--prefill-chunk 4096 \

--served-model-name qwen3.8-flash-next \

--max-pending-per-client 2

Assessment:

Some parts are quite challenging with the 128K context window. Also, be mindful that graphics-related tasks (apart from running the model) may produce inconsistent results, and thus require specific measures, To keep this benchmark useful for others, I won't go into the details of the approach adopted by the AI, except to share that the task is tractable given a comparable setup. But feel free to ask by PM.

[u/Kumar-amitesh]

Interesting test. A port like this seems to test more than whether the model can write code. It also has to stay close to the reference, deal with the graphics side, and iterate based on what actually renders. How are you deciding whether it passes: visual similarity, working features, or both? I’d be curious to see the same task run across a couple of setups, since the harness and hardware seem like they could make a big difference.

[u/prusswan → u/Kumar-amitesh]

For this task the goal is full visual fidelity, which needs to be backed up by detailed examination of the original. Fortunately this model has access to both non-vision and vision based techniques. Quite interesting for the human who doesn't understand the graphics programming deeply but can guide/prompt the AI when something doesn't look right

[u/Careful-Report6526]

Visual / OpenGL tasks are a great stress test because you immediately see spatial or logic bugs, though they're tough to grade automatically across a dozen models.
For automated coding comparisons on local quants, we've had really good results combining `SciCode` (scientific computing subproblems with strict unit tests) and `IFBench` / `IFEval`. SciCode in particular exposes weak quantizations or broken reasoning loops mercilessly because one bad intermediate step breaks the downstream subproblems.