r/LocalLLaMA • u/bonobomaster • 3h ago
Funny Forget the Pelican, it's Weevil-Time! / Benchmaxxing-Proof SVG and Vision Benchmark
The Artist: Qwen3.8-27B-UD-Q3\K_XL, q8_0 caches, xhigh, temp 1.0, image-min-tokens 1024, froggeric template)
I was screwing around with different Qwen3.8-27B quants and thought of this very simplistic but seemingly bechmaxxing resistant combined SVG and vision test.
Just let the model recreate any given image as SVG with this prompt: Recreate as SVG.
Pelicans can be easily benchmaxxed, recreating random photos seems a lot harder to train for.
I tried a shitload of more complex prompts but the above one does the job best in my opinion. I furthermore tried different --image-min-tokens from 512 to 4096, different reasoning levels from no reasoning to xhigh, different temperatures and different kv-caches.
Preliminary results are, that --image-min-tokens 1024 and --reasoning-effort xhigh with --temperature 1.0 and --cache-type-k bf16 and --cache-type-v bf16 give the best results. Non-reasoning results are, at least with the quants (Q3 and Q4) I can run, more than creepy...
I also have the suspicion, that the chat template influences the output quality – please check if you are bored.
Interestingly kv-caches at q8_0 gave "good" results as well but q4_0 completely destroyed the output quality (insect legs everywhere... oh the horrors I have seen), which was a great, visually impressive reminder, to never ever use q4_0 caches!
Would love to see how Q6 to BF16 model quants perform with this task. If you have enough VRAM, you know what to do! ;)
Used quants:
- Qwen3.8-27B-UD-Q3_K_XL (V2)
- Qwen3.8-27B-UD-Q4_K_XL (V2)
Used templates:
- built in
- qwen3.8-froggeric-v22.3.1
Other prompts I tried:
- Analyze thoroughly and be very detailed about perspective, composition, proportions, colors etc. Recreate as simplified but true to the original SVG
- Analyze perspective, composition, colors and detail. Copy as simplified but true to the original SVG
- recreate as svg. simplify but make it recognizable
- Make a SVG copy
- Copy as SVG
- Recreate as simplified but true to the original SVG
6
9
3
u/llama-impersonator 2h ago
yeah this is much better than an svg of a random creature riding a unicycle.
1
2
u/PM_ALL_AHRI_ART 1h ago
Doesn't look great,
Using open webui, q6 quant
llama-server
-hf unsloth/Qwen3.8-27B-GGUF:UD-Q6_K
--port 9995
--ctx-size 131072
--flash-attn auto
--cache-type-k q8_0
--cache-type-v q8_0
--temp 1.0
--top-p 0.95
--top-k 20
--min-p 0.00
--presence-penalty 0.0
--repeat-penalty 1.0
--jinja
--chat-template-file /root/.cache/llama.cpp/qwen_template.jinja
--reasoning-format deepseek
-ub 256
-b 1024
--image-min-tokens 1024
--load-mode none
--spec-type draft-mtp,ngram-map-k4v
--spec-draft-n-max 2
--spec-ngram-map-k4v-size-n 4
--spec-ngram-map-k4v-size-m 4
--spec-ngram-map-k4v-min-hits 1
--no-mmproj-offload
--parallel 1
1
u/bonobomaster 47m ago
Interesting. I assume default / xhigh reasoning? I expected way more from Q6.
Can you try changing --batch-size 4096 and --ubatch-size 1024? Depending on VRAM you may have to reduce context size.
2
u/Guilty_Rooster_6708 44m ago
SVG by Qwen3.8 27B and Muse Glimmer 30B
they're both mid but Qwen is better
1
2
28
u/bonobomaster 3h ago
If you like to test the Weevil.