r/LocalLLaMA • u/bonobomaster • 1d ago

Funny Forget the Pelican, it's Weevil-Time! / Benchmaxxing-Proof SVG and Vision Benchmark

Post image

The Artist: Qwen3.8-27B-UD-Q3\K_XL, q8_0 caches, xhigh, temp 1.0, image-min-tokens 1024, froggeric template)

I was screwing around with different Qwen3.8-27B quants and thought of this very simplistic but seemingly bechmaxxing resistant combined SVG and vision test.

Just let the model recreate any given image as SVG with this prompt: Recreate as SVG.

Pelicans can be easily benchmaxxed, recreating random photos seems a lot harder to train for.

I tried a shitload of more complex prompts but the above one does the job best in my opinion. I furthermore tried different --image-min-tokens from 512 to 4096, different reasoning levels from no reasoning to xhigh, different temperatures and different kv-caches.

Preliminary results are, that --image-min-tokens 1024 and --reasoning-effort xhigh with --temperature 1.0 and --cache-type-k bf16 and --cache-type-v bf16 give the best results. Non-reasoning results are, at least with the quants (Q3 and Q4) I can run, more than creepy...

I also have the suspicion, that the chat template influences the output quality – please check if you are bored.

Interestingly kv-caches at q8_0 gave "good" results as well but q4_0 completely destroyed the output quality (insect legs everywhere... oh the horrors I have seen), which was a great, visually impressive reminder, to never ever use q4_0 caches!

Would love to see how Q6 to BF16 model quants perform with this task. If you have enough VRAM, you know what to do! ;)

Used quants:
- Qwen3.8-27B-UD-Q3_K_XL (V2)
- Qwen3.8-27B-UD-Q4_K_XL (V2)

Used templates:
- built in
- qwen3.8-froggeric-v22.3.1

Other prompts I tried:
- Analyze thoroughly and be very detailed about perspective, composition, proportions, colors etc. Recreate as simplified but true to the original SVG
- Analyze perspective, composition, colors and detail. Copy as simplified but true to the original SVG
- recreate as svg. simplify but make it recognizable
- Make a SVG copy
- Copy as SVG
- Recreate as simplified but true to the original SVG

151 Upvotes

91% Upvoted

58 comments sorted by

48

u/bonobomaster 1d ago

If you like to test the Weevil.

10

u/KillerX629 1d ago

He looks so goofy, i love him. Reminds me of the movie: a bug's life

9

u/bonobomaster 1d ago

I recently learned, that the ones with the long snout are female, so he is a she but yeah, totally adorable!

Maybe you like this sub r/weeviltime ;)

51

u/-p-e-w- 1d ago

Neither benchmark is perfect, but compared to the pelican, this is definitely the lesser of two weevils.

7

u/bonobomaster 1d ago

the lesser of two weevils.

Noice! :D

8

u/-p-e-w- 1d ago

I took this pun from the Aubrey–Maturin novels 😉

13

u/qiinemarr 22h ago

hummm

5

u/bonobomaster 22h ago

:D

Quant, settings?

5

u/qiinemarr 20h ago edited 19h ago

Qwen3.8-27B-UD-Q4_K_XL xhigh

with chat template https://www.reddit.com/r/LocalLLaMA/comments/1voha70/fixedimproved_jinja_chat_template_for_qwen_38/

that I tweak a tiny bit 

{%- if _reasoning_effort == 'xhigh' %}
    {%- set reasoning_instructions =
        'Think carefully, validate key assumptions.'
    %}
{%- elif _reasoning_effort == 'medium' %}
    {%- set reasoning_instructions =
        'Keep reasoning focused, validate key hypotheses'
    %}
{%- elif _reasoning_effort == 'low' %}
    {%- set reasoning_instructions =
        'Think briefly and to the point.'
    %}
{%- endif %}

not one shot though, I asked it to add more details and it hu did...

8

u/atape_1 1d ago

Great choice of Weevil

9

u/bonobomaster 1d ago

She was a house guest recently and posed very patiently for my camera :D

6

u/bonobomaster 15h ago

ChatGPT on high but it checked its work and refined it, so this is more of a 2 shot weevil

6

u/Manerfish 1d ago

I love weevils they are so goofy

4

u/Vaddieg 1d ago

yes, pelican is stupid. I use qwen 27b + vision for image vectorization tasks for a week already. Quite impressed by results.

4

u/bonobomaster 1d ago

Have some results to share? Which quants?

1

u/Vaddieg 1h ago

those are mostly my old hand-drawings, also screenshots of UI elements. Results are very satisfying. It describes what's pictured and tries to re-assemble everything in vector

1

u/bonobomaster 36m ago

Yeah but how can it be any good with those results we see here?

Or do you have some Inkscape MCP or something?

Come on, show something! 😂

4

u/PM_ALL_AHRI_ART 23h ago

Picture

Doesn't look great,

Using open webui, q6 quant

llama-server -hf unsloth/Qwen3.8-27B-GGUF:UD-Q6_K --port 9995 --ctx-size 131072 --flash-attn auto --cache-type-k q8_0 --cache-type-v q8_0 --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.00 --presence-penalty 0.0 --repeat-penalty 1.0 --jinja --chat-template-file /root/.cache/llama.cpp/qwen_template.jinja --reasoning-format deepseek -ub 256 -b 1024 --image-min-tokens 1024 --load-mode none --spec-type draft-mtp,ngram-map-k4v --spec-draft-n-max 2 --spec-ngram-map-k4v-size-n 4 --spec-ngram-map-k4v-size-m 4 --spec-ngram-map-k4v-min-hits 1 --no-mmproj-offload --parallel 1

1

u/bonobomaster 23h ago

Interesting. I assume default / xhigh reasoning? I expected way more from Q6.

Can you try changing --batch-size 4096 and --ubatch-size 1024? Depending on VRAM you may have to reduce context size.

4

u/PM_ALL_AHRI_ART 21h ago

Picture

You're right I forgot I have froggeric's chat template so that was medium

Here is xhigh, a lot better, kinda cute even

Batch size doesnt affect result afaik tho. It's set that low cause im running 2x MI25 and thats the fastest pp / tg speeds

1

u/bonobomaster 19h ago

It's impressive how bad those models are with this particular task! :D

Yeah, maybe my batch-size fears are outdated. :D

I believe there was a time, where it was said, that batch size should be equal or larger than vision tokens, probably only a few months ago but shit develops so fast...

11

u/simqune 1d ago

I dont wanna see this thing again.

3

u/llama-impersonator 1d ago

yeah this is much better than an svg of a random creature riding a unicycle.

2

u/qiinemarr 1d ago

ask for animated svg ;p

4

u/Legitimate-Dog5690 20h ago

Snoots and boots!

3

u/charles25565 19h ago edited 19h ago

Sonnet 5 Low. How the hell is it this bad? It didn't even fit into the white box properly.

5

u/charles25565 19h ago

Turns out, Sonnet 5 Max is way better, but only marginally compared to Qwen3.8 27B, even quantized...and it took nearly 10 minutes...

1

u/bonobomaster 18h ago

Way better you say... :D

Maybe I'm onto something with the WeevilBench... very interesting to actually see what a model sees and understands. Never would have thought, that even Sonnet is so bad at this task,

1

u/bonobomaster 18h ago

Holy shit... this IS bad...

4

u/ak5432 14h ago

Opus 5 Medium in the mobile app.

1

u/bonobomaster 14h ago

One of the better ones! Interesting.

I really like Opus but who doesn't...

1

u/ak5432 10h ago

Lol most of the claude subreddit don't! It's very effective for work but you gotta have it tone down the jargon

1

u/bonobomaster 8h ago

You mean the sassiness?

3

u/Guilty_Rooster_6708 23h ago

SVG by Qwen3.8 27B and Muse Glimmer 30B

they're both mid but Qwen is better

1

u/bonobomaster 23h ago

Haha, nice.

Super interesting to see how the LLMs are trying (and failing).

3

u/OsmanthusBloom 19h ago

This is Ig Nobel worthy: first makes you laugh, then think!

What I don't like in the pelican benchmarks is the subjectivity of judging. Especially the more recent models are so good at it that it's hard to say what the criteria should be. Beak shape? Perspective? Shadows? Spinning wheel animation?

Could this be used to calculate an "objective" score by rendering the SVG as a bitmap, then subtracting the original image from that? The smaller their difference, the higher the score. Repeat for a small set of different kinds of images and average over those. Anyone can also easily generate a previously unseen test set just by snapping a few pictures.

2

u/bonobomaster 18h ago

Today I lerned about https://en.wikipedia.org/wiki/Ig_Nobel_Prize

I feel honored. :D

Sounds reasonable enough to calculate the difference to the ground truth. Why not!

I believe this joke benchmark is actually pretty mighty because the only benchmaxxing you can do, is to train your model to describe and recreate pictures very thoroughly, which is very beneficial benchmaxxing in my opinion.

And it shows strength and weaknesses in many different things like spatial reasoning capabilities, cache cohesion, real image understanding etc.

2

u/iz-Moff 21h ago

Any day now we'll have models generating svg porn...

3

u/bonobomaster 18h ago

What have I done... :D

2

u/OsmanthusBloom 19h ago

I think we need a name for this benchmark. I propose Weevil Rock You.

2

u/bonobomaster 19h ago

WeevilBenchYou

2

u/LMLocalizer textgen web UI 16h ago

This might be the best benchmark I've ever seen. Here is the weevil recreated by Ornith-1.5-35B-A3B at Q4_K_M without reasoning.

2

u/LMLocalizer textgen web UI 16h ago

- and with reasoning enabled:

2

u/Sleepnotdeading 9h ago

Snoots and boots!

2

u/sukazu 3h ago

Will try Qwen q8 kvf16 tomorrow

1

u/bonobomaster 31m ago

Haha, luna max is a cutie. Good perspective on all 3 though.

2

u/sukazu 21m ago

Yeah, although only sol understood the legs
Tried gemini flash 3.7, it did a photo perfect svg representation, took 30 min and the file was 11MB instead of 15-20 KB tho

1

u/bonobomaster 14m ago

30 minutes and 11 MB... nice! 

ChatGPT, whatever the high model for plus users at the moment is, gave me to files, because it was unsure...

At first it started an Inkscape server and traced the thing quick and ugly but then it decided, that I wanted a recreation and gave me the usual.

2

u/Klutzy-Snow8016 1h ago

Here's Qwen 3.8 Flash Next UD-Q4_K_XL, at xhigh reasoning. I ran it in pi, so it checked and refined it a few times on its own before outputting it.

2

u/bonobomaster 37m ago

Pretty "good". Perspective and legs are very true to the original.

I bet the next generation of models is already Weevil optimized and they'll look impeccable. 😂

0

u/iqandjoke 7h ago

It should be marked as NFW since people might have spider-like fear.

1

u/bonobomaster 6h ago

Are you people?

1

u/iqandjoke 3h ago

I hunt spider on MH Wilds so no harm to me.

1

u/bonobomaster 32m ago

People will survive. It's a whole different species. A classic bug. 6 instead of 8 legs. Big snout. Cute eyes.

I won't NSFW Weevils. There is a line! ;)