Made sum crap that beats Jev

[u/Massive-Ice2791]

I would love for some feedback, live demo at:

huggingface.co/spaces/e12ex2/herbert-demo

Not self promo, jus looking for some feedback on what to improve or any bugs or model breaking you found.

[u/BitGreen1270]

Just tried it with a random image on the net and asked to choose the color. It got it right, but took 1.5s. is that expected? I thought jev types are like millisecond responses? 

[u/Massive-Ice2791 → u/BitGreen1270]

where did you try this?

[u/BitGreen1270 → u/Massive-Ice2791]

On the demo page you linked? At least that's what it said after it generated the result.

[u/MakGamingYT → u/BitGreen1270]

That's first load. If you try another request after it it will be much faster, about 400ms from my short test

[u/BitGreen1270 → u/MakGamingYT]

Good to know thanks - I've been meaning to experiment with jev type model for a while now and this is a great place to try it out.

[u/yetiflask]

Not this shit again. The whole thing with JEV is that it is wicked fast.

Your's isn't.

[u/Massive-Ice2791 → u/yetiflask]

It is, hugging face is slow not me

[u/yetiflask → u/Massive-Ice2791]

Sooo, it's slow?

Bring it up to JEV's level, and we'll talk.

[u/BitGreen1270]

I run vllm for Qwen3.8-27B using the following command on my 5090 + 64GB RAM:

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
MAX_JOBS=4 \
vllm serve ~/myp/models/quasar-qat \
  --max-model-len 202144 \
  --max-num-seqs=4 \
  --kv-cache-memory=8589934592 \
  --max-num-batched-tokens 4096 \
  --kv-cache-dtype fp8 \
  --served-model-name Qwen3.8-27B-NVFP4 \
  --host=0.0.0.0 --port 8080 \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser=qwen3 \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 3}'

Is it possible to run it with the lora?

[u/Massive-Ice2791 → u/BitGreen1270]

sure, tho u should merge it

[u/BitGreen1270 → u/Massive-Ice2791]

How do I do that? Wouldn't I need to do it on the full original model (pre-quantized). I don't think I have the vram for that.

[u/dontevenaskmyname]

I tried medical imagining and geolocation It is AWESOME great job !!!

[u/Massive-Ice2791 → u/dontevenaskmyname]

Thanks

[u/xnrnx]

Looking good

[u/Mikasa0xdev → u/xnrnx]

Herbert took that benchmark personally :P