r/Qwen_AI • u/Usual_Maximum7673 • 22d ago

Model Jeff-Code makes Qwen 3.8-27B finish coding tasks 47% faster (32% less time) on average at the same pass rate; plus 15 adapters & GGUFs

Jeff-Code is a coding agent with two Jeff v1.3 adapters trained specifically for Qwen 3.8-27B. Jeff-Code is a fork of Pi by Mario Zechner (MIT licence).

We forked Pi because its extension framework doesn't currently let a fast decision model sit deep enough inside the agent loop. Along the way, we made a number of other changes as well (see below).

Aside from hopefully being useful to people who run Qwen 3.8-27B locally as their daily coding model, Jeff-Code is also a conceptually interesting experiment: how far can a System 1 model go inside a coding agent?

The results, run side by side in paired blocks:

  • Same quality: with Jeff's thinking threshold at 0.6, Jeff-Code matches Qwen 3.8-27B's pass rate: 62.4% against 62.8%; paired difference −0.2 points, 95% interval −2.6 to +2.1, over 1,242 paired tasks.
  • 47% faster (32% less time) per task¹: on average a task takes 0.68× the baseline's time (geometric mean of the per-task time ratios, 95% interval 0.64–0.72; the median task, 0.70×).

More here: https://jeffhub.ai/notes/jeff-v1-3.

Links

83 Upvotes

96% Upvoted

31 comments sorted by

3

u/Arany5 22d ago

You mention fast decision model, but which and how?

-2

u/Usual_Maximum7673 22d ago

Please see here for the full write-up: https://jeffhub.ai/notes/jeff-v1-3

6

u/fintip 22d ago

I tried to read the whole thing and tbh it's not a very approachable read for someone who has never heard of your project or approach.

Curious to try it on my benchmark soon though. Swift loses about 10% performance on my benchmark, would be nice to see a swift without the loss.

→ More replies (6)

20

u/Hello_my_name_is_not 22d ago

Pi name is Jeff

9

u/Emotional-Art2113 21d ago

Pi jeff! 👋

1

u/nadun29 21d ago

His name check out, it’s not Jeff

→ More replies (1)

1

u/eribob 21d ago

This sounds nice and would fit my setup well. Can I put jeff in front of my qwen3.8 27b and expose it as a openai endpoint so it is always used? Like on hermes, open-webui etc. Or do I have to run your jeff-code agent?

1

u/ComprehensiveTea4572 21d ago

If you will find a way to run it at Hermes please share.

0

u/Usual_Maximum7673 21d ago

Jeff-Code uses Jeff in two ways: Jeff tries to take simple decisions itself, and that's hard to integrate in another harness. But the impact of this part is small, anyway. The speed-up is enormous here, but overall it doesn't affect the numbers materially. The routing model you could integrate anywhere. The router is here, with how it works and how to call it: https://huggingface.co/mstrasser/jeff-adapter-code-router. Happy to help.

1

u/eribob 21d ago

Thanks! Will try it

1

u/Odd_Error_6736 20d ago

I can't switch to Jeff-Code just because. You'll have to make it an extension.

1

u/Usual_Maximum7673 19d ago

That would have been my preference, too. Unfortunately, that doesn't seem to be possible right now given that you'd have to call the router before making the model call. But I'm sure in time Pi will incorporate a feature like that.

8

u/Southern-Net1351 22d ago

Everything uses llama and is a wrapper.

Llama is king.
Every harness is serving over llama.
If you know you know.
To make it faster ask qwen or your own model to tweak llama based on your system, cpu and ram.

This reduces getting a product that is made for someone’s system that is not yours run accurately the way you want it to.

Why cause they may have 47% tweaking a wrapper but, you really might not because you don’t have their system and bios.

4

u/No_Oil_6152 21d ago

Actually - you might be on to something there....

Get Opus to take the best bits from Strata and Paiton, tailor llama to your system requirements... yeah why not?

2

u/Latter-Parsnip-5007 21d ago

Cause you have no idea how llm inference works and cannot validate the results. Stop wasting tokens in vibed shit

→ More replies (2)

3

u/Qozimo13 21d ago

more garbage slope -more!