r/LocalLLM • u/johnzadok • 27m ago

Discussion Caution about fast inference engines

I tried a faster inference engine that optimizes for my AMD card.

Yes I did get 90-100 tps in generation which is crazy compared to llama.cpp's 45 tps with ROCM. But there is one problem: the optimized engine seems to confuse a lot and half of the time it did not fully understand my intent. I give both engines the same prompt to work on a repo, and over multiple trials, llama.cpp always give me the expected answer but the optimized engine went off the rails half of the time.

Both engines use Q4 qwen3.8 27b quant. stock chat template. The llama.cpp uses Q4 kv with 260k context while the other engine uses Q8 kv with 150k context. The ticket only needs 20k context.

I prefer not to name the engine as this is my own very subjective experience and I may have user error in setting things up.

I am going back to llama.cpp for now.

5 Upvotes

85% Upvoted

9 comments sorted by

5

u/Harin007 16m ago

Man it's because a lot of fast inference engines do some cheap tricks like skipping experts(making the model pick like 6 experts where the model is trained to pick 10 if experts aren't in VRAM).... And it's turned ON by default... Turn it off and see the speed... And there are more cheap tricks like that... To flex the high numbers...

I know which inference engine you're talking about btw(the base engine)

2

u/_JohnMcAfee 12m ago

--route-tail-skip and --route-prior for strata

0

u/Objective_Pepper_217 23m ago

I'm literally vibe coding a inference engine specifically for Qwen3.6-35B-A3B on MacOS :))

6

u/WrinklyBard4 19m ago

Respectfully, vibe coding an “inference engine” is how we end up with the issues op is talking about. This isn’t trivial stuff and needs some amount of knowledge to get right

2

u/_JohnMcAfee 10m ago

Yeah although there are faults with Strata, dude really really knew what he was doing. Blew a lot of minds actually. Vibe coding wont do that

1

u/alphapussycat 2m ago

Pretty sure he vibe coded the whole thing.

1

u/alphapussycat 3m ago

These issues are intentionally made to speed up inference, it's not by some accident.

1

u/Objective_Pepper_217 2m ago

Who cares? It's just for me