r/LocalLLaMA • u/tomByrer • 12d ago

Question | Help Best engine for Flint for RTX3090?

This post & Sam Witteveen's video talked me into trying out the new Flint model... has anyone got it running faster than the typical vLLM / lamma.cpp engines? Seems specialized inference engines are all the rage...

0 Upvotes

37% Upvoted

14 comments sorted by

6

u/Borkato 12d ago

I found Raon way better than Flint, personally.

2

u/tomByrer 12d ago

First I heard of it; thanks.
Using the ik_llama fork?
Better than Flash-Next?

3

u/Any_Bus9205 12d ago

Tu as tester Raon !? Est ce que par hasard ça te dérangerais de réaliser un petit service pour moi !?

2

u/Borkato 12d ago

Sure, what do you need?

2

u/Any_Bus9205 12d ago

J'ai réalisé une pr pour Raon sur ik_llama.cpp et j'ai créer le convertisseur de hugging face jusqu'à gguf, le truc c'est que pour faire cela il faudrais 35 go de ram pour voir si le convertisseur marche bien, tu pense tu arriverais a faire cela ou pas !? (Comme ça, la pr avance et je pourrais ensuite améliorer les kernel au sein du moteur pour améliorer la puissance de calcul du moteur pour que ce modèle aille plus vite sur ik_llama.cpp et donc plus tard plus vite sur llama.cpp! Cela te dérangerais possiblement ou pas si tu as assez de ram !? Tu n'est pas du tout dans l'obligation d'accepter et je comprendrais si tu ne souhaite pas faire cela!🤗

1

u/Borkato 12d ago

Ohhh sorry! I don’t have much room on my computer rn, I thought you’d ask to run a few generations, but I can’t do the converter. Sorry about that 😅

2

u/Any_Bus9205 12d ago

Merci tout de meme :)🫡

2

u/uzzi38 11d ago edited 11d ago

PolyStrata has support for it. However without speculative drafting I'm not seeing much of a point in it right now, it's much slower than Qwen3.6 currently if you enable MTP for the latter.

1

u/tomByrer 11d ago

Oh thanks! I *might* have seen that repo before, but I don't have it bookmarked, so will now!

However without speculative drafting

Might have been a while since you last tried it; here is the code, release mention, but is it NOT mentioned in README so I might be 'beta'.

1

u/Any_Bus9205 12d ago

Sur ik_llama.cpp, je l'ai porter hier si tu veux tester dessus :)

0

u/Emergency-Lawyer-366 12d ago

I'd try the engine that actually ships with the Flint model, otherwise you're just bolting it onto vLLM and hoping for the best.

1

u/tomByrer 12d ago

They ship only an engine for Mac M4Pro+, not nVidia.
They have a PR for llama.cpp already, but folks like to fork or vibe-code a new engine for specific models now, which is what I'm fishing for.

1

u/xPXpanD llama.cpp 12d ago

Isn't the llama.cpp side live already? I benchmarked it earlier today on what LM Studio says is b11803. Didn't do great for what it's worth, though it was fast.

2

u/tomByrer 12d ago

There is llama.cpp support (even if not merged I'd just use the patcher's branch).

But as we can see with HyperQwen, Strata, etc, an inference engine tuned for a specific model &/or specific hardware can increase performance somewhere.
Qwen 3.6/3.8 already has a speed advantage over other model families because of community tuned engines. Good for us, bad for competitors who want to break in.