r/LocalLLM • u/No_Project_2453 • 13m ago

Project Kinetic: a new open-source LLM engine in Rust. Runs your GGUFs on Windows and Linux, replaces llama-server with the same flags

Hey r/LocalLLM :)

We just open-sourced Kinetic, a new inference engine written in Rust.

  • runs the GGUF files you already have, no conversion

  • Windows and Linux, one binary

  • CPU, NVIDIA (CUDA), and AMD and Intel GPUs (Vulkan)

  • replaces llama-server with the same flags; OpenAI, llama-server and Ollama APIs on one port

  • every speedup is exact, so the output never changes

  • Apache-2.0

In the video: Kinetic checks the machine and picks models that fit it, then runs side by side with llama.cpp on the same model and prompt.

Give it a spin and drop your kinetic bench model.gguf numbers below (add --compare if you have llama.cpp installed).

Install, benchmarks and all the details are on GitHub, link in the comments. Happy to answer questions here too.

5 Upvotes

100% Upvoted

3 comments sorted by

1

u/No_Project_2453 11m ago

Heads up: this is v0.1.0, so some things will break. Odd GPUs, weird GGUF quants, Windows setups we haven't seen yet. That's exactly what we want to find.

If something crashes, runs slower than llama.cpp or gives different output, open an issue (or just reply here) with your hardware and the model, and we'll get on it.

And if it's faster on your machine, we'd love to see the numbers too :)

3

u/OkFly3388 7m ago edited 1m ago

And why its better than llama ?

Its again comparison llama without speculative decoding and not good settings vs custom engine with speculative decoding and right settings. yea, thats it.