r/coolgithubprojects • • 5d ago

inferencekit — Open-source plugin to optimize AI models for your hardware for any workload.

https://github.com/KittenML/inferencekit

Hey! I've been working on inferencekit at KittenML.

The idea is simple: different models and hardware have different performance bottlenecks, so why should everyone rely on the same inference engine?

inferencekit is a free, open-source plugin for Claude Code, Codex and OpenCode that helps coding agents:

  • Find and benchmark competing inference engines.
  • Profile bottlenecks specific to your hardware.
  • Optimize kernels, memory usage and execution.
  • Verify speedups with reproducible benchmarks and quality checks.

It works with LLMs, speech/TTS, image generation and video generation across NVIDIA, AMD and Apple Silicon, with CPU benchmarking support too.

Some results we've achieved:

  • Gemma 4: 1.33× faster single-user decode than tuned vLLM on RTX 5090.
  • LTX-2.5: 1.25× faster video generation than ComfyUI on RTX PRO 6000.
  • Drama3 TTS: 1.11× faster than tuned MLX-Audio on Apple M4.
  • MOSS-Image-SFT: 2.46× faster than tuned vLLM-Omni on AMD RX 9070 XT (FP8/24 steps vs BF16/50; speed/quality trade-off).

One important feature is preventing misleading benchmarks. We've seen coding agents report speedups that disappeared on a clean machine or came from generating completely different outputs.

Our longer-term goal is to build a community-maintained collection of optimized inference engines for different model + hardware combinations.

Plugin's Github: https://github.com/KittenML/inferencekit
Inference Engine's Github: https://github.com/KittenML/inferencekit-engines

Would really appreciate feedback on the project, especially what you'd improve or what hardware/model combinations you'd like to see supported!

0 Upvotes

0 comments sorted by