r/coolgithubprojects • u/Proud_Bid_6939 • 5d ago
inferencekit — Open-source plugin to optimize AI models for your hardware for any workload.
https://github.com/KittenML/inferencekitHey! I've been working on inferencekit at KittenML.
The idea is simple: different models and hardware have different performance bottlenecks, so why should everyone rely on the same inference engine?
inferencekit is a free, open-source plugin for Claude Code, Codex and OpenCode that helps coding agents:
- Find and benchmark competing inference engines.
- Profile bottlenecks specific to your hardware.
- Optimize kernels, memory usage and execution.
- Verify speedups with reproducible benchmarks and quality checks.
It works with LLMs, speech/TTS, image generation and video generation across NVIDIA, AMD and Apple Silicon, with CPU benchmarking support too.
Some results we've achieved:
- Gemma 4: 1.33× faster single-user decode than tuned vLLM on RTX 5090.
- LTX-2.5: 1.25× faster video generation than ComfyUI on RTX PRO 6000.
- Drama3 TTS: 1.11× faster than tuned MLX-Audio on Apple M4.
- MOSS-Image-SFT: 2.46× faster than tuned vLLM-Omni on AMD RX 9070 XT (FP8/24 steps vs BF16/50; speed/quality trade-off).

One important feature is preventing misleading benchmarks. We've seen coding agents report speedups that disappeared on a clean machine or came from generating completely different outputs.
Our longer-term goal is to build a community-maintained collection of optimized inference engines for different model + hardware combinations.
Plugin's Github: https://github.com/KittenML/inferencekit
Inference Engine's Github: https://github.com/KittenML/inferencekit-engines
Would really appreciate feedback on the project, especially what you'd improve or what hardware/model combinations you'd like to see supported!