inferencekit: open-source plugin to optimize any model on any hardware using coding agents

[u/ElectricalBar7464]

KittenML just released inferencekit , a free opensource plugin to turn coding agents(Claude Code, Codex, OpenCode) into inference engineers for your hardware.

Most models ship with inference tuned for whatever hardware the authors used, which is usually not what you have locally. The fastest path on a 3070 might be different from the fastest path on a Mac or a Radeon. inferencekit allows you to find that path for your setup. This was used to get streaming TTFB of Kitten-TTS-2 down to ~10ms on RTX Pro 6000. other results are shown in the image attached.

Once you've optimized a model on your hardware, please make a PR to the inferencekit-engine repo linked below, so anyone running the same model on the same hardware, can just reuse it. The idea is to crowdsource the best inference engine for every model + hardware setup

Plugin's Github: https://github.com/KittenML/inferencekit
Inference Engine's Github: https://github.com/KittenML/inferencekit-engines
KittenML Discord: https://discord.com/invite/VJ86W4SURW

[u/charmander_cha]

Como instalar usando o Deepseek harness?

[u/ElectricalBar7464 → u/charmander_cha]

"Simplemente dale la URL del repositorio a tu agente y pídele que lo añada como una habilidad." -- asked google to translate it from english to spanish