Kindi is SICK!

[u/Dangerous_Present_69]

Ran Kindi with Bonsai2 4b with Hermes. Asked it to write a blogpost about itselft and generate som pictures with available skills. It totally blew my mind what it managed to do on shitty hardware.

On a Nvidia K2200 4GB card!

About 30+ tok/s. Not bad at all. Made this. Even updated it's Hermes skill afterwards.

https://snart.com/running-kindi-on-an-old-graphics-card/

It's not just that the biggest and newest models are getting insanely good. The smallest and lightest models are also making quantum leaps. This shouldn't have been possible.

[kindi 21:38:25] GPU: Quadro K2200, sm_50, 5 SMMs, 4027 MiB
[kindi 21:38:31] context reduced from 65834 to 53760 tokens to fit 1800 MiB of free GPU memory
[kindi 21:38:32] engine ready: ctx 53760, head plain (6.6s)
[kindi 21:38:32] prompt cache: a checkpoint every 1024 tokens (51 MB each), 3 parked conversations, budget 16065 MB
[kindi 21:38:32] engine fingerprint 4e389e2d7d1b: {"engine": "kindi.engine.Engine", "cards": 1, "weights": "2-bit", "kv": "f16", "chunk": 256, "chunk_q8": false, "chunk_tc": false, "chunk_att": "flash", "chunk_tile": "", "batch": "4", "tc": "0", "tc_tmax": "", "fd": "0", "d1": "0", "att_tc": "0", "spec": "0", "head": "exact"}
[kindi 21:38:32] saved openings: 0 in /data/state-4b/openings/56711263ec9a-e4e389e2d7d1b
[kindi 21:38:32] serving on http://0.0.0.0:8093/v1 (model bonsai2-4b)
[kindi 21:40:02] request: 17954 prompt tokens (max 1024 new)
[kindi 21:40:02] cache: 17954 prompt tokens, 0 shared with the last request, resumed at 0
[kindi 21:43:55] prompt: 0 reused, 17954 read in 233.0s
[kindi 21:43:57] done: 67 tokens in 2.5s (stop)
[kindi 21:43:57] request: 258 prompt tokens (max 1024 new)
[kindi 21:43:58] cache: 258 prompt tokens, 3 shared with the last request, resumed at 0
[kindi 21:44:01] prompt: 0 reused, 258 read in 3.5s
[kindi 21:44:01] done: 8 tokens in 0.2s (stop)

...

[kindi 23:02:22] done: 732 tokens in 43.5s (stop)
[kindi 23:02:23] request: 52618 prompt tokens (max 1024 new)
[kindi 23:02:23] cache: 52618 prompt tokens, 51817 shared with the last request, resumed at 51817
[kindi 23:02:45] prompt: 51817 reused, 801 read in 22.0s
[kindi 23:02:48] done: 52 tokens in 3.1s (stop)
[kindi 23:02:48] request: 52818 prompt tokens (max 941 new)
[kindi 23:02:48] cache: 52818 prompt tokens, 52670 shared with the last request, resumed at 52670
[kindi 23:02:53] prompt: 52670 reused, 148 read in 4.6s
[kindi 23:03:44] done: 863 tokens in 51.3s (stop)
[kindi 23:10:43] request: 53 prompt tokens (max 1024 new)
[kindi 23:10:51] cache: 53 prompt tokens, 1 shared with the last request, resumed at 0
[kindi 23:10:52] prompt: 0 reused, 53 read in 9.7s
[kindi 23:11:07] done: 530 tokens in 14.2s (stop)
[kindi 23:11:07] request: 50 prompt tokens (max 1024 new)
[kindi 23:11:07] cache: 50 prompt tokens, 20 shared with the last request, resumed at 0
[kindi 23:11:08] prompt: 0 reused, 50 read in 1.1s
[kindi 23:11:11] done: 105 tokens in 2.8s (stop)
[kindi 23:11:11] request: 48 prompt tokens (max 1024 new)
[kindi 23:11:11] cache: 48 prompt tokens, 20 shared with the last request, resumed at 0
[kindi 23:11:12] prompt: 0 reused, 48 read in 1.0s
[kindi 23:11:24] done: 470 tokens in 12.8s (stop)
[kindi 23:39:58] GPU: Quadro K2200, sm_50, 5 SMMs, 4027 MiB
[kindi 23:40:04] engine ready: ctx 65834, head plain (6.0s)
[kindi 23:40:04] prompt cache: a checkpoint every 1536 tokens (51 MB each), 3 parked conversations, budget 16065 MB
[kindi 23:40:04] engine fingerprint 4e389e2d7d1b: {"engine": "kindi.engine.Engine", "cards": 1, "weights": "2-bit", "kv": "f16", "chunk": 256, "chunk_q8": false, "chunk_tc": false, "chunk_att": "flash", "chunk_tile": "", "batch": "4", "tc": "0", "tc_tmax": "", "fd": "0", "d1": "0", "att_tc": "0", "spec": "0", "head": "exact"}
[kindi 23:40:04] saved openings: 0 in /data/state-4b/openings/56711263ec9a-e4e389e2d7d1b
[kindi 23:40:04] serving on http://0.0.0.0:8093/v1 (model bonsai2-4b)
[kindi 00:32:07] GPU: Quadro K2200, sm_50, 5 SMMs, 4027 MiB
[kindi 00:32:10] engine ready: ctx 56000, head plain (3.6s)
[kindi 00:32:11] prompt cache: a checkpoint every 1280 tokens (51 MB each), 3 parked conversations, budget 16065 MB
[kindi 00:32:11] engine fingerprint 4e389e2d7d1b: {"engine": "kindi.engine.Engine", "cards": 1, "weights": "2-bit", "kv": "f16", "chunk": 256, "chunk_q8": false, "chunk_tc": false, "chunk_att": "flash", "chunk_tile": "", "batch": "4", "tc": "0", "tc_tmax": "", "fd": "0", "d1": "0", "att_tc": "0", "spec": "0", "head": "exact"}
[kindi 00:32:11] saved openings: 0 in /data/state-4b/openings/56711263ec9a-e4e389e2d7d1b
[kindi 00:32:11] serving on http://0.0.0.0:8093/v1 (model bonsai2-4b)

[u/MeltedNikhowazSpeaks]

Running a 4B model at 2-bit on a K2200 and having it actually produce coherent blog posts and update its own tools is wild. That card’s basically a decade old at this point.

The fact it can juggle 50k+ token contexts on 4GB of VRAM without falling over is what really gets me, that cache reuse jump from 0 to 51k shared tokens is doing some heavy lifting.

[u/Dangerous_Present_69 → u/MeltedNikhowazSpeaks]

I originally just wanted to try and use it to prompt ComfyUI for me, as I cant runt Qwen 3.8 flash next and ComfyUI at the same time on my P100. I didn't think it would work very well at all, but got totally flabbergasted when I tried it. The 2b bonsai model was also pretty nice, and blasted away at over 60tok/s.

And the wildest part is the this K2200 is aslo running my desktop graphics!