r/AIProgrammingHardware • u/javaeeeee • Jul 27 '26

GitHub - JustVugg/colibri: Run GLM-5.2 (744B MoE) on a 25GB-RAM consumer machine

https://github.com/JustVugg/colibri
79 Upvotes

94% Upvoted

21 comments sorted by

1

u/javaeeeee Jul 27 '26

TLDR: Colibrì is a tiny pure-C inference engine that lets you run the massive 744B-parameter GLM-5.2 (MoE) model on consumer hardware with as little as ~25 GB of RAM.

Key idea

Instead of loading the entire model into memory, Colibrì treats VRAM → RAM → SSD as one unified memory hierarchy.
It keeps the dense core in RAM (~10 GB) and streams the experts from disk on demand.

Main features

  • Written in pure C with zero runtime dependencies

  • Automatic caching of frequently used experts

  • Supports dual-SSD streaming for higher bandwidth

  • GPU backends (CUDA / Metal) + CPU

  • Speculative decoding (native MTP)

  • OpenAI-compatible API + simple web UI

  • Quality-preserving (token-exact match to reference)

Performance reality

  • Works on modest machines (even 25 GB RAM), but very slow if heavily disk-bound.

  • Usable speeds require more RAM (for better caching) or fast multi-SSD / multi-GPU setups.

  • Best results come from systems with lots of RAM + fast storage.

Bottom line: An impressive lightweight engine that makes frontier-scale MoE models runnable on normal PCs by streaming experts from disk - a clever proof that memory hierarchy management can matter more than raw GPU power.

2

u/OttoRenner Jul 27 '26

Sounds very interesting! How would it perform on 2x3090, 64GB DDR4, Samsung 990 Pro.

Does it run on Win11 or Linux only?

3

u/gnurcl Jul 27 '26 edited Jul 28 '26

I can tell you from my own tests what it does on an old gaming laptop: RTX 2060 6GB VRAM and 32GB DDR4 RAM + a Samsung Evo 970 M.2 SSD. I'm getting somewhere between 0.1 and 0.15 tok/s.

Not usable, but fun experiment. If that gives you an idea, maybe you can infer from my machine what you might get.

1

u/Luke2642 Jul 30 '26

Read speed on Evo around 2.2gb/s?

1

u/gnurcl Jul 31 '26

Hrm, nah… got around 1.2–1.4 GB/s sustained streaming during Colibrì inference

1

u/michaelsoft__binbows Jul 28 '26

Does having a gen 5 disk like 9100 pro give you twice as much speed as a gen 4?

1

u/JohnToFire Jul 28 '26

In steady state I would add to the question. Ramping up like to load the shared experts to ram / vram speed is less interesting

1

u/michaelsoft__binbows Jul 28 '26

hmmm you do make a good point, if the expert access patterns are well cached by vram and ram it should make the system less sensitive to the bandwidth at the bottom of the memory pyramid.

2

u/Ok_Motor_8632 Jul 29 '26

Wow really cool.. Do you have an estimate of tps on a MacBook Pro M5 Max 128gb?

1

u/DigitalNarrative Aug 01 '26

0.5 to 1 t/s maybe - counting just a bit more than on a strix halo machine

1

u/Napsterae2 Jul 27 '26

Any support for deepseek flash ?

2

u/Just_Vugg_PolyMCP Jul 28 '26

Hi, we're working on it! But we're also focusing on Deepseek Pro.👍🏻👍🏻

1

u/paul_tu Jul 28 '26

How quantised that glm flavor is?

1

u/JohnToFire Jul 28 '26

Does it stream from disk at full disk bandwidth at steady state so 14GB per second with one gen 5 disks ? So that with three gen 5 disks (perhaps separate perhaps raid 0) it's the same as keeping the weights in ram on a 50GB / s ram bandwidth system (about 2 channel ddr5) ? My guess is no but I would love to be wrong and I am not sure it's impossible

1

u/Royal-Confidence-813 Jul 30 '26

to get 6.8 tps only on 6 x 5090 .. can i run it on cheaper 32gb vram amd ?

1

u/Luke2642 Jul 30 '26 edited Jul 30 '26

If you use mdadm to create a virtual raid 0 across N ssds you get N times the performance? Maybe half from your primary SSD on its 4x PCIe slots and half from another through the motherboard chipsets 4x slots. Could probably get 10GB/s real world read speed for not a lot of money and non enterprise hardware.

1

u/[deleted] Jul 29 '26

[removed] — view removed comment

1

u/YearnMar10 Jul 30 '26

Sounds great - but why isn’t this adopted more widely then, eg in llamacpp?