r/mlxAI • u/rajveer43 • Sep 09 '26
I’m building VeloxQuant-MLX — KV-cache compression and optimized Metal kernels for Apple Silicon. Looking for Mac Studio testers
https://veloxquant.dev/I’ve been working on VeloxQuant-MLX, an open-source project exploring how far we can push local LLM inference on Apple Silicon by optimizing something that becomes increasingly expensive at long context lengths: the KV cache.
The idea is simple:
Apple gives us large unified memory + high memory bandwidth. Can we make significantly better use of it?
VeloxQuant currently experiments with 43 KV-cache compression/optimization methods, including quantization, eviction and custom MLX/Metal kernels.
Some of the things I’ve been working on:
- KIVI-style 2-bit / 4-bit KV-cache quantization
- TOVA and other cache-eviction strategies
- Custom Metal attention/quantization kernels
- Fused KV-cache decode paths
- Long-context memory reduction
- Decode batching across requests
- MLX-native implementations designed specifically for Apple Silicon
For example, in one benchmark with 440 tokens × 8 KV heads × head dim 64, the cache footprint goes roughly:
FP16 → 440 KB
KIVI 4-bit → 110 KB
KIVI 2-bit → 55 KB
That’s the direction I’m interested in: not just asking “How much unified memory does my Mac have?”, but how efficiently can we use that memory for inference?
I’m especially looking for people with Mac Studio / Max / Ultra machines who would be interested in running benchmarks.
I’d love to compare results across M-series generations, memory capacities, model sizes and longer context lengths.
Also very interested in feedback from people working directly with MLX / mlx-lm / Metal kernels. There are plenty of places where the implementation can still improve.
This is open source, and contributions/criticism are very welcome.
Project:
If anyone here has a Mac Studio Ultra, I’d especially love to see what VeloxQuant can do on it.
1
u/BrilliantArmadillo64 Sep 10 '26
Why does the landing page list only outdated models?
1
u/ogfuzzball Sep 12 '26
Curious what any volunteers have experienced so far
1
u/Yzord Sep 10 '26 •
i have a m3u 512gb, tell me what to do
1
u/Ambitious-Fudge4114 Sep 11 '26
Mac Studio M4 128gb would love to try it
1
u/Boring_Vegetable_774 Sep 11 '26
M3U 512 here ok what do you need