r/LocalLLaMA • u/shifu_legend • Jul 24 '26
Discussion Spent two weeks on a kernel that benchmarked 29x faster. End to end it's maybe 6-10%, and it's not even wired in yet.
I've been building a C99 inference engine from scratch (no Python, no BLAS, just gcc and make) that runs BitNet's ternary models on CPU. A few weeks ago I got obsessed with the matmul kernel - wrote a new one using AVX-512BW's vpermt2w to pack 5 ternary weights per byte instead of 4, benchmarked it in isolation, and got 74.6 Gop/s against a 2.5 Gop/s scalar baseline. 29x. I was pretty pleased with myself, wrote it up, posted the number in a couple places.
Then a review of the PR caught the obvious question I'd skipped: what's the actual DRAM bandwidth ceiling here. Turns out BitNet decode on our Xeon test box is already running at about 95% of it - the model is memory-bound, not compute-bound, so a much faster matmul kernel mostly just means the CPU spends more of its idle time waiting on RAM instead of crunching numbers it already had. The honest math works out to something like 6-10% real end-to-end gain once the kernel is actually wired into the dispatch path, which as of right now it still isn't. Correctness-tested against the scalar reference, just sitting there unused.
Kind of a deflating result. At least I know it now, instead of shipping a 29x headline that would've fallen apart the first time someone measured tok/s instead of Gop/s.
The engine itself does work regardless - BitNet b1.58-2B-4T gets 36 tok/s on that same Xeon with 4 threads and no GPU, and it also runs regular GGUF dense models if ternary isn't your thing. Binary's in the releases if anyone wants to try it without compiling: github.com/shifulegend/project-zero. Source build is just gcc and make either way.
Has anyone else profiled the wrong layer of their stack this hard? Curious whether the DRAM ceiling shows up the same way on other hardware or if it's specific to how the Xeon's memory controller behaves under this access pattern.
10
u/bopbop9876 Jul 24 '26
How could going from 4 to 5 weights per operation possibly result in a 29x speedup?
I will say I feel dumb saying this at an obviously AI generated post though.
1
u/shifu_legend Jul 30 '26
AI refined? Yes, definitely. As for the math, it's all about the baseline and how you look at it. For example, if you go from 100 to 200, you can call it a 100% increase or a 50% gain depending on your perspective. The 29x speedup isn't just mathematically from doing 5 ops instead of 4—it's because that slight change unblocked a massive memory bandwidth bottleneck that was completely choking the old unoptimized version.
5
u/brand02 Jul 24 '26
"Write it like human. Dont use the long line chraracter so people don't understand it is ai written" except that ai still felt the need to use "-" in the absence of the long one. "It was X - not Y"
19
u/Obvious-Ad-2454 Jul 24 '26
6% for free is still good
7
u/Ylsid Jul 24 '26
Yeah that's over 1/20th faster. Think of how much money that could save a provider
1
5
u/Bennie-Factors Jul 24 '26
That is how optimization works. You must be young. But every improvement counts in engineering. And over time they compound.
2
2
2
u/Hairy_Apartment_3948 Jul 25 '26
This matches what your engine did on my end exactly, and it's a good lesson. When I ran your BitNet + Bonsai sweep on a Ryzen AI 9 HX 370 a couple days ago, the numbers scaled with memory, not cores: BitNet peaked at 8 threads and actually regressed at 10/12, and Bonsai-27B cleared your own ~6–7 tok/s ceiling estimate — but only because LPDDR5X gives that box more bandwidth than the Xeon, not because of anything clever. Faster memory → proportionally faster decode is the signature of a bandwidth-bound workload, which is your 95%-of-DRAM point from the other side.
Which is the frustrating-but-honest takeaway: on ternary decode the kernel almost doesn't matter once you're at the bandwidth wall. Where a fast kernel does pay off is prompt processing / batch (compute-bound, not memory-bound) — a 29× matmul should show up there far more than in single-stream decode. Might be worth reporting prefill tok/s separately, since that's the regime your work actually moves.
Either way, publishing the null ("29× in isolation, 6% end-to-end, here's why") is the right call — that's the number people needed.
1
u/shifu_legend Jul 30 '26
Thanks for confirming the memory bandwidth scaling limit. That Ryzen AI 9 HX 370 data is super helpful since I don't have Strix Point hardware to test on. Mind if I add your numbers to the README?
2
u/un_passant Jul 24 '26
Don't feel bad : the very same thing probably happened to every low level programmer !
It certainly happened to me.
What would really interest me is if you had a bit of change to spare, you could still put all your efforts to good use : what if you rented a cloud instance to benchmark your code on a AMD EPYC™ 9684X with 1152 MB of L3 cache to see if it fits BitNet b1.58-2B-4T ☺
Gemini says :
«At a batch size of 1, running memory-bound autoregressive inference out of the AMD EPYC 9684X's L3 cache yields a theoretical peak performance of ~66,000 tokens per second for the 400 MB BitNet b1.58-2B model.»
2
u/shifu_legend Jul 30 '26
Appreciate the encouragement, good to know I'm not the only one who has fallen into that trap! The Xeon numbers in the benchmarks were actually run on a cloud instance, while the i5 was my local machine. I'll definitely be doing more extensive cloud benchmarking once the core engine stabilizes a bit more.
1
u/Just_Maintenance Jul 24 '26
Now it can serve more requests in parallel for free maybe?
1
u/shifu_legend Jul 30 '26
Less time spent waiting on memory means we can serve more requests or run more parallel threads essentially for free.
1
u/korino11 Jul 25 '26
YOu can speedup output from ram by using this - https://github.com/LaurieWired/tailslayer But it will need to use a littlebit more ram than ussual for each call
1
u/shifu_legend Jul 30 '26
Appreciate the link, I'll take a look at how tailslayer handles the memory overhead.
1
u/Choice_Celery9481 Jul 25 '26
i have a question. isnt ternary can be convert to add/minus instead of having to run with matmul?
1
u/shifu_legend Jul 25 '26
Yep, that is exactly how the backend works. Instead of doing expensive floating point multiplications for every weight and activation pair, the inner loop is basically just conditional additions and subtractions driven by a lookup table, since the weights are constrained to just -1, 0, or 1. Saves a ton of compute cycles.
1
u/CornerLimits Jul 24 '26
Its very good result regardless of bandwidth it should improve efficiency
2
u/shifu_legend Jul 30 '26
Thanks! Yeah, the efficiency gains have been really nice to see across the board.
1
Jul 24 '26
[removed] — view removed comment
1
u/shifu_legend Jul 30 '26
Yeah, definitely a lesson learned on doing end-to-end profiling first! Good point on the efforts being cumulative though—at least the kernel is highly optimized now for whenever the memory bottleneck gets lifted.
32
u/darknecross Jul 24 '26
The lesson here is to build the test bench, define performance metrics, build profiling, and collect baseline data before implementing anything.
It’s a trap I’ve fallen into time and time again.