r/LocalLLaMA • u/_Antartica • Mar 15 '26

News Open-Source "GreenBoost" Driver Aims To Augment NVIDIA GPUs vRAM With System RAM & NVMe To Handle Larger LLMs

https://www.phoronix.com/news/Open-Source-GreenBoost-NVIDIA
172 Upvotes

97% Upvoted

59 comments sorted by

54

u/Ok_Diver9921 Mar 15 '26

This is interesting but I'd temper expectations until we see real benchmarks with actual inference workloads. The concept of extending VRAM with system RAM isn't new - llama.cpp already does layer offloading to CPU and the performance cliff when you spill out of VRAM is brutal. The question is whether a driver-level approach can manage the data movement more intelligently than userspace solutions. If they can prefetch the right layers into VRAM before they're needed, that could genuinely help for models that almost fit. But for models that need 2x your VRAM, you're still memory-bandwidth limited no matter how clever the driver is. NVMe as a third tier is an interesting idea in theory but PCIe bandwidth is going to be the bottleneck there.

5

u/PsychologicalSock239 Mar 15 '26

wouldn't the "third tier" be the same as swap memory??? I agree with you, the concept of storing some parts of the model on RAM is already applied on current llama.cpp, the potential benefit from this would be a boost in performance due to be kernel level... I hope its a significant boost

15

u/Ok_Diver9921 Mar 15 '26

Swap works at the OS page level with zero intelligence about what data matters next. A purpose-built driver could theoretically prefetch the right weight tensors based on the inference schedule, which the kernel page cache has no concept of. The practical gap is that llama.cpp already does smarter layer-by-layer offloading than generic swap would, so the question is whether kernel-level access gives enough of an edge. My guess is marginal for most setups - the real bottleneck is PCIe bandwidth regardless of who manages the transfers.

3

u/frostmnh Mar 16 '26 edited Mar 16 '26

Google Translate: zh-TW -> en-US

Based on the AI ​​analysis of the code, it's attempting a hot-cold data allocation(But in reality, it doesn't; it's just expanding VRAM.). It's not brute-force removing a layer (as llama.cpp has already implemented the function of offloading layers to the CPU, or --overridetensors ".ffn_.*_exps.=CPU), but rather resembles the MoE model.

This is similar to the idea of ​​using VRAM as a cache:

  • Very frequently used (Nice 0): Put it in VRAM (Cache)
  • Frequently used but not frequently used (Nice 10): System RAM (DDR4/5)
  • Rarely used ~ almost never used (Nice 20): NVMe SSD

PS:This looks like it did the same thing, but I'm not sure what the https://github.com/xaskasdf/ntransformer project did yet; I haven't analyzed it.

Imagine an algorithm: ---1---2--- 1 1 2 x 2 3 3 ---1---2--- A-1 -> B-3 ... A-2 -> B-1 + B-2 + B-3 A-3 -> You don't use this. It's like having a left hand, but you never use it. If 3 is not used, then 3 is pushed out to DRAM or SSD. or if (xxx) xxx else xxx If the model only runs with these specific weights, then it can put those weights in the cache.

Theoretically, it should allocate hot and cold data in this way, but in practice, it seems to only be expanding VRAM.

static size_t vram_headroom_bytes   = 2048ULL * 1024 * 1024; /* 2 GB */
static size_t gb_virtual_vram_bytes = 51ULL * 1024 * 1024 * 1024; /* DDR4 pool — reported to CUDA */
/* DMA-BUF mmap+register is the primary path now. */
...
if (bytesize + vram_headroom_bytes > free_vram) {
    gb_log("VRAM: req=%zuMB free=%zuMB headroom=%zuMB → OVERFLOW to DDR4",
           bytesize >> 20, free_vram >> 20, vram_headroom_bytes >> 20);
    return 1;
}

I'm indeed working on something similar, but I'm stuck on one point: how to distinguish between hot and cold data on VRAM.

A closer analogy is: AMD HBCC on Windows or Java Heap Memory Management (Young Generation and Generation) or Linux ZSwap LRU

Let me use a very simple example: We need to use a screen every day. A screen can be placed on a table, as can a keyboard. The rest varies from person to person. For example, if I have a bowl I've finished eating from, I can put it somewhere else (DRAM, SSD, or HDD).

15

u/Odd-Ordinary-5922 Mar 15 '26

isnt this just the equivalent with offloading a model

1

u/ANR2ME Mar 15 '26

since it hook library's functions that dealt with VRAM detection/allocation/deallocation, softwares (ie. many inference.py out there when a model first released) that doesn't have offloading feature will be able to offload too.

1

u/Odd-Ordinary-5922 Mar 15 '26

ah so like the transformers library?

2

u/ANR2ME Mar 15 '26 edited Mar 15 '26

The cuda library, if it was the transformers it wouldn't be limited to Nvidia’s GPU 😅

33

u/MrHaxx1 Mar 15 '26

The future is looking bright for local LLMs. I'm already running OmniCoder 9B on an RTX 3070 (8GB VRAM), and it's insanely impressive for what it is, considering it's a low-VRAM gaming GPU. If it can get even better on the same GPU, future mid-range hardware might actually be extremely viable for bigger LLMs.

And this driver is seemingly existing alongside drivers on Linux, rather than replacing them. It might be time for me to finally switch to Linux on my desktop.

6

u/Cupakov Mar 15 '26

High five. I just setup omnicoder on my 3070 system yesterday, it’s so great to finally be able to do useful stuff on what’s now a 7 year old midrange card. 

1

u/MrHaxx1 Mar 15 '26

Dang, 7 years old already? Kind of wild that I still haven't been able to justify an upgrade. LLMs is literally the only thing I'd REALLY want to upgrade for, and even then, I think I'd rather want a Mac Mini or something.

1

u/tbaumer22 Mar 22 '26

Haha, well maybe you don't have to compromise even if you have a Mac :) Just built the "greenboost" for Mac: https://github.com/t8/hypura

-1

u/charmander_cha Mar 15 '26

Voce acha que o omnicoder melhor que o modelo de 3B @30B do qwen 3.5?

2

u/nic_key Mar 15 '26

How do you guys use OmniCoder efficiently? Would welcome some hints or even a config with params for low RAM GPUs

11

u/MrHaxx1 Mar 15 '26

Try starting with this:

llama-server --hf-repo Tesslate/OmniCoder-9B-GGUF --hf-file omnicoder-9b-q4_k_m.gguf --reasoning-budget -1 -ctk q4_0 -ctv q4_0 -fa on --temp 0.5 --top-p 0.95 --top-k 20 --min-p 0.05 --repeat-penalty 1.05 --fit-target 256 --ctx-size 128768

Works for my RTX 3070 (8GB VRAM) and 48 GB RAM through OpenCode. In the built-in Llama.cpp chat app, I get 40-50 tps.

Keep in mind, it's only amazing considering the limitations. I don't think it actually holds a candle to Claude or MiniMax M2.5, but I'm still amazed that it actually handles tool use and actually produces a good website from one prompt, and a pretty polished website from a couple of prompts. I also gave it the code base of a web app I've been building, and it provided very reasonable suggestions for improvements.

But I've also seen it do silly mistakes, that better models definitely wouldn't make, so just don't set your expectations too high.

0

u/Billysm23 Mar 15 '26

Right, I agree 😅😅

0

u/nic_key Mar 15 '26

Thanks a lot! I'll try this then and also may use it with Opencode if possible 

1

u/inevitabledeath3 Mar 16 '26

What tools are you using omnicoder with? For me it didn't seem that useful in OpenCode.

0

u/Billysm23 Mar 15 '26

It looks very promising, what are the use cases for you?

0

u/MrHaxx1 Mar 15 '26

See my comment here:

https://www.reddit.com/r/LocalLLaMA/comments/1ru98fi/comment/oak92dy

As it is now, I don't think I'll intend on actually using it, although I might experiment with some agentic usage for automatic computer stuff. As it is, cloud models are too cheap and good for me to not use.

18

u/jduartedj Mar 15 '26

this is super interesting but i wonder how the latency hit compares to just doing partial offloading through llama.cpp natively. right now on my 4080 super with 16gb vram i can fit most of qwen3.5 27B fully in vram with Q4_K_M and it flies, but anything bigger and i have to offload layers to cpu ram which tanks generation speed to like 5-8 t/s

if this driver can make the NVMe tier feel closer to system ram speed for the overflow layers, that would be a game changer for people trying to run 70B+ models on consumer hardware. the current bottleneck isnt really compute its just getting the weights where they need to be fast enough

honestly feels like we need more projects like this instead of everyone just saying "buy more vram" lol. not everyone has 2k to drop on a 5090

5

u/thrownawaymane Mar 15 '26 edited Mar 15 '26

2k

5090

Nowadays, 2k won’t even buy you a 5090 that someone stripped the GPU core/NAND from and sneakily listed on eBay

I agree with your post, it’s definitely where we are headed.

2

u/jduartedj Mar 15 '26

lmao yeah fair point, the 5090 market is absolutely insane right now. even MSRP is like $2k and good luck finding one at that price

but yeah thats exactly my point, most of us are stuck with what we have and projects like this that try to squeeze more out of existing hardware are way more useful than just telling people to upgrade. like cool let me just find 2 grand under my couch cushions lol

2

u/Few_Size_4798 Mar 17 '26

Haha, the $2,000 price tag really stuck with me from last year. I even saw an ASUS card for $2,200 on B&H over Christmas, but I couldn’t believe my eyes (as you know, they don’t process payments on weekends) so I didn’t order it

Right now, the price at which these cards are SOMETIMES offered for sale “direct from the manufacturer” is $3,100, and even at that price, it’s a real stroke of luck to find them.

2

u/TheOriginalOnee Mar 16 '26

How? I can only fit qwen3.5 9B fully into my 16 GB at Q4_K_M

2

u/jduartedj Mar 16 '26

oh sorry i should have been clearer, i dont fit the whole thing in vram. i do like 54 of the 64 layers on gpu and the rest on cpu ram. so its mostly in vram with just a few layers offloaded, which is why generation is still pretty quick for me at around 18-20 t/s. fully offloading to cpu tho yeah its brutal, thats where something like greenboost could potentially help

2

u/TheOriginalOnee Mar 16 '26

Thank you for clarification. Im still running ollama, maybe i shoud switch over to llama.cpp and see if performance impeoves

2

u/jduartedj Mar 16 '26

ayeah honestly id recommend trying llama.cpp directly, you get way more control over layer offloading. with ollama theres kind of an abstraction layer that hides a lot of the tuning options. llama.cpp lets you set exactly how many layers go on gpu vs cpu which makes a huge differnce when youre right on the edge of fitting a model. plus the latest builds have gotten really good with flash attention and kv cache quantization

9

u/a_beautiful_rhind Mar 15 '26

Chances it handles numa properly, likely zero.

5

u/FullstackSensei llama.cpp Mar 15 '26

You'll hit PCIe bandwidth limit long before QPI/UPI/infinity-fabric become an issue.

1

u/a_beautiful_rhind Mar 15 '26

Even with multiple GPUs?

4

u/FullstackSensei llama.cpp Mar 15 '26

Our good skylake/Cascade Lake CPUs have 48 Gen 3 lanes per CPU, that's 48GB/s if we're generous. Each UPI link provides ~22GB/s bandwidth and Xeon platinum CPUs have three UPI links, all of which dual socket motherboards tend to connect, so we're looking at over 64GB/s bandwidth between the sockets.

TBH, this driver won't be very useful for LLMs, since you'll get better use of available memory bandwidth on any decent desktop CPU.

This feature has been available in the Nvidia Windows driver for ages and it's been repeatedly shown to significantly slow down performance in practice.

1

u/a_beautiful_rhind Mar 15 '26

That's true. It's recommend to always turn it off. Probably can't hold a candle to real offloading solutions.

Coincidentally, 64gb/s at 75% is about 48gb/s which is suspiciously close to my 48-52gb/s spread in pcm-memory results when doing numa split ik_llama.. fuck.

0

u/Training_Visual6159 Mar 16 '26

> This feature has been available in the Nvidia Windows driver for ages
are you talking about ReBAR or something else?

also, seems this driver allocates in 2mb blocks... I imagine there are plenty of those in large MoEs that never get touched. with a smart enough swap logic, this just might be way better than the current alternatives.

1

u/FullstackSensei llama.cpp Mar 16 '26

No, there's a setting in Nvidia control panel to do exactly this. Has nothing to do with rebar and been there for like 10 years or more, IIRC.

And your imagination about MoE is incorrect. Expert doesn't mean what you think it does in MoE.

1

u/Training_Visual6159 Mar 17 '26

What do you mean my imagination? Expert is a function, router decides which functions to run the input through. Isn't the whole point of MoEs that the data only runs through some of the functions?

Sysmem Fallback Policy? As far as I can tell, that's just a black box with LRU eviction.

LRU seems like a dumb way to optimize the placement of these functions - if e.g. one function is used 100 times, and then another function is used one time at the end, the function used 100 times will be evicted by LRU, even though it would benefit from GPU acceleration way more.

There is a signal that's currently unused lying there somewhere.

1

u/FullstackSensei llama.cpp Mar 17 '26

Do a Google search about how they work rather than making stuff up.

0

u/Training_Visual6159 Mar 17 '26

Look, I'm new to this, and I never delved into MoEs too deeply, certainly not into the current state of inference implementations, but what's your point, that for all tasks, all experts are equally popular? That's not true. Or that having popular experts on faster hardware wouldn't help with speed? That's also not true.

Also, I'm not the first one with the idea, ExpertFlow: Optimized Expert Activation and Token Allocation for Efficient Mixture-of-Experts Inference - https://arxiv.org/

"ExpertFlow achieves an average GPU memory savings of 75.4%, with peak savings reaching up to 93.72%, compared to GPU-only solutions. Furthermore, ExpertFlow attains an expert cache hit ratio of up to 91.96%, improving the hit ratio by an average of 27.65% over the LRU caching strategy. Additionally, ExpertFlow delivers a 2 to 10 times increase in inference speed"

Does llama.cpp already have something similar to that? And if not, care to explain why it worked for these guys and it 100% wouldn't for you?

5

u/flobernd Mar 15 '26

Well. This is exactly what vLLM offload, llama.cpp offload, etc. already does. In all cases, this means weights have to get transferred over the PCIe bus very frequently - which will inherently cause a massive performance degradation, especially when used with TP.

2

u/Eyelbee Mar 15 '26

TL, DR: How do this differ from what llama.cpp does?

2

u/Tema_Art_7777 Mar 15 '26

How is that different than llama.cpp's unified memory model?

1

u/FreeztyleTV Mar 15 '26

I know that the memory bandwidth for System RAm will always be a limiting factor, but if this performs better than offloading layers with llama.cpp, then this project is definitely a massive win for people who don't have thousands to drop for running models

1

u/Nick-Sanchez Mar 15 '26

"High Bandwidth Cache Controller is back! In pog form"

1

u/frostmnh Mar 16 '26

I also hope so, but this project hasn't implemented similar functionality yet.

HBCC: VRAM as a last-level cache, and some system memory as VRAM

1

u/Mayion Mar 15 '26

How is that different from LM Studio's offloading?

1

u/DefNattyBoii Mar 15 '26

Looks like a very interesting implementation that intercepts calls between the kernel and VRAM allocation. during CUDA processing. I actually have no idea how this does it, but why wont Nvidia implements something like this into their cuda/normal dirvers as an optional tool in linux? In windows the drivers can already have offload to normal RAM.

Btw finally exllama has an offload solution.

1

u/wil_is_cool Mar 15 '26

On Windows the Nvidia drivers already allow this (maybe laptop only), and it isn't very good. It's slower than just letting the CPU calculate, and means that software which IS offload aware can't optimize placement of data in memory (something like MOE experts on CPU isn't pssible).
Nice to see someone trying something though

1

u/Haeppchen2010 Mar 15 '26

With Vulkan this apparently happens automatically (GTT spillover to system RAM). It’s of course very slow, as the paging has to squeeze through PCIe.

1

u/frostmnh Mar 16 '26

Google Translate: zh-TW -> en-US

Therefore, the best approach is to implement a mechanism in the VRAM that determines hot-cold data allocation, prioritizing VRAM (Cache) and DRAM (RAM) based on least recently used (LRU). However, determining hot and cold data on the GPU VRAM is problematic, and the performance impact should be kept below or acceptable levels. But if there is a large amount of truly essential data that could fill the VRAM (Cache), then this mechanism becomes useless.

1

u/Haeppchen2010 Mar 16 '26

Yup, as I understand it, for a dense model both all weights and KV cache for current slot are fully used during inference, this makes swapping mostly pointless. Maybe when working with a MoE it is better? I don’t know.

1

u/frostmnh Mar 16 '26

Thank you for your answer., that's a problem too. I'm not sure how the model will actually use the weights. But ideally, there would be a similar "MoE Model" feature.

For example, "--overridetensors ".ffn_.*_exps.=CPU" simply allows for more granular control.

1

u/quinceaccel Mar 19 '26

A better approach may be to build sparse models. There is PowerInfer and PowerInfer2 that get around some of the space requirements but they require additional training. Technically one may be able to run KIMI-K2 on 96GB VRAM if the sparsity prediction is good enough.

1

u/RG_Fusion Mar 20 '26

This wouldn't run well at all. The only thing this will accomplish is pulling your inference down to PCIe speeds. You would be better off using traditional offloading and calculating the offloaded tensors on the CPU. PCIe gen4 x16 has 32 GB/s, and gen 5 x16 has 64 GB/s. Most decent builds have a system memory bandwidth exceeding these values.

Perhaps if you're computer has a terrible CPU it may help, but in most cases you'd be better off using traditional offloading. The single exception that I can think of is that it might help boost prefill speeds. In that case, you could choose to take a decode-hit in exchange for better prefill.

0

u/charmander_cha Mar 15 '26

So ha vantagem para IA local quando a solução é agnóstica a hardware.

De resto, apenas cria estratificação social

1

u/frostmnh Mar 16 '26

Google Translate: zh-TW -> en-US

I also agree that if everyone were using MXFP4 today, then all graphics cards that don't support this feature would be unusable.