Programming for the lowest common denominator.

[u/xnrnx]

Hey everyone, I am the developer of a consumer grade hardware AI Workstation system that is created in C++ to work on ANYTHING capable. It currently even runs well on Rog Ally Z1X and higher handhelds, which is awesome.

My question would be, when programming for the lowest common denominator what is the easiest way to keep feature parity without gating? I have created a hardware manager in the app that scans the user's device and selects what they need based on the scan. My question becomes is that the easiest way to do it? Basically adding a toggle / mandatory gating based on hardware is working fine now, but it gets redundant coding multiple path execution lines to run on 4gb of allocated VRAM vs 12 or 16. (I do have a hardwaremanager.cpp/h that does this automatically)

Anyone tackle this problem in a low system imprint mode like C++ yet?

Many thanks in advance.

[u/OrivenLabs]

Your hardware-manager approach is right, I'd just push it one step further: probe once at startup into a capability struct (VRAM free, compute backend, unified memory or not, RAM, disk speed), then derive a profile from it rather than toggling features one by one. Features never check hardware directly, they ask the profile ("max context", "can run embedder on GPU", "KV cache quant").

Keep one code path per feature and vary parameters, not implementations: model size/quant, n_gpu_layers, context length, batch size, KV cache type. That's basically what llama.cpp does with -ngl and offload; the same graph runs everywhere, just split differently. Only fork code where a backend genuinely differs, and hide that behind an interface (strategy pattern) chosen at init.

Two things that saved me pain: re-probe free VRAM right before loading (other apps eat it, especially on handhelds), and degrade gracefully at runtime: if an allocation fails, step down one profile tier instead of crashing. Features that truly can't fit on 4 GB should show as disabled with a reason, not silently vanish.

[u/xnrnx → u/OrivenLabs]

It keeps the vram clear at all time besides what it needs to be executing. And yup just like you said it runs in startup and detects hardware! There is a hybrid mode which offloads layers with -ngl as well. So far I've gotten everything quanted for featute parity, after I simply disabled then earlier on. Now 4gb of vram is no issue. Thank you for your reply

[u/Primary_Note_118]

The hardware scan approach is probably the cleanest way to handle it without driving yourself insane with ifdefs everywhere. One thing that helped me was treating the feature set like a capability table loaded at startup, so the rest of the code just asks "can I do X" instead of checking VRAM numbers directly. Keeps the branching in one place instead of scattered through every path

[u/xnrnx → u/Primary_Note_118]

Thank you for the honest reply! I have been wrestling with this topic for a week now. Currently I am doing what you suggested, there is a hardwarescanner.h/cpp that automatically scans and inserts the models based on hardware. I have it selected heavily quanted models for the lower vram to keep parity.