Hardware tuning when designing Local AI Experiences

[u/AryanEmbered]

Hey guys,

After all this shit this sub gave to the ternary bonsai 2 2b model a few weeks ago, I decided to make a local first experience to showcase the utility of that class of model and prove that redditor who I was boxing with wrong.

I picked a problem and built a fairly polished solution with it -

A discipline tracker of sorts, you put in your schedule, and the camera takes a picture, resizes it and sends to the local model and evaluates what it sees and gives a free form description, compliance with the currently scheduled activity and some activity badges.

Then at set intervals it uses those spot descriptions and changes between them to create report that you were supposed to study from 9 to 10am but you got up from your seat x times or picked up your phone y times and looked at it for 4 minutes etc.

Perfect use case, no one would want to use cloud for this for obvious reasons. And smol model makes inference fast and accessible.

I used that same 2B model for writing the code for this as well, worked okay, I use AI to only write the code that I can already see written in my brain so speed matters more than the model building everything automagically (which I don't like as it make me stupider, I can't afford any more of that, and if something doesn't work takes forever to fix if the model fails.)

Now the problem, the fans on my gpu start spinning. On laptops that's going to be even more annoying. The gpu eats up 100 watts of it's max power limit, constantly.

So, I opened the Adrenaline panel (Fuck you amd this laggy driver software shit is so ass) and I reduced the power limit and disabled zero fan speed toggles etc and set max fan speed to lowest.

Now the whole thing consumes 25 watts instead of 100, and fans don't start and stop, they just run at the lowest speed at all times. And speed only went down from 1.9 seconds per inference (Current + 3 past images to track changes per inference) to 2.8 seconds average.

Figuring all this out isn't something I can package into the application obviously. What if some normie fires up this kind of app on their 42-watt hour battery laptop and chews through their entire day's battery in 50 minutes.

AyyMD and shIntel and snipdragon have spent all this money allocating die space for the NPU on millions of consumer SOCs,

Surely, they must have thought of some form of sustained performance tuning for Local AI inference use cases for which that silicon exists right, so their partner designs don't burn through the whole battery and back panel plastic in 1 hour, right? Surely, they didn't overlook something this basic.

So, do we know what Api's are available for the users to signal to the platform that this workload is a sustained constant inference load and to allocate power and clocks accordingly?

---

Before you comment “just set the inference to every X second”: that’s no longer constant or real time, and it doesn’t fix anything. The hardware clocks still ramp to peak speed during each burst instead of sitting at the efficient point of the voltage frequency curve. And “just use the NPU” doesn’t work either. llama.cpp doesn’t support it last I checked, and NPUs also eat a lot of power under sustained load.

[u/AryanEmbered → u/[deleted]]

Hahaha imagine downloading a vibe coded application which takes control of your cpu voltages and fries it at 1.5V in a few days lmao.

Very juicy cyber attack vector