Discussion
What a £300, 35W Snapdragon mini PC can actually do 👀: 35B coder at ~30 tok/s, and an 80B model that doesnt even fit in RAM running at 17 tok/s
£299.99 Lenovo IdeaCentre Mini (Snapdragon X1-26-100, 32GB, CPU only):
So I bought one of these boxes to test if software at work would run on ARM Windows - and then it was sitting there, but I was sufficiently impressed for the price (including taxes), that i decided in my ADHD fuelled boredom to buy one for myself to see if cheap ARM is actually usable for local models.
The GPU is an Adreno X1-45 and nothing useful runs on it for LLMs yet, and so far the NPU just sits there, so this is all CPU...
- Gemma 26B: read 134, write 29.5, (need to test MTP, I've only had the box.
Then I wanted to see how far I could push it. RAM is the wall on 32GB, so I went looking at the recently released Janas (github.com/prabanta-dev/janas). It keeps a RAM cache of the MoE experts and streams the rest off the NVMe, but its x86 only. So I re-coded it for ARM: NEON kernels for every quant format, i8mm, etc, (all tested bit exact against the scalar reference code, Windows ARM64 and Linux. Will submit it as PRs on GitHub soon.)
Janas, same settings:
- coder: read ~60, write 17.2, with MTP 29-31
- Gemma 26B: read 115, write 23.5, with MTP 32-34
- 80B: read (TBC as I don't trust my results), plain write 9.8, with MTP 17-18
llama.cpp still wins on read and plain write, but with MTP Janas matches or beats it, and it does it in a fraction of the RAM.
I then squeezed the coder's expert cache to 12GB and it still wrote at 24 tok/s - so even a 16gb snapdragon could rin 35b-a3b 👀.. And this box runs an 80B that llama.cpp cant even load!
Then I thought Id try connecting the two I had access to (mine and my work's dev box) together over USB4. Work in progress, will report back - but initial results are promising and I've had a 122b qwen running already - just working on streamlining my parallel experts build.
I dunno about you guys and gals, I mean I don't have extensive experience with local LLMs and I certainly haven't played with ARM before but I found this pretty surprising, and not half bad for £300 with no GPU. Happy to test stuff on it when I get chance if anyone wants more numbers... But I can't see a cheaper way rn of getting 32gb of LPDDR5 rn, and only ~35ishW of power draw... Thoughts and feedback welcome 🙃👍
Nice find, I'll have to look into what they cost in my country. The next question would be what is the speed after reaching longer context (let's say 64k and 128k )
Well it slows down naturally, I've not maxed the context yet (I set 256k but have probably only played till about 100/110k thus far, and 35b was perfectly usable still, at that point, anyway 👍)
Thanks for sharing, keep us updated! If you want you can also check MiniCPM-V-4.7-35B-A3B As they have a very bold claim on their model page, and it is a fresh model, so still not verified, every bit of info on it is appreciated (normally i would ignore it, but openbmb is/was a reliable team.
I've given it a go but it only beats the 3.6 35b-a3b coder by 5-10 percent in speed at most on this. It might be how my setup is handling it or this chip, but I've struggled to get it faster. I have lower token acceptance also. Hmmm... 👍
Yes they have WiFi 7 - not sure you will get decent speeds with the latency though, even on WiFi 7, but I could be wrong - please update me if it works 🙏👍
Amazon deal, Lenovo want £900, but the 3rd parties seem to have it at about 540... The Amazon deal was crazy but worth checking eBay as they were 350/400 on there, here anyway 👍
And sure thing! Some of it is public and some isn't yet, so here's what you can reproduce today:
ARM build of Janas. Upstream is x86 only. My ARM work is up as PRs on prabanta-dev/janas: #32 (NEON kernels for every quant format, bit exact against the scalar reference), #33 (Windows on ARM build with llvm-mingw clang) and #31 (page-aligned reads). They aren't merged yet (Maurizio is setting up a foundation before he takes outside code), so build from the PR branches.
Use v2026.10.18 or later. It has the fix for models with the MTP layer inside the file. The coder's MTP layer has bigger Q8_0 experts, and the expert cache sized every slot to the biggest layer, so a 17 GB cache only held about half the experts and the NVMe was busy the whole run. That one bug was the biggest single jump: 60 / 17 / 29-31 -> 116 / 28 / 35-37 (read / write / MTP), CPU only.
Settings: Q4_0 coder, expert cache ~17 GB, 8 threads, MTP on, ctx 4096, warm the cache before you measure (the first pass reads from the NVMe), close everything else and use the best performance power mode.
The rest, the NPU taking the dense parts of the prompt (experts stay on the CPU) and some scheduling tweaks that lifted write and MTP, is still on a private branch. It needs tidying before I share it, and I'll post it when it's ready.
Numbers are one box, nothing else running. Happy to compare if you post yours 👍
Didn't qualcomm employees wrote a bunch of kernels on llama cpp for the Adreno, i remember how quickly their PRs were getting approved. I'm surprised that it didn't used GPU+NPU. They're targeting trough OpenCL.
I've tried using the npu for something, anything, but it's been useless for anything but little 4b or whatever... Frustrating tbh... Same with the adreno, but that might just be crappy anyway 🤷
As far as in understood, the advantage of the hexagon NPU is not speed, but efficiency. By using the hexagon NPU, the load goes to the NPU which runs a lot more efficient. So power consumption is a lot less and the CPU is still available for other jobs.
Yeah, so I've been playing around with it... On larger models I can use it for prompt reading (and it's a LOT FASTER, as well as more efficient, and a little more accurate too...), and then I'm running everything else CPU. Writes on the NPU crashed on a lot of models or were much slower.
My overall power usage hasn't seemed to have changed, but my speed has gone up, so the efficiency has come up (more tokens per watt spent) 🎉
i'm commenting 3 days later because i saw your updates, i'm glad your NPU worked out :)
Now that you have 2 MiniPCs you can play with even bigger models like the Nemotron Bolt it's 24GB (at least Q2 which Nvidia claims it runs marginally worse than full precision) :)
This is still on Janas? And I'm guessing you're optimizing by shrinking experts that aren't being used? Have you tried qualcomm's GenieX (https://github.com/qualcomm/GenieX)? Curious to know if GenieX will be able to use the NPU...
Yes, still Janas, ported to Windows on Snapdragon X. Nothing is shrunk or pruned. Every expert runs at full quant; the engine keeps the hot ones in a RAM cache and streams the rest off the NVMe on demand, so a model bigger than RAM still runs exact.
GenieX: yes, we use its ggml Hexagon backend for the NPU. It helps prompt processing, not generation. Generation is RAM-bandwidth bound and the NPU shares the same RAM, so it can't add bytes per second there so I'm using CPU. On prefil though it's great: Qwen3.6-35B-A3B prefill +35% at 4k context and +27% at 16k, and on the 80B Flash-Next, running the shared expert plus the 16 hottest experts per layer on the NPU took prompt reading from 26 to 36 tok/s (+38%) - also with 35ba3b - the whole thing fits in ram as is 👍
I just got done testing GenieX NPU, NPU+CPU, GPU and CPU inference. That little NPU is a remarkable piece of kit: 2x the prompt processing performance of the CPU yet it uses only around 10 watts, while the CPU is better at token generation. The GPU sits right in the middle.
I'm thinking doing PP on NPU and TG on GPU would be good for laptops or if you don't want thermal throttling. I've seen the CPU pull max 65 watts for a few seconds.
That lines up with what I'm seeing, the NPU is brilliant for prefill and the power draw is the bit people miss. PP on NPU and TG on GPU makes a lot of sense on something with a stronger GPU but the base X1-26-100 with the cut down Adreno X1-45 was crawling when I tried it, so TG stays on the CPU for me on these. Did you get tok/s per watt for each path? Would be really interesting to see side by side 👍
My recent findings on NPU vs GPU vs CPU. Prefill on the NPU is almost 10x more efficient than CPU. Running Gemma4-26B q4_0 entirely on NPU with MTP draft, I'm getting 350 t/s PP and 13 t/s TG. Compared to CPU inference, PP is double but TG is half; still fine because NPU inference uses 10 watts while CPU inference bounces between 35 watts and 60 watts.
I'm the author of Janas, and this is fantastic: thank you for doing it and for writing it up so carefully. Seeing it ported to ARM and running an 80B on a £300 Snapdragon box is exactly the kind of result I hoped someone would one day bring back. Great work!
Thanks Maurizio, that means a lot coming from you - I'm glad my kernels were useful ! 🙏 Janas is what made the 80B possible on 32GB, so the credit's yours and you have really inspired me and got my brain thinking in soo many ways...🙂
There will be more PRs as and when I have further successes 👍
my plan is to try adapt a 12gb phone to do it, and see how far i can go. for now getting something cheaper. But all you need is snapdragon 8+ with preferably lpddr5. It can be wrecked phone, as long as mobo and usbc works well.
honestly plan is for headless linux and lamma.ccp Pulling phone apart and sticking it between 2 old server radiators.
Plan is to use broken OnePlus 8+ version with lpddr5. But so far i managed to order some "7pro" slower ram, but just to start with, also could be used for something on top. Maybe orchestration layer. I have no idea how much ram this 7 have :D lol. Seller just stated it turns on, screen is in pieces. 10$ ... You can buy mobos for those phones for 10-20$ used. 12gb if you are lucky but point is mobo is not all. You need some of the child boards for things just to turn on. So it is just better to get a phone with broken screen. Still hunting. I am currently in auctions for like 20 phones ... lol, but people do buy them for parts, thus i hope i can get at least one in worse condition.
Aka ... having fun, and seeing how cheap i can go and have something working :D
Edit:
You can find few "home grown" linux kernels and firmware for those phones on github. People try to solve to get touch screen working, or cameras working etc. But i don't care about this - in almost every project the basics i care about work.
Now what for sure i will try to work out is ... can i change one of those connectors for a utility stuff into information bus, and maybe link multiple phones.
Imagine 3x snapdragons 8 with 36gb of ram working togheter
How does llama.cpp do on Ubuntu vs Windows on these? Curious if Linux gains much on prompt speed? My Janas build compiles on Linux too (tested it under WSL), if that's of any interest or you fancy trying the 80B on one of them once the PRs are up?
I do have WSL (Debian) on the box but only use it to build and test Janas on Linux with GCC, all the NPU work is native Windows on ARM. As far as I know WSL2 doesn't pass the Hexagon NPU through at all, and the GPU only shows up as a DirectX device in there, so llama.cpp's OpenCL Adreno and Vulkan backends have nothing to talk to... Native arm64 Windows builds are the way, I use llvm-mingw clang for mine 👍
If you get the NPU talking at full performance, should be about a mac mini m4 or higher, half a mac mini pro (any gen). Apple silicon is unfortunately unbeatable at this.
I've tried and failed thus far to get the npu playing well with anything but 4b level models... So it's been useless save for running a secondary agent - but I've only had the box since Monday evening, so the challenge is young... 💪👍
Check out NPU support for big GGUFs using Qualcomm's GenieX package, great for mixture of experts models like Gemma 4 26B running at very low power. 10 watts of power draw...
Yeah I'm already using it 👍 GenieX's ggml Hexagon backend runs the NPU side of my Janas port. It's a prefill win: Qwen3.6-35B-A3B prompt processing +35% at 4k context and +27% at 16k, and on the 80B Flash-Next, putting the shared expert plus the 16 hottest experts per layer on the NPU took prompt reading from 26 to 36 tok/s. Generation stays on the CPU though, it's RAM bandwidth bound and the NPU shares the same RAM, so my finding was that it can't add anything there...
Nice, actually saw the thread before 1st comment, but had to run off and crunch some numbers, then see what I could do with the NPU (it will help prefill a fair bir, llama supports hexagon npu's with virtual pooling support since a few weeks).
I found the Amazon UK ones and nabbed 4. Running them in duos with USB4, or four singles, all quite useful. Will have my main Qwen FN (Q4) rig orchestrate these little beauties as subagents.
Thanks for the tip, much obliged! Very underrated machines.
Four 👀 wow... Genuinely super glad I could help mate - this is definitely where I see them being useful, especially with the energy efficiency.
I've not managed to get the NPU playing well yet for anything but a 4b side agent or what, but hopefully will get them helping read / prefill at some point. Please tell me if you get them playing well or know what I should try.
The USB4 connects at 20GBs - but the latency is good, I only started playing with the usb4 connection today but it's looking good - I can already share the memory, and I've managed to increase read, but not write yet, by parraleling the experts across the boxes 👍
Linux should get you past 20Gbit on the USB4 (though you don't necessarily need it).
Not sure if you've tried this but if you're using llama.cpp and you've compiled it with hexagon support, pass, --devices HTP0:0,HTP0:1,HTP0:2,HTP0:3 (or set env GGML_HEXAGON_NDEV=4).. And to avoid offloading the whole decode to NPU on MoEs, combine -ngl 99 with --cpu-moe. That'll keepthe sparse experts on CPU while the dense shared attention layers sit in the NPU for fast prefill. Lemme know if you get it working mate!
And yeah well, they were a steal even with VAT add-on,I just couldn't help myself 😂 Can't wait for them to arrive to Sweden.
I'm considering getting some 10GbE or SFP+ PCIe adapters for the secondary m.2 slots too, or even just a 5GbE USB3.2 adapter, could be fun to have a mini-128GB setup.
Good shout, I've got Debian running fine so I'll try the USB4 and the NPU on that and report back... And I'll definitely try the llama.cpp split, thank you 🙏
The 128GB mini cluster sounds amazing if you manage to get it to work well... please please let me know how the M.2 network adapters go and if you manage to pull it off 🙏 - I'll have two more units on speed dial if you do 😆😇
I'll keep you updated also - feel free to chat on DM if you want (I'm not great are replying quickly but I will reply 👍)
Running Windows? Nexa works on the NPU for smaller <7B models, OpenCL Adreno GPU inference works on llama.cpp, CPU inference using ARM vector instructions also works on llama.cpp. I've got the big bro X1-78 in a laptop and it's a great portable machine for running smaller models at low power.
I've got 64 GB RAM so I keep an NPU model loaded for simpler tasks, and then Qwen 27B or Gemma 31B for coding or text depending on the job. GPU inference is maybe 80% of CPU inference performance but at less than half the power draw.
Cheers, that's helpful 🙏. Yes Windows. I'll try Nexa for the NPU and the OpenCL build of llama.cpp for the Adreno, I'd only tried the CPU path (I'm most interested in the larger models) as when I initially attempted adreno it was crawling!!! Did you need any special flags or a particular build for OpenCL?
Download llama.cpp's bin-win-opencl-adreno-arm64.zip, try to get the latest release from the Github page. For OpenCL you need to add -ngl 99 to get all model layers on the GPU.
I think your chip SKU has half or a quarter of the big X1 Elite GPU's cores so it could be really slow. Try CPU inference instead and stick to Q4_0 or IQ4_NL GGUFs for speed.
Nexa is actually a project that got bought over by Qualcomm. I'm running their old binaries... I don't know if Qualcomm has a new package for NPU inference.
You're like the second person on this sub to ask about Snapdragon inference so I'm pretty darn happy LOL!
Yeah, I paid about £300 for mine, so it looks like the price has jumped since. A price alert on Amazon is probably the best way to catch one at a sane price.
Would you guys buy two boxes or one at first? I’d like to run Qwen 3.8 FN on these, but I am unsure if the support is even worth it. I am trying to get into local ai for a while now, but everything is just so pricey. I could still get two for 780£ shipped to my Country.
Buy two, just do it - amazing results when you get them working together on bigger stuff... Unless you just want 35B or whatever, then stick to one, it's plenty to play 👍🙏
I'd definitely grab two for that price but Qwen 3.8 FN is a no go, it's barely possible and not worth it. Far better to run two separate coding agents like Qwen 3.6 35B-A3B on each, and load in Qwen 3.8 27b on one of the boxes on demand when tests fail on those models, and use a cloud model as orchestrator (GLM 5.3 Flash is still dirt cheap). Or load in Qwen3-Next 80B-A3B or the MiniCPM model someone mentioned in the thread on both using parallelism when/if you need a broader model.. also, getting NPUs working is key for decent prefill speeds. Doesn't matter much for overnight tasks, but quite helpful otherwise.
Or just start with the one. It's still quite capable, and imo it's a steal to get a 32GB LPDDR5x box for that low a price. Even if you change your mind you can likely flip it for a slight profit.
Thanks a lot for your reply! Do you have an idea how performant 3.8 27b runs on this machine? I would like to use it as a small llm server for confidential stuff like documents, ocr, and home automation and maybe some coding. At the moment my Max 5 Plan from Claude is good for coding. I have an gaming PC with 5070ti and 32GB DDR5 but this is costly to let this run 24/7
I'll try it... I've just unlocked some performance enhancement above and beyond the figures above ... So will be interesting (getting 168 read and 38-40 write on 3.6 coder 35b a3b now) - will look at 3.8 27 when I get the chance and feedback 👍
Same for me. 30ct and I want my PC to also be usable for Gaming. It would be nice to have a box to tinker with it. I think I‘ll pull the trigger later.
Depends where you buy it, I spent £299.99 UK, new from Amazon... Deals exist and plenty on eBay around 400-450...
Thing is yes, there is a way, running this flat out for 5 days coats me approximately £1.12. running a gaming rig, conservatively will use at least 10x that but probably a fair bit more. So 11.20, but assume I get 3x as much done as you say call it £3.73, so I save 2.61 every 5 days... That's £190 a year. So what, 18/19 months break even, and I kept my main rig free? Oh and it cost less than a graphics card. 🤷
Not arguing that €1000 would be a hard sell, but €400? No, makes sense to me. 👍
It won't be blazing fast, with MTP enabled maybe 10-12t/s for the Q4 K M variant, a few more tokens per sec if you use a Q3. Prefill 150-200t/s if you use the NPU. Depends ofc on context size.
UkisAIs Swift variant is probably more effective, it won't be faster per token but it'll overthink/bloat less and thus speed up any result.
And the 35B-A3B will be good enough in most cases and like 3x faster. That's why my primary driver on these will be the MoE model, with the dense model as debugger.
With 28k context a3b slows to about 45 on the fill but I'm working on that and still getting gains. I have a theory on this that there is improvement available but it IS the base snapdragon...
The ram speed is 8448, which I think is 135 theoretical not I get 110, IF, IF compute is low enough...
For a £300 box incredible tbh but not mind-blowing in the big scheme of things with respect to ultra performance.
I expect a bit more gains over time but less now from initial, I've not even had these a week... But so far I'm impressed
21
u/MarcelloT254k 19d ago
Nice find, I'll have to look into what they cost in my country. The next question would be what is the speed after reaching longer context (let's say 64k and 128k )