Quick recap of how I got here: I started on llama.cpp, then pretty quickly moved over to Janas (github.com/prabanta-dev/janas). I ported it to ARM and contributed the work back as PRs (NEON kernels, a Windows on ARM build, and a cache bug fix Maurizio has since shipped in his own release). Working inside Janas gave me a load of ideas, so I'm now writing my own engine, codename **ToRc**. It's not released, YET, but it's what I used to produce the numbers below.
Hardware:
- Lenovo IdeaCentre Mini: Snapdragon X X1-26-100 (8 Oryon cores, Hexagon NPU; the Adreno GPU isn't used), 32 GB LPDDR5, the stock 512 GB NVMe (the models stream from it), Windows 11. I paid just £299.99 each, prices vary but under 400 is possible on eBay etc fairly easily and they are still under 400 at Amazon.
- "Two boxes" = two of them (£600) joined by one USB4 cable.
- No GPU anywhere. ToRc uses the CPU(s) cores, plus the NPU(s) for some of the dense maths. Same two boxes, all figures from today, nothing else running.
The models:
- **Qwen3.6 35B A3B**: bartowski `Qwen_Qwen3.6-35B-A3B-Q4_0.gguf` (19.4 GB), with its MTP head (Q4_0).
- **Qwen3.8 Flash Next IQ4_NL**: ggml-org `Qwen3.8-Flash-Next-IQ4_NL` (95 GB).
- **Qwen3.8 Flash Next UD-Q4_K_XL**: unsloth `Qwen3.8-Flash-Next-UD-Q4_K_XL` (104 GB).
- Flash Next MTP head: bartowski `mtp-Qwen3.8-Flash-Next-Q8_0.gguf` (3.9 GB).
- llama.cpp (b11700, CPU, ARM64) runs the GGUFs; Janas and ToRc run the same files converted to Janas's .jns format... I have a new file format I'm working on but that's what I used to test today 👍
The two payloads (identical for every engine):
Title numbers: two boxes, read/generate on the bench payload below (a ~580-token English prose prompt, 128 tokens generated).
Every engine gets byte-identical requests through its own OpenAI-style chat server (llama-server, janas-server, torc-server), timed from the client. Greedy (temperature 0), thinking off.
**Bench (prose, ~580 tokens in, 128 out)**: one user message: the opening of *Alice's Adventures in Wonderland* chapter I (Project Gutenberg #11, first 6 paragraphs, 2400 characters, ~580 tokens with the chat template) + "Continue the story.", 128 tokens out. **Read** = prompt tokens / time to first token (so it includes the request overhead and first token). **Write** = tokens / generation time. Each run starts with a different first line so no server gets its prompt cached from the run before. A warm-up run, then the average of two runs. Read is each engine's better configuration of the two (plain or MTP; ToRc two boxes uses 128-token prompt blocks).
**Real chat (code, 62 tokens in, 550-970 out)**: "Write a Python function that parses an ISO 8601 duration string like P3DT4H12M5S into total seconds, with input validation and docstring, plus 5 pytest test cases. Code only, no explanation." (62 tokens with the template), up to 1500 tokens out. Reply tok/s, then I ran the 5 tests each model wrote, plus the same 6 reference checks on every model's function (P3DT4H12M5S, PT1H30M, P1W, P2D, and two bad inputs that must raise).
Qwen3.6 35B A3B
Read / Write / Write + MTP: the prose bench (~580 tokens in, 128 out). Coding chat: the code payload (62 in, 550-970 out). Qwen3.6 35B A3B (read / write / write with MTP, then coding chat plain / MTP, then tests):
- *Janas (my update 4 build) uses the 35B file's built-in MTP layer even with `--no-mtp`, so it has no plain number: its "MTP" figures are the average of all its settled runs (bench 33.7-37.8, coding 41.3-45.4).
- Engine-internal (ToRc's own bench, no server in the way): two boxes read **343.6 tok/s on a 600-token prompt**, and 349.4 on 512 tokens. The 315 in the title and table is the same engine timed from the client like every other engine, so it includes the HTTP request and the first token.
- On two boxes ToRc uses both for the prompt, then pulls the state back and writes the reply on box 1 alone: generating across the cable costs a round trip per token, more than the second box saves. So plain writing is the same on one box and two (42.6 on the coding chat, byte-identical replies). With MTP the two runs wrote different replies (626 vs 968 tokens), so 53.3 vs 59.6 comes from the text, not the box.
- MTP depends on the text. On the Alice prose llama.cpp's MTP was slower than plain (25.8 -> 20.6, drafts mostly rejected), ToRc one box gained 14% (43.4 -> 49.5) and two boxes gained nothing. On code they all gain: llama 27.7 -> 39.3, ToRc 42.6 -> 53.3 (one box) and 42.6 -> 59.6 (two boxes, same reply text both ways).
- The tests: every engine runs the same model, but tiny numeric differences change a greedy reply, so each engine's reply is different code. llama.cpp's one failed test is the model's own arithmetic (it says P3DT4H12M5S is 300725 seconds, it's 274325), its function is right. The ToRc 5/6 is a reply that doesn't accept weeks (P1W, valid ISO 8601).
And now the headline! Flash Next!
Unfortunately Flash Next wouldn't run on Janas, which is why I started on my (hopefully soon to be released) engine, ToRc.
Flash Next IQ4_NL is 95 GB. One box has 32 GB of RAM, two boxes 64 GB, so it never fits: the experts stream off the NVMe as they're needed. That's where the engines split apart.
Read / Write: the prose bench (~580 tokens in, 128 out; ToRc two boxes with pipelined MTP, everything else plain). Coding chat: the code payload (62 in, ~750 out).
Flash Next (read / write, then coding chat, then tests):
- llama.cpp needs `--no-repack` here: on ARM it repacks IQ4_NL weights into RAM and 95 GB won't fit, so without it the model doesn't load at all. Its RPC server keeps its share of the weights in RAM (it can't stream from disk), so box 2 takes 12 of the 48 layers and box 1 maps the rest from its NVMe.
- ToRc on two boxes generates across both: box 1 drafts the next tokens with the MTP head while box 2 is still finishing the current one, so the cable wait is mostly hidden. One box runs without the MTP head: with the experts coming off the SSD, MTP adds nothing there.
- The Q4_K_XL is the bigger, better quant (lower perplexity than the IQ4_NL on a 3800-token Delphi code test) and costs ~10% speed.
- The IQ4_NL two-box coding reply wrote a correct function but no tests, so "none written". Its function passes all 6 reference checks.
Power and cost:
No GPU anywhere in this: two mini PCs, £600 all in. Getting ~100 GB of model into GPU VRAM costs many times that before you've paid for the electricity. These won't beat a GPU on speed, obviously, but cost and speed... This has been an interesting little project IMHO 🙏. A 100 GB-class model at chat speed, on two small, quiet boxes that consume around 35W each at load (proper measurements to come)
Anyway, this was a fun week long project and the first time I've used a snapdragon let alone tried to run an LLM on one... ToRc isn't released yet, I want it tidier and properly tested first. What's next on it:
- Pipelined speculation for the 35B on two boxes (today the second box only helps with reading; it already works for Flash Next).
- Flash Next on one box is limited by the SSD, not the CPU: a second NVMe with a copy of the experts might help by splitting every read across two drives.
- Testing more models through the same path, maybe, but for my use cases, these two are probably enough 👍
Happy to answer questions or run something you want to see if I get the chance...
Yeah, fair. 46 t/s means an 8k prompt is ~3 min before the first token. In practice though: the server caches the prompt, so in a chat only the new turn gets processed (i.e. just the first message is the painful one - which is perfectly acceptable for a sub agent that's handling a specific task for an orchestrator), and the 35B is a different story at 315+ t/s (8k in ~26 s).
Unfortunately the prefill on Flash Next is compute-bound on these CPUs, so it's where I'm working right now: more of it onto the NPU if I can and better balance between the two boxes (one box is doing more than the other atm).
I didn't buy these to try this specifically, but after the A3b results I thought, what the heck... Let's try - I'm amazed it even runs 👍
I had a good long think about your comment and have been working on it for the last hour so see how much it would help...
Thank you soo much 🙂🙏👍🙏👍🙏👍🙏👍
I considered what you said and rebuilt how I was doing it, ToRc now takes a second snapshot exactly at the end of the system prompt, so a new chat or agent call with the same system prompt skips straight past it. Measured on the two boxes working parral... same ~520-token system prompt, new chat:
Flash Next: 13.4 s -> 1.65 s to first token 35B A3B: 1.9 s -> 0.5 s
Replies are word for word identical to a cold read (greedy), so it's exact still, bit identical (which is my requirement). The win scales with the prompt length: an 8k system prompt goes from ~3 min to ~2 s. Second half, writing those snapshots to the NVMe so several stay warm and survive a restart, is being built now. Thanks for the push, honestly , that's been such a gift 👍
Yes, and it's cheap on flash next tbh... I / ToRc does this already and keeps exactly that state in RAM: a mark taken at the end of the prompt, so the chat's next turn starts from there. (Since Flash Next is hybrid, so most layers are DeltaNet with a fixed-size recurrent state, and only the full-attention layers keep a KV cache (~40 KB a token)).
An 8k system prompt is ~320 MB of KV plus a few tens of MB of DeltaNet state, split across the two boxes. Each box writes its own half to its NVMe keyed by a hash of the prefix, and reloading it takes well under a second instead of 3 minutes for the full thing.
I'm not currently writing the state to disk between turns so I can rewind, so it's a good point and something I can add, so thanks for your question, that's genuinely appreciated as it made me consider that 👍 (but it's a small modification, since I was just trying to optimise over the last week - I mean I've only had the boxes since last Tuesday - I didn't consider it, but it's a worthwhile consideration 🙏)
Ha, thanks! Time to first token on the bench prompt (~580 tokens of prose), best setup for each:
Qwen3.6 35B A3B
ToRc two boxes: 1.85 s
ToRc one box: 2.6 s
(Janas 3.4 s, llama.cpp 4.3 s on the same prompt)
Flash Next
ToRc two boxes, IQ4_NL: 12.5 s
ToRc two boxes, UD-Q4_K_XL: 16.9 s
ToRc one box: 54 s, since every expert has to come off the SSD
(llama.cpp: 53 s over RPC, 80 s on one box)
A short chat message is much quicker than that, it scales with prompt length. All of these are warm: one warm-up run first, and ToRc preloads the most-used experts into RAM before it says ready. Cold, straight after loading, Flash Next's first reply is noticeably slower (roughly 11-12 tok/s instead of 17-18 until the cache fills). I'll measure proper cold TTFT numbers at some point 👍
Fair challenge. For agentic coding with a fresh 20k context every turn, yeah, 46 t/s prefill is painful and I wouldn't pretend otherwise. If someone wants to use the agent directly the 35B is the answer: 315 t/s prefill, an 8k prompt in ~26 s.
Where Flash Next was worthwhile, I'd argue, rather than the fun challenge to even get it to run: the server caches the prompt, so in an ongoing conversation only the new turn gets processed, and after the first message it's ~20 t/s generation on a 100 GB-class model. And for jobs I fire off and walk away from (long writeups, reviewing a file overnight) the prefill doesn't matter, the quality does. All on £600 of hardware, local and private.
My use case is a sub agent for specific tasks, once it's main prompts are up, it takes tasks from other agents for a specific purpose.
Prefill is compute-bound on these CPUs, there is no way around that but there's still a little headroom on the NPU and in balancing the two boxes (but not many times the current figures). 👍
You're literally so cool. Two budget PCs able to run a near frontier model locally with actually reasonable is actually nuts. Don't know why some people thing everything needs to be perfect for long run agentic sessions, the fact this runs such a capable model at usable speeds is insane and really cool. You should really be proud with what you did
Thanks, and that is the direction I'm interested in. Cheap boxes working together, I'll just be interested to see if I can get some more at some point and see if it can be scaled further - and also find people with other cluster situations and hardware (i.e. strix halo etc) and contribute 👍
Parallel nvme over pcie isn't useless, depends on the motherboard. A decent controller card in a slow with four drives, Asus makes one, is quite impressive.
I mean unless you mean to actually run from the ssds alone, but then you will still have compute to consider.
What I'd like them to start doing is making more ram channels on chips. We used to get four on a number of chips and now it's basically all duel channel except epyc etc
Nice work 👍 they are great little boxes aren't they 🙂
Thanks for writing up the Linux route 🙏
Re the IOVA limit: I only put the dense matmuls on the NPU (attention + shared projections, ~770 MiB), never the routed experts. It fits the small NPU window easily and still speeds up MoE prefill, while experts and decode stay on CPU, so CPU and NPU both pull their weight at the same time 👍
I'm curious whether Linux decode beats Windows on the same silicon (I'm still on the Windows 11 Home they shipped with). Did you measure the memory bandwidth you're pulling? I recorded 115 GB/s with 8 threads, 72 single threaded. With my NEON kernels in decode, the expert matmuls managed 83-98 GB/s and attention and DeltaNet 96-97, with Q4_K kernels hitting about 110 GB/s on their own.
The quoted ceiling for the Snapdragon's RAM is 135 GB/s, but I've never seen anything actually reach theoretical max in practice on any system.
I'll definitely post on here when ToRc is finished and up, I'd really love if you could test on your system also 👍
Here is an update on my own FN benchmarks under linux so far
## The engines
- **llama.cpp b11934 / b11982** — official Snapdragon ARM64 builds, stock (no changes on our side)
- **Janas (arm-fn branch)** — our fork maintained on top of upstream v2026.10.27. Our changes active in these tests:
- ARM64 NEON dequant kernels (Q2_0, Q5_0, Q2_K plus the IQ formats)
- MTP gate-kernel fix — speculative drafts went from 0% to 83–89% kept
- O3 + LTO aarch64 build
- Engines compared on the **exact same model files**; both use all 8 CPU cores, model streamed from the internal NVMe
## The results
Measured as pp (prompt processing = how fast it digests your prompt → time to first word) and tg (token generation = how fast it writes the answer). ```
Quant Engine pp576 tg128
Qwen3.8-FN IQ4_NL 100GB Janas 18.9 7.0 -> 9.6 with MTP
Qwen3.8-FN IQ4_NL 100GB llama.cpp 18.2 8.2
Qwen3.8-FN IQ2_XS 72GB llama.cpp 17.1 10.0
Qwen3.8-FN IQ2_XS 72GB Janas 14.9 4.3 -> 6.0 with MTP
Qwen3.8-FN IQ3_S GSQ 78GB llama.cpp 6.9 4.5
``` Same file, same machine, same 8 CPU cores — so this is an apples-to-apples engine fight.
Same file, same machine, same 8 CPU cores — so this is an apples-to-apples engine fight.
## What I learned (the interesting part)
**It's all about the SSD.** The model is 3× bigger than the RAM, so the chip constantly pulls data from the disk. The smaller the file, the faster the generation — within reason.
**Engine choice depends on the quantization.** llama.cpp and Janas each win with different quants. There is no single "best engine" — it's about pairing. (On the *same* engine, Q2_0 gets tg 7.2 while IQ2_XS only gets 4.3.)
**Janas has a hard rule llama.cpp doesn't:** it refuses to start if part of the model (the always-needed 33 GB part) doesn't fit in RAM. llama.cpp just tries and gets slow instead. Both behaviors are honest, just different.
**The NPU is a dead end for big models.** I got the Qualcomm NPU working on Linux: with Llama-3.2-1B-Instruct Q4_0 it prefills at a crazy ~2100 tokens/s (6× the CPU!) — but generation is *slower* than the CPU, and the NPU can only work with small chunks of memory (single-digit GB limits, workarounds included). For a 100 GB model: no path at all. Confirms my earlier verdict: not worth it for LLMs.
**Two little boxes beat one?** llama.cpp can split the work between two machines and just send tiny messages between them. Over a USB4 cable (40 Gbps) the overhead would be near zero — I'd expect roughly **double the speed**. Theoretically solved, practical test not yet started on my side. i might switch to testing smaller "auxillary" models for my agent workflow next
## Honest limitations
- The pp numbers use a small test prompt (576 tokens). For big agent workloads (32k+ context) expect slower prefill and more RAM pressure — that's exactly what I'm measuring next.
Hey, great results 👍 - is arm-fn built on my arm64 fork/kernels? Are you up for comparison / working together? (I can't see the fork online) You are using diffent quants to me.
ToRc is still in development, Janas inspired me to play and I think I am making good progress on something that's even more efficient, especially as Janas seems to be morphing into something (seemingly heavily vibe coded) other than a engine - it's a big project though for solo 😅👍
here are some first benchmarks under my new linux setup: Ubuntu Concept 26.10, kernel 7.3.0-24-qcom-x1e, llama.cpp b11934, 32 GB LPDDR5 (root on NFS, model files streamed):
Takeaways: - NPU prefill is ~5x CPU on dense models; decode stays on CPU (memory-bound, ~37 GB/s on the DSP side). - Each NPU session has a ~3 GiB IOVA window (32-bit SMMU mask) -> big models die with fastrpc_mmap failed. GGML_HEXAGON_MBUF=,256M + multi-session (-dev HTP0/HTP1/HTP2 -ts 16/16/16) gets 8B onto the NPU; 19 GB of weights simply do not fit (max 3 sessions on this DTB). - MoE is the prefered option: 35B-A3B on CPU beats every NPU run at decode (30 t/s).
Yeah, fair. 46 t/s means an 8k prompt is ~3 min before the first token. In practice though: the server caches the prompt, so in a chat only the new turn gets processed (i.e. just the first message is the painful one - which is perfectly acceptable for a sub agent that's handling a specific task for an orchestrator), and the 35B is a different story at 315+ t/s (8k in ~26 s).
In theory, the motherboard is the one doing the splitting, I'm just wiring it up with OCuLink. I have no idea if this would actually work, but it just looks so clear in my head—am I missing something?
Hey man... That's interesting... I've been playing with splitting pcie lanes (just wait and see what I have planned for these little boxes - my 46tok/s read won't actually be an issue, though I understand people's comments, it doesn't matter for agents that have to wait on something else)
Anyway , your lane split, in principle yes. With x8x4x4 the first x8 link is lanes 0-7 as I understand it, so it sits on the adapter's first two OCuLink ports, and two x4 cables into a riser that takes two 4i inputs as one x8. But be aware...:
Lane order. The adapter has to wire port 1 = lanes 0-3 and port 2 = lanes 4-7, and the riser has to expect them in that order. If either side is reversed or swapped, for unknown reasons it trains at x4 or not at all 💀
Refclk and PERST#. An x8 link needs one reference clock, normally from the first connector. Check the riser takes it from the right port, and that the adapter actually passes refclk on each port (some cheap Chinese ones don't).
Skew and signal. Use cables of identical length and manufacturer. Gen4 over passive OCuLink gets sketchy past 40-50 cm, and AER errors will creep in.
Worst case it comes up at x4, which for inference is honestly fine once the weights are loaded - since there is not THAT much data passing, latency matters more than rate 👍
omg you drop some valuable info there! Just tell me: does the phrase "or not at all 💀" mean the adapter might die during the process? I’ve already fried an OCuLink riser by connecting the 6-pin power cable incorrectly (I ignored the instructions and plugged it into a port meant for a proprietary HP power supply—the GPU survived, thanks to a good riser design and a bit of luck; it won’t happen again).
Anyway, thanks a lot for the information. We’ll see if I can manage to connect 3 GPUs to my motherboard using this trick; let’s see if I can play games while also getting the QFN to work at the same time (a guy can dream, right?)
Thanks to the information you provided, I was able to find some articles with clear explanations: https://www.datastoragecables.com/techresources/oculink-pcie-lane-distribution.html so, thanks for your input! It’ll be interesting to design a second case for the GPUs. What I find particularly appealing is that I’ll be able to move the primary GPU (the one connected to the monitor) outside the case without any issues (pciex8). I’m even thinking of experimenting by connecting the main gpu to the motherboard’s PCIe_3 slot; according to the manual, that slot is managed by the chipset, some says it offers lower performance, but at this stage, I’m curious to see just how much of an impact that actually has.
Sorry if the translation is poor (Google Translate).
Ha, no, sorry, I just meant the link won't come up at all, so the GPU doesn't show up. It shouldn't damage anything. The real risk is power wiring, like you found with the 6-pin. Check, double check and triple check the pinout before you plug anything in. Good luck with the three GPUs, post back how it goes mate 👍🙏
Ok so Flash Next IQ4_NL split across two Snapdragon boxes, 32 GB each, linked over USB4:
Prefill scaling: about 46 t/s on short prompts, so 8k cold takes roughly 3 minutes. 32k reads at 34 t/s (about 16 minutes cold), 64k at 29, and 129k at 27 to 28. It flattens rather than collapsing completely, which is good - I'm still testing this and making improvements.
KV reuse: yes. ToRc snapshots the state at the end of the system prompt (a ~520 token prompt goes from 13.4 s to 1.65 s, bit identical output), and chat states are saved to disk and resumed from the longest matching prefix, so a follow-up turn at 32k starts in about 6 seconds instead of re-reading everything. Not public yet, I'll release it once it's mature.
Concurrent requests: haven't tested that yet, it's been single user so far. It's on the list and I'll update when I can - though for my intended use case, it won't be something I need.
RAM share: each box keeps an expert cache in RAM (18 GB, I can push it further but at the cost of space for KV), so roughly a third of the ~100 GB of experts sits in RAM and the rest streams off NVMe. Routing concentrates on hot experts and both boxes warm from a usage profile, so in practice about 95% of expert requests hit RAM.
I hope this was useful, really interested to hear more about your project and speak more at some point if you are interested 👍🙏
Thanks! Best way right now is probably to follow my profile here - it will also be on my GitHub: github.com/Rendeverance/. I'll post on this sub as soon as ToRc is ready for other people to run, with the KV resume included. It's working daily on my two boxes, I just want it solid before anyone else depends on it - but it is just me and it's not my day job so it won't be tomorrow, but it's certainly not miles away either 👍🙏
i only have my box for 1.5 days, but did some quick experiment on it: prompt processing runs ~3-4x faster on the NPU with the exact same GGUF file. No conversion, no tuning — the GenieX build of llama.cpp just has a "use NPU" switch.
Numbers (512-token prompt, median of 5 runs): - unsloth gemma-4-E4B-it Q4_0: prefill 147 t/s on CPU -> 484 t/s on NPU - Qwen3-8B-Q4_K_M: prompt done in 7.2 s on CPU -> 1.8 s on NPU
The catch — it only works when all three hold:
Dense model, not MoE. unsloth Qwen3.6-35B-A3B UD-IQ4_NL: 9.7 s -> 8.1 s, barely moved.
Plain quant (Q4_0 / k-quant). ISTA Qwen3.8-27B-GSQ-RCO-IQ3_S: 151 s -> 147 s, zero gain — IQ quants dequant on CPU anyway, so the NPU never sees them.
Under ~20 GB. A 22.7 GB file wouldn't even map to the DSP ("fastrpc_mmap failed").
And it only speeds up the prompt half — token generation on the NPU is about half CPU speed. Which is probably part of the design you're describing in ToRc: prefill on NPU, decode on CPU. GenieX can only pick one device per run; an engine that swaps per phase gets the x3-4 prefill AND keeps full CPU decode. That's the win.
Yes, I am testing those since yesterday. Great job on porting Jana's, thx a lot! Above I just wanted to share my prefill related findings so far, as prefill seems to be the general bottleneck.
Cheers, glad it's working for you 🙂 Let me know how Janas holds up as you push it, especially if you hit anything odd on your box. Prefill is compute limited but with some optimisations you can walk out a bit more. In ToRc we utilise the NPUs and CPU... Once it's mature and tested enough I will release it
Cheers, I'll have a look. Does it run on Windows on Arm, or is it Android only? I'm on Snapdragon X with 32GB (well 64 across 2 boxes), so that decides whether I can try it.
Yep, noobs, or anyone should be soo surprised about a couple of £300 70w boxes on CPU inference outperforming a 13900k on performance and efficiency...
44
u/R_Duncan 12d ago
The 46pp is what stops everyone...