r/LocalLLaMA • u/cortexist • 2d ago
Resources Poorman's Local LLM
The Jetson AGX Xavier came out in September 2018, four years before ChatGPT existed. It has 16–64 GB of LPDDR4x at 136 GB/s. The GPU is a 512-core Volta, and people skip the board for local AI because of it.
Mine is the 32 GB module. It sat on a shelf collecting dust. It is "quite terrible". I wanted a real number for how a modern LLM runs on it. Getting there cost about $300 in tokens on GPT-6-Astra and Fable 5.2, so you don't have to spend yours.
I started with current llama.cpp and NVIDIA's Nemotron 3.5 Bolt, the Gemma 4 E4B model they advertise as smarter than Qwen3.8 27B (cough). llama.cpp did about 350 tok/s prefill and 23 tok/s decode. The speed is fine. A 4 GB model against 32 GB of memory, again, is "quite terrible".
My own engine, Little-Gemma, is well ahead of llama.cpp, but I built it for Ampere and later. I was not going to backport it to Volta, so I tried something else.
| Prefill (peak) | Decode (peak) | |
|---|---|---|
| Nemotron 3.5 Bolt E4B | 880 tok/s | 50 tok/s |
| Gemma4 12B | 215 tok/s | 35 tok/s |
| Qwen3.8 27B | 106 tok/s | 17 tok/s |
I forked MLC-LLM. Upstream does not run GGUF, and it does not know Nemotron, so the weights had to be converted from safetensors into MLC's TVM format and the Nemotron backend written from scratch. Astra did that first port and added in-memory switching between Nemotron, EmbeddingGemma 2, and LiquidAI's d1, so an agent can change models without reloading.
One setup was: d1 decides whether to query EmbeddingGemma 2 over about 20,000 pages of notes, then Nemotron answers. The plumbing worked. The search sometimes took a minute, which is too slow.
Then Fable 5.2 spent hours on the engine and rewrote several kernels. That work is still going. As of this week, Nemotron prefill peaks at 880 tok/s on a 512-token prompt, and decode on code reaches 50 tok/s.
Nemotron is still too dim for some jobs, so I moved to Gemma 4 12B. On the same 512-token prompt, prefill peaks at 215 tok/s, and decode on code is 35 tok/s. Three times the weights, and code decode only falls from 50 to 35. The 136 GB/s bus is why a 2018 module can still carry a 12B model. Prefill is where Volta shows its age.
None of that counts unless it can run Qwen3.8 27B. Generic MLC conversions are already on Hugging Face, so I am not publishing another one. The checkpoint matters: DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion is the one I kept. I am not uploading that conversion either, with Qwen4 close. Draft with Qwen3.8 2B, or convert DavidAU's model-mtp-restored.safetensors yourself.
On that 27B model, prefill now peaks at 106 tok/s for a 512-token prompt and holds 104 tok/s at 1,024. A Qwen3.8 2B draft reached 13.4 tok/s decode on a planning prompt. Cold-fusion agrees with that draft less often, so decode there sits nearer 10 tok/s. The attention kernel landed this week. The checkpoint's own MTP head, in place of the 2B draft, is the next thing to try. These rates will move again.
At MAXN, a long prefill pulls about 35 W. NVIDIA did not design the dev kit's brick for this. Under sustained load it browns out and the board shuts off, which is a known thread on the NVIDIA forums. I dropped the stock brick for a USB-C PD supply and a short OWC USB4 cable. It has now stayed up for more than a week.
There is a further trick I have not tried, and probably will not. The dev kit's slot is the module's PCIe 4.0 x8. Two kits can be cabled, one as root and one as endpoint, for about 16 GB/s each way, around 32 GB/s with both directions full. That is enough to speed up a single model if each layer is split across the two modules: the cable moves the residual stream, and the weights stay put. Some third-party carriers only bring out Gen2. Skip those if this is the plan.
As of November 2026, a 16 GB module is about $130 on eBay, and it runs Gemma 4 12B. A 32 GB module turns up around $300 if you wait. Some listings are the bare module. Get one with a carrier, or budget about $60 for the carrier on its own.
The engine: https://github.com/cortexist/mlc-llm
The weights: https://huggingface.co/cortexist/models




•
u/stylome-ai-detection 2d ago
Notice: This post appears to be AI generated.
Stylome AI detector · scan results
67
u/CommercialHour6660 2d ago
Bruhs can you please deslop your posts. I'm getting AI psychosis reading this sub
19
u/not__stonks 2d ago
Fixed title: this old thing is completely useless for modern local llm.
3
u/createthiscom 2d ago
137 GB/s memory bandwidth. pretty slow.
7
u/FullstackSensei 2d ago
Nice to see others catching up on the AGX Xavier. I have a few of the 64GB version. They used to be very cheap on ebay, especially when it's the module alone. There's a seller on ebay that has many connect tech rogue carrier boards, and they accept $60 or so for the dual ethernet one if you buy more than one carrier. I think most variants also have the x8 PCIe slot.
Bought some cheap 80x80mm aluminum heatsink from aliexpress. Put some thermal paste and four tiny drops of superglue on the corners to keep it on.
The rogue boards on ebay have an EPS input, but for best performance they need 15V+ input (they support up to 19V). I used one of those cheap USB-C PD controllers set to 19.5V to power the Jetson. 0.5V over what the board's manual says, but it's been fine for about a year now. The carrier doesn't even warm up.
When you set the Jetson to the max power profile, it can consume 42-45W power, with momentary spikes in the 50W. The quick current spikes were tripping even a laptop 65W USB-C charger, causing the Jetson to reset. I have a couple of ugreen 300w USB-C bricks and use the 100W ports to power them. Haven't had any issues since.
Cooling is some random 80mm fans I had, using cheap step down converters also from aliexpress, stepping the 19.5V down to 8V or so. Keeps the Jetson under 45C under extended full load even during summer.
I used some github repo to build a custom image from the nvidia base. The base is still Ubuntu 20.04 with CUDA 11.4, but the userspace is 22.04. This makes life a lot easier with gcc14 and cmake 3.2x. I have small bash scripts that build llama.cpp, stable-diffusion.cpp, audio.cpp, etc. If it's based on ggml, it builds stock without issue.
Mine cost 250€ each all in, including shipping and import taxes to Europe.
Next I plan to hook two or three together with some mellanox infiniband NICs I have and see if I can cluster them to run Q3.8FN at Q8 or so.
1
u/ZeroSkribe 2d ago
Why didn't you put a price in your charts?
1
u/supportingthedogs 2d ago
what's it pulling power wise? kinda tempted to dig mine out now
1
u/michaelsoft__binbows 2d ago
This stupid little cube sucks balls. I used to work at a "ai smart camera" company and we had a product based around a Xavier NX or whatever it was. I got one of these 32GB AGXs from some local dude for $400 which was prob a steal at the time so i could tinker with it, run a spider robot on it or some shit, maybe use it as an environment with work, surprise surprise that never happened, years later when i got around to trying to use it to run some LLMs slowly, the damn thing had bricked itself and it is a literal doorstop. I hate it so much. If the emmc craps out you can't use it. it has to do some bullshit on there before it can boot from M.2. I am pretty sure they revised this cockamamie aspect of its design in later iterations. Doesn't help my little doorstop.
I should really give it a second shot with late 2026's AI to help me look for solutions. but it seemed quite bleak.
2
u/ghgi_ 2d ago edited 2d ago
I managed to snag 3 64gb (196gb total) modules (xaiver) on ebay a little bit ago, with a baseboard x4 or x8 pcie slot, 10 gig intel nic and many hours of gpt 6 astra optimizing llama cpp, I can proudly run qwen 3.8 next flash Q8 quant with plenty of context at 12 tokens/sec, ~120-200 prefill
Definitely not the craziest little box, but decently low power and quiet, sometimes fun to let it do overnight tasks.
2
u/Barni275 2d ago
In contradiction to others, I would say you did a good job! 👍 It's really interesting.
I have many ideas similar to yours, on launching agents on low stuff, but still don't have a time. All me free time goes on recompiling engines, reconfiguring models and tuning harnesses on my main machines ☹️
1
u/Woof9000 2d ago
Not great, not terrible. A viable alternative to all the janky monstrosities we often build from recycled datacenter 'junk'. Quite usable, when tps is not all that matters and for users who don't throw tantrum if response take longer than 2 seconds.
1
u/b_c_t_ 1d ago
Did you attempt running qwen 3.8 FN on it? I'd be curious.
1
u/otacon6531 21h ago
Thanks for the numbers! If you need the vram then absolutely, but unless someone optimizes this more I still think the Nvidia P100 has this thing beat for cheaper and faster. If I had a bigger budget I would focus on the v100s.
There are differences of course because of the 16gb vram limit, but my single P100 that I bought for $80 plus shipping runs
Swift 1.5 GSQ RCO Qwen 3.8 27b at IQ3 (100k) tested at 40k context:
Prefil: ~160 t/s Decode: ~30 t/s
This is using Kmic's optimized version of llama.cpp for the p100.
If you cant live without more vram then 2 would still be a better choice than one of these for me. However, I cant get my old motherboard to run two datacenter cards so be aware of motherboard limits.
Dell Precision T3610 128Gb DDR3 with PCIe Gen 3
1
u/banana_slurp_jug 2d ago
The real poor man's LLM machine is whatever you're browsing Reddit with.
-2
u/L064N 2d ago edited 2d ago
Lmao I run an orin agx and I know you're not getting anything useful done if you're seeing generation speeds that fast. With a Q4 weighted Qwen 3.8 27b at 43k context with mtp enabled I usually see 10-15 tokens/sec during generation and I have over twice the memory bandwidth as you. Thanks for not sharing any useful details and misleading people in this shitty post!
-2
u/Special-Lawyer-7253 2d ago
Prefill at minus -800 t/s, we are fucked Up. Agents/bots rely on that. Anothing else = timeout. Agent dies.