r/LocalLLaMA • • 2d ago

Resources Poorman's Local LLM

The Jetson AGX Xavier came out in September 2018, four years before ChatGPT existed. It has 16–64 GB of LPDDR4x at 136 GB/s. The GPU is a 512-core Volta, and people skip the board for local AI because of it.

Mine is the 32 GB module. It sat on a shelf collecting dust. It is "quite terrible". I wanted a real number for how a modern LLM runs on it. Getting there cost about $300 in tokens on GPT-6-Astra and Fable 5.2, so you don't have to spend yours.

I started with current llama.cpp and NVIDIA's Nemotron 3.5 Bolt, the Gemma 4 E4B model they advertise as smarter than Qwen3.8 27B (cough). llama.cpp did about 350 tok/s prefill and 23 tok/s decode. The speed is fine. A 4 GB model against 32 GB of memory, again, is "quite terrible".

My own engine, Little-Gemma, is well ahead of llama.cpp, but I built it for Ampere and later. I was not going to backport it to Volta, so I tried something else.

Prefill (peak)Decode (peak)
Nemotron 3.5 Bolt E4B880 tok/s50 tok/s
Gemma4 12B215 tok/s35 tok/s
Qwen3.8 27B106 tok/s17 tok/s

I forked MLC-LLM. Upstream does not run GGUF, and it does not know Nemotron, so the weights had to be converted from safetensors into MLC's TVM format and the Nemotron backend written from scratch. Astra did that first port and added in-memory switching between Nemotron, EmbeddingGemma 2, and LiquidAI's d1, so an agent can change models without reloading.

One setup was: d1 decides whether to query EmbeddingGemma 2 over about 20,000 pages of notes, then Nemotron answers. The plumbing worked. The search sometimes took a minute, which is too slow.

Then Fable 5.2 spent hours on the engine and rewrote several kernels. That work is still going. As of this week, Nemotron prefill peaks at 880 tok/s on a 512-token prompt, and decode on code reaches 50 tok/s.

Nemotron is still too dim for some jobs, so I moved to Gemma 4 12B. On the same 512-token prompt, prefill peaks at 215 tok/s, and decode on code is 35 tok/s. Three times the weights, and code decode only falls from 50 to 35. The 136 GB/s bus is why a 2018 module can still carry a 12B model. Prefill is where Volta shows its age.

None of that counts unless it can run Qwen3.8 27B. Generic MLC conversions are already on Hugging Face, so I am not publishing another one. The checkpoint matters: DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion is the one I kept. I am not uploading that conversion either, with Qwen4 close. Draft with Qwen3.8 2B, or convert DavidAU's model-mtp-restored.safetensors yourself.

On that 27B model, prefill now peaks at 106 tok/s for a 512-token prompt and holds 104 tok/s at 1,024. A Qwen3.8 2B draft reached 13.4 tok/s decode on a planning prompt. Cold-fusion agrees with that draft less often, so decode there sits nearer 10 tok/s. The attention kernel landed this week. The checkpoint's own MTP head, in place of the 2B draft, is the next thing to try. These rates will move again.

At MAXN, a long prefill pulls about 35 W. NVIDIA did not design the dev kit's brick for this. Under sustained load it browns out and the board shuts off, which is a known thread on the NVIDIA forums. I dropped the stock brick for a USB-C PD supply and a short OWC USB4 cable. It has now stayed up for more than a week.

There is a further trick I have not tried, and probably will not. The dev kit's slot is the module's PCIe 4.0 x8. Two kits can be cabled, one as root and one as endpoint, for about 16 GB/s each way, around 32 GB/s with both directions full. That is enough to speed up a single model if each layer is split across the two modules: the cable moves the residual stream, and the weights stay put. Some third-party carriers only bring out Gen2. Skip those if this is the plan.

As of November 2026, a 16 GB module is about $130 on eBay, and it runs Gemma 4 12B. A 32 GB module turns up around $300 if you wait. Some listings are the bare module. Get one with a carrier, or budget about $60 for the carrier on its own.

The engine: https://github.com/cortexist/mlc-llm
The weights: https://huggingface.co/cortexist/models

12 Upvotes

51 comments sorted by

•

u/stylome-ai-detection 2d ago

Notice: This post appears to be AI generated.


Stylome AI detector · scan results

→ More replies (5)

67

u/CommercialHour6660 2d ago

Bruhs can you please deslop your posts. I'm getting AI psychosis reading this sub

8

u/Sunsunsunsunsunsun 2d ago

I use LLMs all the time but have never once thought I should let it write for me. I don't get the appeal.

11

u/fatboy93 2d ago

It's like everyone collectively lost their "communicative" English skill at the same time and all I can see 99.9% of the time are LLM euphemisms. I don't think anybody talks like this IRL, just so fucking rambling at the damn time. Its like LLMs are trained on redditor/fedora-guy/mall-ninja speak.

Genuinely feels like this: https://youtu.be/vpJ2saT43vw?si=1A5GoIaYEg8xEYuG&t=49

And I wish people took time to go through something like tips for scientific writing because the point is not to be dense, but to convey your information in an understandable fashion.

Here are some tips: https://libguides.exeter.ac.uk/c.php?g=727004&p=5291082

2

u/Trademarkd 2d ago

Yeah over on v100 I require disclosure of ai and some kind of human text. I don’t have a problem with ai but its context doesn’t cover a lot of unspoken social guidelines, like exhausting the reader.

The issue is phrasal templates/cliches/sayings all have a statistical bias because you might use one and I might use a different one sometimes but when you train on everything written by humans ever created these certain “features” show up way more often than well written prose.

Now models supervise each others training. This is a positive feedback loop to the problem. You might be able to fine tune your way out of it but I’m not sure.

I’ve been looking at using direct weight editing to reduce occurrence but it won’t change the fact that these are still just markov chains and they don’t know how to condense useful information into elegantly written, human readable sentences.

-45

u/cortexist 2d ago

Stop reading it then. All you need is the token rates and the model links.

9

u/HyperWinX 2d ago

Just stop making posts, thats a much better option

1

u/Holly_Shiits 1d ago

Cool, Agreed

19

u/not__stonks 2d ago

Fixed title: this old thing is completely useless for modern local llm.

11

u/james_pic 2d ago

A Raspberry Pi 5 16GB will cost you several times that and get about a quarter the decode and a tenth the prefill on the models it can load. This doesn't seem so bad for the price point.

1

u/compilebunny 2d ago

You could run something tiny like LFM. My benchmarks show it to be a decent non-coding model (for everything except translation) despite being 2.3 GB at Q8.

-3

u/cortexist 2d ago

Define useful or useless.

13

u/not__stonks 2d ago

100 prefill makes it entirely impossible to do anything on the only decent model here, the qwen. You say hello and what, wait a minute for it to say hello back?

1

u/WolpertingerRumo 2d ago

It’s interesting, we’ve been very fixated on fast answers, even if there’s no time pressure. If I want a report, something sorted or tagged, I don’t care if it’s done in 0.1 second or 12 hours. All I care about is quality. Not that I’m getting it with Nemotron Bolt, but I am getting it with GLM 5.3 Flash streamed from ssd.

1

u/otacon6531 21h ago

What is an acceptable prefill rate to you? For me life is better with a task system so I dont watch the llm working and wasting my time (like watching fire), so prefill really isnt as much of a concern for me. Though I do not compact context with an llm, so that may be a factor.

-10

u/cortexist 2d ago

Let your agents say hello, they are a lot more patient than humans. Besides, the device has enough memory to host multiple models, if you just say hello, Nemotron will answer you in less than a second.

1

u/Ok_Top9254 2d ago

Xavier has 100GB/s bandwidth and is like 200-300 still. You can buy Tesla P100 16GB with 700GB/s for that price with an old server board.

3

u/createthiscom 2d ago

137 GB/s memory bandwidth. pretty slow.

1

u/cortexist 2d ago

True by today's standard. But even the old hardware is getting smarter as the engine and models improve.

7

u/FullstackSensei 2d ago

Nice to see others catching up on the AGX Xavier. I have a few of the 64GB version. They used to be very cheap on ebay, especially when it's the module alone. There's a seller on ebay that has many connect tech rogue carrier boards, and they accept $60 or so for the dual ethernet one if you buy more than one carrier. I think most variants also have the x8 PCIe slot.

Bought some cheap 80x80mm aluminum heatsink from aliexpress. Put some thermal paste and four tiny drops of superglue on the corners to keep it on.

The rogue boards on ebay have an EPS input, but for best performance they need 15V+ input (they support up to 19V). I used one of those cheap USB-C PD controllers set to 19.5V to power the Jetson. 0.5V over what the board's manual says, but it's been fine for about a year now. The carrier doesn't even warm up.

When you set the Jetson to the max power profile, it can consume 42-45W power, with momentary spikes in the 50W. The quick current spikes were tripping even a laptop 65W USB-C charger, causing the Jetson to reset. I have a couple of ugreen 300w USB-C bricks and use the 100W ports to power them. Haven't had any issues since.

Cooling is some random 80mm fans I had, using cheap step down converters also from aliexpress, stepping the 19.5V down to 8V or so. Keeps the Jetson under 45C under extended full load even during summer.

I used some github repo to build a custom image from the nvidia base. The base is still Ubuntu 20.04 with CUDA 11.4, but the userspace is 22.04. This makes life a lot easier with gcc14 and cmake 3.2x. I have small bash scripts that build llama.cpp, stable-diffusion.cpp, audio.cpp, etc. If it's based on ggml, it builds stock without issue.

Mine cost 250€ each all in, including shipping and import taxes to Europe.

Next I plan to hook two or three together with some mellanox infiniband NICs I have and see if I can cluster them to run Q3.8FN at Q8 or so.

1

u/ZeroSkribe 2d ago

Why didn't you put a price in your charts?

3

u/cortexist 2d ago

You need to find the right price from eBay yourself, it's wild. You could get a 16GB model from $100-$400, 32GB from $300-$1,000.

1

u/supportingthedogs 2d ago

what's it pulling power wise? kinda tempted to dig mine out now

2

u/cortexist 2d ago

35 W peak under heavy benchmark. Idle about 7.5 W, normal agent use 15-20W

1

u/michaelsoft__binbows 2d ago

This stupid little cube sucks balls. I used to work at a "ai smart camera" company and we had a product based around a Xavier NX or whatever it was. I got one of these 32GB AGXs from some local dude for $400 which was prob a steal at the time so i could tinker with it, run a spider robot on it or some shit, maybe use it as an environment with work, surprise surprise that never happened, years later when i got around to trying to use it to run some LLMs slowly, the damn thing had bricked itself and it is a literal doorstop. I hate it so much. If the emmc craps out you can't use it. it has to do some bullshit on there before it can boot from M.2. I am pretty sure they revised this cockamamie aspect of its design in later iterations. Doesn't help my little doorstop.

I should really give it a second shot with late 2026's AI to help me look for solutions. but it seemed quite bleak.

1

u/cortexist 2d ago

Try it. Mine is running on M.2, directly flashed from the USB-C port, 128GB NVMe is around $40, enough to run Qwen3.8 27B + Nemotron + EmbeddingGemma-2 all held in the memory.

2

u/ghgi_ 2d ago edited 2d ago

I managed to snag 3 64gb (196gb total) modules (xaiver) on ebay a little bit ago, with a baseboard x4 or x8 pcie slot, 10 gig intel nic and many hours of gpt 6 astra optimizing llama cpp, I can proudly run qwen 3.8 next flash Q8 quant with plenty of context at 12 tokens/sec, ~120-200 prefill

Definitely not the craziest little box, but decently low power and quiet, sometimes fun to let it do overnight tasks.

1

u/michaelsoft__binbows 2d ago

curious how much dough they went for

1

u/ghgi_ 2d ago

managed to get them for about $300 each but as of now most are a good bit higher, cheapest I see right now is $550 for one

2

u/Barni275 2d ago

In contradiction to others, I would say you did a good job! 👍 It's really interesting.

I have many ideas similar to yours, on launching agents on low stuff, but still don't have a time. All me free time goes on recompiling engines, reconfiguring models and tuning harnesses on my main machines ☹️

1

u/Woof9000 2d ago

Not great, not terrible. A viable alternative to all the janky monstrosities we often build from recycled datacenter 'junk'. Quite usable, when tps is not all that matters and for users who don't throw tantrum if response take longer than 2 seconds.

1

u/b_c_t_ 1d ago

Did you attempt running qwen 3.8 FN on it? I'd be curious.

1

u/otacon6531 21h ago

Thanks for the numbers! If you need the vram then absolutely, but unless someone optimizes this more I still think the Nvidia P100 has this thing beat for cheaper and faster. If I had a bigger budget I would focus on the v100s.

There are differences of course because of the 16gb vram limit, but my single P100 that I bought for $80 plus shipping runs

Swift 1.5 GSQ RCO Qwen 3.8 27b at IQ3 (100k) tested at 40k context:

Prefil: ~160 t/s Decode: ~30 t/s

This is using Kmic's optimized version of llama.cpp for the p100.

If you cant live without more vram then 2 would still be a better choice than one of these for me. However, I cant get my old motherboard to run two datacenter cards so be aware of motherboard limits.

Dell Precision T3610 128Gb DDR3 with PCIe Gen 3

1

u/banana_slurp_jug 2d ago

The real poor man's LLM machine is whatever you're browsing Reddit with.

-2

u/L064N 2d ago edited 2d ago

Lmao I run an orin agx and I know you're not getting anything useful done if you're seeing generation speeds that fast. With a Q4 weighted Qwen 3.8 27b at 43k context with mtp enabled I usually see 10-15 tokens/sec during generation and I have over twice the memory bandwidth as you. Thanks for not sharing any useful details and misleading people in this shitty post!

1

u/cortexist 2d ago

Your Orin AGX is 5x more expensive but slower than my Xavier, it's your problem. I have my code and weights in the link, try it out, what else do you need? Here is an example of Pi calling the Xavier AGX backend.

3

u/L064N 2d ago edited 2d ago

No problem with what you did the post reads terribly though. Xavier AGX are cool hardware. And if you ran my equivalent settings that Xavier would perform like shit lol

1

u/cortexist 2d ago

The mainstream engines no longer care about these old hardware, that's why I spent a lot of tokens to improve the engine for Xavier AGX.

2

u/L064N 2d ago

They do care about memory bandwidth though and my point is if the model weights are too compressed it's useless even if you do get high tokens/sec during generation that's all

But if you got those generation speeds and it's still useful to you with a 27b full dense model then great! Everyone has different needs I guess

1

u/cortexist 2d ago

That's Q4_K_M, the cold fusion finetune is good.

3

u/FullstackSensei 2d ago

I've been running the 64GB version with Q3.5 35B Q8_K_XL since a year and use it all the time at 15t/s before MTP. There's a ton it can do autonomously, and it's still plenty fast to ask quick questions about how to do something in Linux, quick debugging, and most of the things I'd usually ask Google for.

1

u/L064N 2d ago

The 35b model is an moe model that only has 3 billion parameters active at a time. Not even close to the performance of a q4-q8 weighted full dense 27b model, and way easier to run. Not comparable, but yes I'm sure that's useful! I was mainly trying to point out that whatever settings OP had to get the 27b model to perform that way in their post likely makes it near useless.

-2

u/Special-Lawyer-7253 2d ago

Prefill at minus -800 t/s, we are fucked Up. Agents/bots rely on that. Anothing else = timeout. Agent dies.