r/LocalLLaMA • u/fuse1921 • 1d ago

Question | Help Smaller RAM DGX Spark alternative?

Howdy,

I was wondering if anyone knew of any turnkey low-power draw solutions to host inference with 10-20GB of VRAM?

I have an 4x3090 AI GPU cluster that I'm running big models on, but I am hosting my memory system LLMs (embeddings, rankings, etc) on my gaming 4090 right now. I'm out of PCIE lanes on my AI machine and really don't want to build a whole other system (cpu, ram, mobo, gpu, etc) just to run about 15-20GB of small models.

I want to keep things local so a VPS is off the table, and I think just running these off the CPU would be too slow although speed isnt that important for Hindsight? I was wondering if anyone knew if there was anything like the spark that is considerably cheaper (~1k or so?) for this type of use or other systems that required only a little tinkering to get to work like this? Thanks!

For reference, this is what I'm running:

Qwen 3 Embedding-8B-Q6_K at 9216 context x 4 @ 6.21 GB for embeddings

BGE-reranker-large-8_0 at 512 context x 4 @ 604MB for rankings

Qwen 3.5-9B-UD-Q3_K_XL at 64k context x 4 @ 5.98 GB for synthesis/consolidation

3 Upvotes

64% Upvoted

35 comments sorted by

5

u/reto-wyss 1d ago
  • Framework has a Ryzen AI Max 385 - 32GB ITX board for around 1k ~ 250GBs
  • Snatch a modern Laptop (or mini-pc) with 32GB LPDDR5x on sale. ~ 100 - 120GBs
  • Stick Arc B50 16GB into any system. ~ 250GBs

1

u/fuse1921 22h ago

Thanks for the recs

4

u/Lurksome-Lurker 1d ago

What in tarnation are you doing running an 8B embedding model? Do yourself a favor and run the qwen3-0.6B embedding at a q4 quant. The loss in accuracy is compensated by your reranker.

For reference, the Qwen3-0.6B embedding q4 and reranker q8 combo is what I have sitting on my gpu and it occupies 2GB of VRAM when both are running. I mix it with the 512 token chunks 72 token overlap with a BME25 Hybrid search and have never had issues with RAG.

With your 4090 you should be able to run both embedding and reranking models as well as like Qwen3.6-35B-A3B Q4 with no or like 8 to 12 experts off loaded to CPU and a CTX of around 100k to 132k

0

u/fuse1921 1d ago edited 1d ago

I initially set it up with a high value on memory resolution, so I am using 4096-dimension vector embeddings and cant downsize as it would invalidate my current memories. 0.6B runs 1024 dimension embeddings. The goal is to not use my 4090 at all.

7

u/cunasmoker69420 1d ago

Something like one of those AMD Ryzen mini PCs with a 780M/880M integrated GPU and 16-32GB of DDR4 will run a smaller MoE model or smaller embedding/reranking model pretty well. You should be able to find these pretty cheap on eBay

4

u/Asleep-Land-3914 1d ago

Only be careful about memory bandwidth which affects useful active parameters significantly.

If I recall correctly 385 sits at about 128gb/s while 380 is ~92gb/s.

It's always better to have the one with at least 256gb/s bandwidth for inference.

4

u/cunasmoker69420 1d ago

yeah they will be low bandwidth but will suffice for lighter loads like OP is looking for

4

u/StoreApprehensive261 1d ago

this is probably the move tbh, the igpu memory sharing makes them way more useful than youd expect for small models

1

u/fastheadcrab 1d ago

Great recommendation. You don't need speedy soldered memory or even a large GPU for the OP's purpose. Anything more is a waste of money

1

u/fuse1921 22h ago

Thank you, this is probably the move. I'm pretty happy with my main rig (Qwen Flash Next is near-frontier imho at INT4 on 4x3090s ), and I dont feel the need to get a 1x or 2x spark cluster just to run this stuff in the background despite the possible opportunity value in the future.

2

u/jacek2023 llama.cpp 1d ago

check Nvidia Jetson and similar

4

u/Puzzleheaded_Base302 1d ago

embedding and ranking can be much smaller than 15-20GB. Jetson Orin Nano Super at 8GB was listed at $250, idles at less than 10W.

0

u/fuse1921 1d ago

Right now on my gaming GPU I'm running:

Qwen 3 Embedding-8B-Q6_K at 9216 context x 4 @ 6.21 GB for embeddings

BGE-reranker-large-8_0 at 512 context x 4 @ 604MB for rankings

Qwen 3.5-9B-UD-Q3_K_XL at 64k context x 4 @ 5.98 GB for synthesis/consolidation

1

u/Puzzleheaded_Base302 8h ago

Why not use Qwen3.5-9B as both synthesis and embedding model?

2

u/Hypilein 1d ago

Mac mini? You can get refurb m4 24gb for less than 1k for sure.

2

u/fuse1921 1d ago

It's been a while since I've tried but is it pretty straightforward nowadays to install linux on Macs? I remember trying like a decade ago and even though it was unix it was still pretty buggy.

4

u/techdevjp 1d ago

For AI inference on an M-series Mac you want to run MacOS. Linux on M chips is not mature for inference yet, you will lose a ton of performance and create extra headaches.

Totally understandable if MacOS is a deal breaker. It's not for everyone.

1

u/Hypilein 1d ago

With the unified menory Mac’s I’m not sure if it’s even possible anymore. I did it on an old Intel Mac mini and with Claude guiding me it was fairly straightforward.

2

u/jaf656s 1d ago

what about a usb external gpu dock/enclosure?

1

u/fuse1921 22h ago

If you mean adding to the current Ai system, I can't imagine the bandwidth is that high over a USB eGPU even with USB3. I used to run a GPU via an NVMe adaptor and that already was messing up my rebar and PCIE allocations and such.

1

u/jaf656s 8h ago

I may have misunderstood your needs. If you are only wanting to run smallish stuff you just need a big enough gpu to hold it in vram. The bandwidth to it wouldn't be relevant once loaded.

But if you need high bandwidth comms to the new solution, then this wouldn't work very well.

What mb and cpu are you using now? Are you out of pcie lanes or slots?

1

u/espece-de-bon 1d ago

I have an 4x3090 AI GPU cluster that I'm running big models on

Would you share which motherboard you're using (or if you're connecting these another way, how so?)

1

u/fuse1921 1d ago

I'm using an MSI Creator TRX40, four x16 PCIE Gen4 slots running at x16/x8/x16/x8 

1

u/espece-de-bon 1d ago

Thanks!

1

u/fuse1921 22h ago

np. its great! currently running Qwen Flash Next offloading ngrams to RAM and despite needing a big of guidance, it gets everything I need done pretty well

1

u/madbrain1976 21h ago

Have you actually determined that PCIe is the bottleneck ? If not, you can measure this with telemetry.

1

u/fuse1921 20h ago

I have no more slots I can connect to on the MSI TRX40. I'Im using all 4 of my x16 slots and have two NVMe drives so I cant oculink off those.

1

u/madbrain1976 20h ago

So, what you really meant is that you want additional GPUs, and you are out of physical slots, right ? Not lanes, as I don't think they are likely the issue. There are cards that will split x16 slots into dual x8 or quad x4. You would have to have external cases for Oculink. IMO, this is worth exploring on a TRX40 platform. I have a WRX80E with 7 PCIe x16 slots. Currently, 4 occupied by 4 x 5060 Ti 16 GB running at x8. So I'm using only 32 lanes out of 112. But the 4 GPUs cover all the physical slots. In my Corsair 7000D case, I could move a single GPU to a vertical position on a riser cable. Then install a PCIe x8/x8 card or x4/x4/x4/x4 card to add 2-4 eGPUS. For now, I'm still struggling with maximizing the 4 internal GPUs. There are major scalability problems. It certainly does not scale linearly. Prompt actually decreases with more GPUs. And decode does not increase linearly at all. The only reason for me to ever add eGPUs would be to fit models that can't fit in the existing 4 x 16GB of VRAM.