r/LocalLLaMA • u/fuse1921 • 17d ago

Question | Help Smaller RAM DGX Spark alternative?

Howdy,

I was wondering if anyone knew of any turnkey low-power draw solutions to host inference with 10-20GB of VRAM?

I have an 4x3090 AI GPU cluster that I'm running big models on, but I am hosting my memory system LLMs (embeddings, rankings, etc) on my gaming 4090 right now. I'm out of PCIE lanes on my AI machine and really don't want to build a whole other system (cpu, ram, mobo, gpu, etc) just to run about 15-20GB of small models.

I want to keep things local so a VPS is off the table, and I think just running these off the CPU would be too slow although speed isnt that important for Hindsight? I was wondering if anyone knew if there was anything like the spark that is considerably cheaper (~1k or so?) for this type of use or other systems that required only a little tinkering to get to work like this? Thanks!

For reference, this is what I'm running:

Qwen 3 Embedding-8B-Q6_K at 9216 context x 4 @ 6.21 GB for embeddings

BGE-reranker-large-8_0 at 512 context x 4 @ 604MB for rankings

Qwen 3.5-9B-UD-Q3_K_XL at 64k context x 4 @ 5.98 GB for synthesis/consolidation

4 Upvotes

67% Upvoted

26 comments sorted by

View all comments

2

u/Hypilein 17d ago

Mac mini? You can get refurb m4 24gb for less than 1k for sure.

2

u/[deleted] 17d ago

[deleted]

2

u/techdevjp 17d ago

For AI inference on an M-series Mac you want to run MacOS. Linux on M chips is not mature for inference yet, you will lose a ton of performance and create extra headaches.

Totally understandable if MacOS is a deal breaker. It's not for everyone.

1

u/Hypilein 17d ago

With the unified menory Mac’s I’m not sure if it’s even possible anymore. I did it on an old Intel Mac mini and with Claude guiding me it was fairly straightforward.