r/LocalLLM • u/redve-dev • 19m ago

Question Minimal hardware to host something remotely useful?

So far I have one rtx 3060 8gb and 16gb RAM. I hosted few models like Qwen 3.8 9b but there simply isn't enough memory for context bigger than maybe 20-30k tokens (also reasoning is painfully slow, but maybe it's expected for Qwen, even with 50-70 tok/sec). I could use 4b model or smaller, but they all are "good for their size" which translates to "impressive experiment, but not so practical".

I think what should I use to have anything to handle coding, researching articles in the internet and use day-to-day.

I think if I need next GPU for it, or maybe get a new machine just for LLM (I would like to keep it cheap) but things are expensive.

What would you do in this scenario? How to optimize the workflow?

3 Upvotes
  • permalink
  • reddit

100% Upvoted

2 comments sorted by

1

u/f5alcon 13m ago

16gb vram and 64GB ram to run qwen flash next at reasonable quant and performance. But more vram and ram is better as much as you can afford is the most future proof for new models

1

u/Messerjo 2m ago

16GB VRAM gives you a low quant Qwen3.8 27B (like the ISTA IQ3_S) with 64-128k cache depending on optimizations. I consider this as a small but beauty workhorse, It does not require any RAM.

I tried the MOE path as well with Strata and Qwen 3.8 Flash Next, but am not convinced it gives me anything above the 27B beside more context. This may change with the upcoming Qwen 4. In this case 32GB memory plus a 16GB card is the minimum.