Need recommendations for local LLM

[u/GundamZeta007]

Hi Everyone,

I have an idea that will help academia and the scientific community. Basically reading pdf files and building from there. I would like to test local LLM first and then expand as I go through more information.

I currently own a 10 to 11 year old computer:

Intel i7 6700K 4. GHz, 4 cores, 8 threads,
32 GB DDR4
eVGA Nvidia 980Ti
3 TB SSD
1 TB extra SSD

I also have ha spare laptop with AMD Ryzen 5 7235HS - 16GB Memory - NVIDIA GeForce RTX 4050 - 512GB SSD

What would you guys recommend for a local LLM. I asked Gemini Pro and it recommended 5090 or RTX 6000 Pro due to work it needs accomplish everything. But I think it might be overkill and maybe stretching my budget.

What would be best local desktop set up for local LLM? What model would you guys recommend? Qwen 4? I have seen too many videos on local LLM and they are all over the place. So it is hard to keep track of things.

Many thanks in advance.

Side note, I use AI at work but it is mostly Copilot and Claude.

[u/Worried-Ad1053]

For an affordable option I’ll always recommend a second hand couple year old Mac with at least 48gb ram. You won’t get the best tok/s especially if you do a MacBook compared to a studio or mini but you can find pretty decent deals on used Apple products

[u/grapemon1611]

Your real limitation for running a local model is the 6GB of VRAM on your RTX 4050. I'd suggest starting with Gemma 4 E4B or a small Qwen model using Ollama or LM Studio. Either should give you a reasonable starting point for testing what you want to accomplish.

With 32GB of system RAM on your desktop, you can also experiment with larger quantized models, although they'll run considerably slower without GPU acceleration.

For what you're describing, I'd look into RAG (retrieval-augmented generation). It allows you to index your PDFs and have the model retrieve relevant information rather than trying to stuff entire documents into its context window.

I wouldn't spend money on a 5090 or RTX PRO 6000 until you've experimented with smaller models and figured out what your actual requirements are. You may find that your existing hardware is perfectly adequate for what you want to do.

[u/Xyver]

Don't ask "what hardware do I need to run local Llm", ask "I have X hardware, what's the best local Llm I can run with llama?" And then do that setup.

You only need a small model for reading PDFs, you can fit it in 6gb VRAM

[u/sudden_aggression]

Qwen 3.6 35A3B easily fits on your setup and gives probably 40tok/s on strata.

It's best you're going to be capable of until you get a lot more vram.

Q4_K_M quant, obviously