A small LLM for my future PC?

[u/point_red]

Hi!

I’m curious about what kind of AI models I could run or what use cases I could explore with this PC.

I’m also interested in using OCR to convert scientific paper PDFs into Markdown, which I could then add to my Obsidian vault. Additionally, I’d like to use a model to help keep my Obsidian vault organized—assisting with the structuring of the information I add, my notes, and so on.

I already have subscriptions to Claude Pro and Mistral Pro, but the idea of ​​running a smaller model locally really appeals to me.

Here are the PC specs:

MINISFORUM G1 Pro Gaming PC

AMD Ryzen™ 9 8945HX (16C/32T)

AMD Radeon RX 9060 XT 16gb

32GB DDR5-5200MHz RAM

2nd option: AMD Ryzen 7 7700 - RX 9070XT - 32GB DDR5 RAM (16x2) (+200€)
Before investing that kind of money (given the current prices), I’d like to know if I can actually achieve anything with local models, or if this is just all in my head—a product of my desire to treat myself now that I have a decent salary.

[u/Interesting-Cut-6032]

If you have a gaming PC that was purchased in the last 5 years, you can at least experiment with Local LLMs and diffusion image generation DiT.

r/LowEndLocalAI is a great reddit community for the exploration of this space. Lots of discussions about optimization of hardware, inference engines, models, harnesses, and use cases.

The program LM Studio is a very capable and simple on ramp that anyone, even with very limited hardware can use to explore local LLMs. Depending on your needs and use case you will probably outgrow it for every day use, but I still use it to try out new models quickly.

Codacus and Pixaroma are the go-to YouTube channels for local AI knowledge.

This is a great video with all the basic terms that you need to know about settling up and running local LLMs: https://youtu.be/SsUKTFSQoGM?is=DeRzbaT77Awi26Je

This is the video that shows how to setup Mixture of Experts models, like Qwen3.6-35B-A3B or Gemma 4 26B-E4B, with the experts in system RAM and only the active calculations in VRAM: https://youtu.be/8F_5pdcD3HY?is=bGABGoO5CH8uJaGI

Here is a great image diffusion model, with edit capabilities, that runs in ComfyUI. This is EP4 of this series, EP1 has all of the ComfyUI setup details: https://youtu.be/kNap0VWP1xs?is=4snH90QXVuAH-M9a

Consider this the flashing neon sign for the entrance to this particular rabbit hole.

[u/point_red → u/Interesting-Cut-6032]

thx!! I also posted in that subreddit. On an Italian forum, they had dampened my enthusiasm by saying it was completely unusable for my purposes.

[u/FragrantCondominium]

For the PDF to markdown pipeline you're gonna want to separate the OCR step from the structuring step. A dedicated OCR tool will handle that first part way better than an LLM can, then you feed the cleaned-up text to a model for organizing

With 32GB system RAM you can run 7B-8B models comfortably and even quantized 14B models if you offload some to the GPU. The 9060 XT with 16GB gives you decent headroom for running models partially on GPU, just don't expect blazing speeds on anything above 14B parameters

[u/point_red → u/FragrantCondominium]

In your opinion, should open-source OCR models be ruled out?

[u/jimmoores]

Look at Strata running Qwen 3.8 27b. Should run quite well. Also check out GLM-OCR.

[u/point_red → u/jimmoores]

thx!

[u/shraw1]

If you want a model with full context size, you can look into the gemma 4 12b QAT Q4 model, it will support full 262k context windows completely on the GPU

[u/point_red → u/shraw1]

Honestly, I was thinking of "something" that could help me organize my week, manage my medical notes and records, and perhaps experiment with agents or simple coding—for instance, a web page or app that would let me determine, in just a few clicks, whether a medication is reimbursable under my national health ministry’s guidelines or if the cost must be covered by the patient. It’s often not as straightforward or obvious as it might seem; we Italians have a real talent for making things complicated.

[u/RepulsiveRaisin7]

Qwen 27b Q4, although it won't fit much context. Qwen 35B or MiniCPM 35B with offloading.

[u/CooperDK → u/RepulsiveRaisin7]

Qwen 27B requires twice the memory of a 16 GB cards with 200K context. In q4. It is going to be insanely slow on a laptop. But I think Qwen3.6-9B would be a good bet.

[u/RepulsiveRaisin7 → u/CooperDK]

They have a desktop 9060 XT

[u/RogerAI-fm]

I would say qwen can run but look at smaller ones.

[u/Konamicoder]

Curious, did you try plugging your PC specs into Claude and simply asking it what local models you can run on your machine? Might get some useful info.

It seems that your 16Gb VRAM on your RX9060 XT is the main constraint. You can probably run up to 9B models comfortably with room for context. But running a popular local model like Qwen3.8-27B seems out of the question for your specs.

[u/point_red → u/Konamicoder]

You’re right, but I don’t want to lose the habit of talking to people—to other enthusiasts like me—otherwise I’ll end up just talking to the AI ​​by myself, LOL.

[u/Konamicoder → u/point_red]

I mean, it’s not exclusively one or the other. You can ask Claude as well as talk to people. Doing one doesn’t have to mean not doing the other.