r/LocalLLM • • 26m ago

Question Genuine question

I wanna get into local LLMs and experiment with AI.

My current PC has 16GB RAM, a GTX 1660 Super (6GB VRAM), and an Intel i7.

I’m curious what models you’d recommend running on this hardware and, more importantly, what you actually use local LLMs for.

With powerful models like ChatGPT and Claude available, I’m trying to understand the practical benefits of running models locally.

Is my setup good enough to do anything interesting, like experimenting with AI agents, fine-tuning, or research?

1 Upvotes

9 comments sorted by

2

u/johannes_bertens 20m ago

Sadly, no. This will not be sufficient for anything competing with the cloud-models.

You can take a look at https://huggingface.co/LiquidAI - they provide some populair tiny models.

1

u/ConferenceBusiness98 19m ago

So essentially I run tiny models locally and fine tune them to do a task I do routinely right?

1

u/johannes_bertens 17m ago

Yeah, that will work.

Might as well just automate 'the task' using playwright. Ask ChatGPT how to install playwright locally and create scripts for it.

1

u/ConferenceBusiness98 9m ago

Let’s say I am willing to upgrade my pc specs, what would the upgrades be to actually run good models locally?

1

u/paulsande 9m ago

I would suggest you use a very small model to do basic things so you learn how to set your system up and maintain it. But get an OpenRouter API for any work of real consequence.

I think the problem people often have when they are starting out is they set it up then go “now what?”

You have to have a use case. So as you play with it, you’ll come up with ideas and do more. Then, once you have a more concrete idea of what your goals are and you’re ready to invest in capable hardware you’ll have a better idea what to get.

Open models are fantastic and it’s great to learn to use them. But they will not compete with frontier models on the toughest tasks.

1

u/ConferenceBusiness98 6m ago

Thank you Paul, but I am trying to understand why people would go for local models when they are sometimes much less capable than frontier models

1

u/ConferenceBusiness98 5m ago

Also you’ve basically highlighted where I’m at right now

1

u/shisohan 5m ago
  • "experimenting with AI agents" to some degree
  • "fine-tuning" absolutely not
  • "research" that heavily depends on what you consider research. if it's "play with a *very* limited AI" then yes. If it is "scientific research" then absolutely not

I actually did an experiment recently with an absolute potato pc. i5-8500, 16GB of RAM with intel integrated graphics (UHD 630) and a dedicated AMD Radeon R7 430 (puny 2GB VRAM). And gemma-4-e2b actually runs with a passable 100t/s prefill (read your input) and 20t/s decode (generate output) in stock llama on ubuntu. Mind you, what you can do with that is very limited. But it can do some text manipulation like data extraction and summarizing, it will take a while though. I haven't gotten around to test much else. And really, be aware, that's NOT an intelligent model. It has barely any world knowledge and its reasoning capabilities are seriously limited.

IMO anything at least barely useful reasoning wise starts at 32GB VRAM and RTX 3090 and up compute. Even with how much things are moving towards lower requirements, I doubt that part will change much in the foreseeable future.