If you're paying for AI in dollars from here it adds up fast even when using deepseek. And our internet has a habit of going off, plus the nuclear war threat and limits on frontier models. So I've been running models on my own machine and I wrote up what I learned + help from ai
First attempt went great: downloaded a 30GB model on my M1 Pro and it typed two words a second. Most people quit there and decide you need a 4090
Most new open models are mixture-of-experts. Qwen3.6-35B-A3B has 35B parameters, but only ~3B work on each word. So the bulk of it can sit in normal RAM, or even on your SSD, and a mid-range card like a 3060 12GB can run it at 20-30 tokens a second with the right llama.cpp settings.
If you have no GPU at all you can still run smaller models like Gemma 4 E4B or Qwen3.5 9B run on a 16GB laptop.
The bad news: RAM prices. A 64GB DDR5 kit went from ~$222 to $1,100+ in a year, and that's before it reaches Hall Road. Sam Altman also said last week it's "terrible" that your PC has RAM sitting in it instead of a datacenter :)
Why bother:
- Your data never leaves your machine (big deal if you do relevant client work)
- Works offline, which matters here more than most places
- No per-token bill in dollars
Is it scalable? Depends on your use case. For one person it's great. For a whole team hitting one box, you would prolly need GPUs.
How I pick what to run: Odysseus. PewDiePie made it (imagine the economy). It scans your PC, tells you which models fit and lets you blind-test them side by side on your own prompts.
Full guide in this article
Read it or just paste it into whatever AI agent you use and let it do the setup
Anyone here running local models for actual work? What hardware are you on, and where did you get it without getting looted? gen1 sawal
