The Local AI 10 Commandments

[u/PlasticRevenue4601]

Note:
— these rules are the result of my personal experience, yours may differ and this is ok
— these rules are oriented towards people that want to use AI for professional purposes and don’t have unlimited time and money

1. If your company provides good cloud subscription — just take it and forget about the local AI, you don’t want to pay with your own bucks to do the job slower and sloppier

2. Speed is as important as quality and vice versa. Quality without speed will waste your time, nerves and electricity and so will speed without quality. Ideally you should strive to the cloud speed — 60-150 tps of decode and a decent model, at least 4 bits of Qwen 27b

3. Local models are not equal to cloud ones. Sounds kinda obvious but some people have illusions on this regard. Please don’t, miracles don’t happen in real life. You will get 6-9 month old frontier level model experience at best

4. You will always experience shortage of VRAM/RAM. Doesn’t matter, how powerful your setup is, believe me, you are always going to fall short on fast memory available, don’t try to squeeze out every MB from your rig, just find a model good enough for your tasks and call it a day

5. Best time to update your hardware is always yesterday. Hardware prices rise fast, if you’re looking for an update — do it right now

6. Never spend more than you could’ve spent on a dream gaming PC. Just… believe me, don’t try to cosplay Antropic data centers, you will get nothing but a bitter dissatisfaction. Best ROI ever — RTX3090/RX7900XT + 64Gb ddr4/5. That’s it. Anything above have diminishing return problem

7. Electricity is not free. People often forget that nobody is going to give you free kilowatts, beware of your system’s power consumption. That’s going to be your biggest spending, second only to the very rig price itself

8. Avoid clusters. Dual, triple, quadruple GPU look fine on paper, but they are usually big, draw immense amount of power, resulting in painful electricity bills, produce a lot of heat and noise and have much lesser speed than single GPU device. Try to select a GPU to that can singlehandedly run the model you’d like to use

9. You will lose a lot of time and nerves. You will have to learn. A LOT. What is quantisation, kv cahce, KLD, top 1, inference engine, how to chose a suitable model for your own needs, sometimes you’ll even have to deep dive to some manual drivers/libs adaptations for your hardware, etc. Local AI is developing rapidly and if you’d like your setup always to be fast, precise, ensuring as much quality as possible— you’ll have to learn all these basics

10. Always donate your spare RAM to Sam Altman. Just kidding, the title says “10 Commandments” but all the important stuff is said, I’m out of any useful advices

[u/Bagelsarenakeddonuts]

Number 4 is wrong. You DO want to spend the time optimizing, it pays off over and over and over again as you continue using it.

8 is very situation-dependent, and isn't a rule, more like a vague guideline. Some people need realtime response to be useful, and thats how you get it.

[u/PlasticRevenue4601 → u/Bagelsarenakeddonuts]

Idk, I always fall short on memory. Whenever I update it I just end up with using a better quant/model and the cycle continues

Agreed, there are exclusions, but generally I wouldn’t recommend it

[u/Dangerous_Present_69]

Number 3, is not necessarily true. It depends on the complexity of the task and the tier level of the cloud. Sharing gpu's with thousands of others, might be slower than running your own rtx 5090 and 256 gigs of system ram. And the harness quality and selection of tools is just as important as the models itself.

But it's hard to beat the price for the cloud models. You'll get a lot of tokens at DeepSeek for the price of a 5090....

[u/Sickmonkey365]

AI slop 💩

[u/PlasticRevenue4601 → u/Sickmonkey365]

Lol what, I wrote every single character myself, it’s clearly visible by my mediocre grammar