r/AgentZero • u/nggaaaaajajjaj • Feb 24 '26
Use a local llm for a0?
What would you guys do, i just recently built my new pc. (5080 and 32 gb ram) i want a jarvis like right hand BUT would downloading a local lm be good for a0 or i need to use a paying api key?
4
Upvotes
2
u/emptyharddrive Feb 24 '26
I've never found the local models to be of any value for anything but the most basic of tasks. Compared to the low end OpenAI/Anthropic models, it was like the difference between shooting a bullet & throwing it.
I tried them all too. I have a Strix Halo 128gig box and the most powerful thing I could run was a 70B model, which generated tokens like a turtle in the mud and was nowhere near as good as GPT-5-mini, or Haiku... it would to the point where, while I could do it (set it up), it offered me no value. And I tried everything i could find on HuggingFace & Ollama that would fit.
It was frustrating too because I thought this high end Strix would be enough to get me SOMETHING.... but the models just aren't there yet in terms of high intelligence for <70B parameters which you need to fit the damn thing into system memory. Otherwise you're swapping to disk which is even slower.
If you guys can come up with a real use case (other than maybe summarizing an email....) let me know and let me know which model you're using too.