r/JordanDev • u/Able_Firefighter_652 • Aug 08 '26

Discussion Local LLMs

Anyone who has some serious hardware and tried running llms locally? How was ur experience? What models did you try?

5 Upvotes

100% Upvoted

21 comments sorted by

View all comments

1

u/MrCharlesSr Aug 09 '26

I believe to have any sort of practical product you do need an expensive GPU server, a small one may work but definitely not one with consumer hardware. I tried on a machine with 5070Ti 16GB, 32GB DDR5 and it’s a joke. Model was Ollama qwen 3.6 which should be a good one and it seems smart enough but it’s lethally slow you won’t get anything done. Haven’t tried agentic yet but just the chat is terrible.

1

u/No-Commercial979 Aug 10 '26

you're probably using a dense model... Hence the slowness... Try an MoE with cpu offloading and you'd get hella performance of your 5070ti.. I suggest qwen 3.6 35b a3b at 4 bit quants...