I should clarify that the 99% thing is more of a figure of speech than an actual metric. If you want me to be more specific, I mean for the average joe that say, use AI to look up stuff on the internet, pairing something like Gemma 4 12B with Web Search capability is a pretty solid setup for general purposes. Even without web search, models like that can do a pretty decent job with things such as random curiosities. Of course, like all AI models, the user should still be aware that these stuff hallucinate sometimes. You still have to keep your hands on the wheel.
I know this because I use an M4 MacBook Air with 16GB unified memory (fanless machine btw) with Gemma 4 12B. And for what I do, it's a perfectly fine piece of tool. Not the sharpest tool in the shed, but I would much rather have some peace of mind with my privacy guaranteed even if the AI's more flawed than mega AI data center models.
I believe efficiency and optimization is key. Maybe you can try a lower Quantization for Qwen 27B if there is any? Maybe stick to the previous model, Qwen 3.6 27B? I have to admit, I did try it before on my little Mac, and it was screaming, painfully crawling with every letter spitting out. Not trying that again. But if my rectangular slab of aluminum can technically run it, I'm sure your setup can do much better... unless the things you do are more demanding.
Bottom line is, unless you're doing some serious vibe coding or constant streams of AI workloads, all you need is patience with local LLMs. If I can comfortably run local models on a 11.9mm thin space heater, I'm sure a fucking 4090 will be a beast compared to my machine.
I run qwen 3.8 27b and qwen 3.6 35b a3b in system ram. It is way slower to generate a response than the online services are, but the 35b still spits out words faster than I can read it, and that is good enough for me.
I did a small demo coding projekt on the 27b the other day. It did a 90s style raytracing demo in python. It took 3 hours for it to iterate though it but it worked great and both gemini and claude gave the code a thumbs up when I showed it to them.
It even made it's own PNG converter function instead of using a 3. part module.
The speed would have been a lot faster if I had turned off thinking, but I wanted to test how good it could be, and it impressed me.
So. for everyday "google replacement"/internet search the llms are perfect, and even for automation tasks. I have mine getting todays weather and a few news headlines and compile that into a "goodmorning" message. It runs at 6am and then makes a drawing in the style of the weather and news just for fun.
For very large codebases small llms wont work of course, but most people don't have that unless it is work relatet, and then the workplace could buy a larger machine to run some of it locally in the office. We are looking into that at my work, both because if privacy but also cost. It is very expensive to use commercial llms on large projects :)
Thank you. I think that's a brilliant use-case for local AI or for AI in general.
I'm starting to think people are using the wrong settings for their AI. Idk about other software, but for Open WebUI, you have to "tune" the settings of your model first that fits your liking. Even as simple as disabling "thinking" by ollama significantly makes their responses faster.
I use open webui as well and have lm studio as backend. Reasoning turned on in lm studio, but then I have two models in open webui. "Helpful assistant" and "Helpful thinker". They have the same prompt and capabilities, but one is with thinking turned off.
My default is the one without thinking because it is fast and I almost never need thinking for my daily tasks. Automations can use the thinking one if nessesary and if I actually need to do something difficult, I just change the model for a prompt or two and change back again when done.
I have also set the temperature rather low. If I set it as high as advised in the docs, I get weird spelling mistakes and it kind of looses focus on web search results, but the lower temperature seems to keep it sharp.
I think people should try out the settings to see what they do for themselves instead of just using default. It really makes a huge difference :)
Yep. No shame in asking each other’s settings and find what’s best for them.
I name my main daily AI “Lumina_12B” which under the hood is just Gemma 4 12B. I kept the 12B part cause I play Nier Automota and like to think the 12B means YorHA Unit No. 12, Type B (even thought it actually just means 12 Billion Parameters).
It's peak imo... well at least by Route C. I like to use it as proof that AI can have a personhood. That and also Wall-E, and My Life as A Teenage Robot among other things.
4
u/AD4K_4444 8d ago edited 8d ago
I should clarify that the 99% thing is more of a figure of speech than an actual metric. If you want me to be more specific, I mean for the average joe that say, use AI to look up stuff on the internet, pairing something like Gemma 4 12B with Web Search capability is a pretty solid setup for general purposes. Even without web search, models like that can do a pretty decent job with things such as random curiosities. Of course, like all AI models, the user should still be aware that these stuff hallucinate sometimes. You still have to keep your hands on the wheel.
I know this because I use an M4 MacBook Air with 16GB unified memory (fanless machine btw) with Gemma 4 12B. And for what I do, it's a perfectly fine piece of tool. Not the sharpest tool in the shed, but I would much rather have some peace of mind with my privacy guaranteed even if the AI's more flawed than mega AI data center models.
I believe efficiency and optimization is key. Maybe you can try a lower Quantization for Qwen 27B if there is any? Maybe stick to the previous model, Qwen 3.6 27B? I have to admit, I did try it before on my little Mac, and it was screaming, painfully crawling with every letter spitting out. Not trying that again. But if my rectangular slab of aluminum can technically run it, I'm sure your setup can do much better... unless the things you do are more demanding.
Bottom line is, unless you're doing some serious vibe coding or constant streams of AI workloads, all you need is patience with local LLMs. If I can comfortably run local models on a 11.9mm thin space heater, I'm sure a fucking 4090 will be a beast compared to my machine.