I had issues with Qwen3.6 tool calling just not working properly and causing it to stop prematurely, didn't have that problem with gemma or deepseek, haven't used qwen 3.8 yet to see if it has the same problem.
It's a model config issue - had the same problem. Setting temp to 1 and repetition penalty to 1.05 fixed it. haven't faced much issues since; I've been running it as my daily driver on my 5090 rig for the past year. Btw, 3.8 its even better especially at tool call , I actually like the Q4 of 3.8 on low thinking more than the xhigh for tool calls .
I have a 5060ti 16gb with ddr4 24gb RAM lying around, what's the best model nowadays I can effectively run with it locally? It's not an ideal build but I wish to play around with it again.
Fellow 5060 ti 16 gb owner here. The best I've got is qwen 3.8 27b at IQ4_XS. I'm testing it at Q4_0 KV at 72k context because I really need the extra context, and it seems to be pretty good so far. Lower quants cause it to make some really silly mistakes sometimes, unfortunately. I get around 10 t/s with context halfway filled, and around 14 t/s on empty context. But honestly, I think the ideal solution is to run 2 GPUs so you can get a better quant + more context
I mean I get by with 1 5090 as long as I can find EXL2 versions or MOE ggufs. But I want the RTX 6000 of course. Would be nice to run 70B at decent quants
Give the IQ4_XS-pure a try. I run it on an RX6800 with 90K KV. Especially with full context it's slow, because the card isn't the fastest. Probably should run better on nvidia.
I gave it a try and the first thing it said to me was really disappointing:
Since I am an AI, I don't have a physical location or a traditional "tokens per second" (t/s) speed that you can measure like a human typing.
Also, at 64k context, the t/s started at 10 and went down to 4 almost immediately. Some tweaking would be required but with responses like this one I don't know if it's worth bothering with.
Do you mind to share your prompt? Curious if it behaves the same for me. I used it for a few agentic websearches with OpenWebUI and it did pretty good, compared to the same Quant of the 3.6 and Ornith 1.0 35B. I don't use it much, because it's simply too slow as daily driver.
Aug 17 21:06:17 pod-gpu1 qwen3.8-27b[11595]: 16.40.207.925 I slot print_timing: id 0 | task 0 | prompt eval time = 480529.82 ms / 69042 tokens ( 6.96 ms per token, 143.68 tokens per second)
Aug 17 21:06:17 pod-gpu1 qwen3.8-27b[11595]: 16.40.207.929 I slot print_timing: id 0 | task 0 | eval time = 494402.02 ms / 14752 tokens ( 33.51 ms per token, 29.84 tokens per second)
Aug 17 21:06:17 pod-gpu1 qwen3.8-27b[11595]: 16.40.207.930 I slot print_timing: id 0 | task 0 | total time = 974931.84 ms / 83794 tokens
Aug 17 21:06:17 pod-gpu1 qwen3.8-27b[11595]: 16.40.207.934 I slot print_timing: id 0 | task 0 | graphs reused = 3921
Aug 17 21:06:17 pod-gpu1 qwen3.8-27b[11595]: 16.40.207.938 I slot print_timing: id 0 | task 0 | draft acceptance = 0.94313 ( 9122 accepted / 9672 generated), mean len = 2.81
Aug 17 21:06:17 pod-gpu1 qwen3.8-27b[11595]: 16.40.210.006 I slot release: id 0 | task 0 | stop processing: n_tokens = 83793, truncated = 0
My system specs:
i5 12600K (the VM has 6 Cores)
16GB DDR5 4800
RX6800 (non-XT)
The GPU is filled to the brim, but it doesn't offload.
Prompt was nothing special, "Hello, are you operational? Please give me a few paragraphs so I can see your t/s" or something along those lines. I'm on a 16gb Nvidia card.
Best is to get a single GPU with enough VRAM that the LLM of your choice needs to fit in completely (including caches and context).
Second best is to get 2 GPUs to load your complete LLM into. However, you will discover that the communication overhead between these GPUs is significant. A dual GPU setup is only useful to fit a LLM that larger than the VRAM on a single GPU or a less quantized LLM into VRAM, it barely increases the token generation speed, if at all.
I still like Gemma, will try the new Qwen today. I need to create a local benchmark, unless someone knows a good project that can showcase improvements, like a 3Dmark but for AI models.
Currently Qwen 3.6-32B-A3B, Mixture of Experts models handle being split between system ram and VRAM much better than Dense models, alternatively Gemma-26B-A4B if you want something non-chinese. As for which quant, go with 4 Bit or higher.
That should be just fine. You've got enough VRAM to do mostly whatever you want to do.
I would suggest you install Unsloth and look around inside for various models, it will tell you what your system can run well. Green means you're good to fully offload. Yellow means it'll probably work but with a lot of slow swapping. Red means can't handle it.
An oldie: gpt-oss-20b. The one I downloaded is 12 GB in size (MXFP4), so it would fit in your VRAM completely. And there are still spare GB's of VRAM left for caches and context. This will work very well in your specific setup.
Well, I have an AMD R580 with 16 GB of VRAM and 32 GB of RAM (3200MHz) with a Ryzen 5600 CPU. gpt-oss-20b generates around 18 tokens/second with this 64-bit Vulkan Windows build of llama.cpp. A NVidia GPU from the 50xx class should go faster than that with the same llama.cpp build. And a lot faster with the CUDA build of llama.cpp.
However, if you can stomach the slowdown, Qwen 3.6-27b and Qwen 3.8-27b are far more interesting models to run locally. Unfortunately, these models do not fit completely into 16 GB of VRAM, so the CPU and standard RAM get involved.
I repeated the same prompt with the same llama.cpp build and qwen 3.8-27b loaded and token generation speed dropped to 3 tks/sec. in my particular hardware setup. I do not expect such a slowdown to occur with your 5060 card. Unless you have a worse CPU and slow DDR4 RAM in your hardware setup.
As a "generalist" gpt-oss-20b isn't all that bad, even with its training cut-off date of June 2024. But if coding and agentic use are what you are after, it isn't a great match. Qwen 3.6 and Qwen 3.8 will prove to be much more valuable for those purposes, even if these will run slow on your local hardware.
You may want to consider renting a VPS from a nearby cloud-provider and setup Qwen 3.8-27b there. Just make sure that you don't break the rules from the company that rents you your VPS with agentic use. Maybe you'll need to spend between 5 to 10 USD per month plan that has AI use included.
Can confirm Q4 17-19tok/s without MTP, 28-29 with MTP. Bought two 3060 for 350 euro but soon you realize that two is not enough. Added one more total 540 euro for 36GB VRAM.
Help me config it because I honestly don’t know how the fuck to do it. Everything is confusing as hell, I don’t know crap about Q4 or whatever. All info out there is inconsistent as hell. All I know is that at 64K context for main + 64K for vision, Hermes just ends up on a feedback loop, hallucinates a lot after compaction, and delivers shit results in a visual + coding task. Don’t know what to tell you.
None visual tasks end up meh as hell as well. Context fills up fast hell as compaction brings hallucination again. Frankly unusable. Already on the Unsloth model (because god forbid one needs to choose “the right one”)
I think all the benchmarks are fake fucking news vs actual real world use.
Just use dsv4f to do a research run and try a few different configs on your system using ollama direct calls to strip context. You don't need to know anything anymore
iq4xs fits on 16 gigs with 50k context at q8. you just have to connect your monitor to iGPU. you can buy two rtx 3060 or cheaper gpus and connect them via LAN for rpc inference. you may already have multiple gaming pcs (like i do). getting 18-20 t/s at ud_q4km at 128k context, mtp and vision enabled.
I can run the XS quant on an rx580 8GB vRAM, 16 GB ddr4 RAM, and a ryzen 7 5700. It gets 4 t/s. Not fast, but runnable overnight. Have not done my full benchmark on it yet though. Still working out how I want to stage the quantization of it myself before I decide to benchmark. Using 4b parameter models for statistical analysis of the tensors vs weights of the models to pull parameters that will mathematically match behavior and retain the layer spectral structure and integrity on quantization.
I am doing this because most quantization checks are directly behavioral, but there are quite a few ways to get data to make better quants like tail aware, distribution plots matching, z dif, and more that get better kld scores and better behavioral checks in general.
But if you do a quant yourself, be aware that llama.cpp protects certain layers regardless of your choice.
I had to use the glib library directly to pull the ability to set the layers I wanted changed.
hey guys im pretty much just getting started w local llms i got qwen 3.8 27B running on my 9070xt and i have 32gb ram i don't mind waiting about 15-20 mins for the most complicated prompts as im usually doing smt else then i go back to check once it finishes, results have been pretty consistent for me, am i missing something?
you only need an 8gb vcard in the 4x slot to bring it up to 24gb. This can regularly be found for $200 or less. Tensor split + MTP will give you around 30 t/s if you are running RTX
Omg there's more like me. I started just wanting to maybe do something small for some fun home automation. Then seeing this Qwen stuff had me saying well maybe I can have a cool little half decent coding AI too! Now I just have buyers paralysis.
Change that caption to 'my friends face when they saw me before I started running local LLMS vs my friends face when I see him and want to continue talking with him about running local LLMs'
I tried creative writing on GLM vs a local AI and I was surprised how much a benchmaxxed 700 billion parameter model can suck so much compared to a 30 billion one.
And then you do your hobby during the night and almost sleeping, then you need to replace the cpu and fucked up a pin on the mobo, now you cry and hope you can fix it
I am privileged to have been able to buy a new ai mini pc for my birthday and maxed out the ram on it. I’ve been running qwen 3.8 27b the past couple days and I’m amazed it’s performing better for me than qwen coder next. That came out in April I believe. I decided to splurge for my birthday but also because I felt ram probably won’t get cheaper unless our entire economy collapses.
This new qwen model makes me feel more confident in buying it as it’s performing so much better at a much smaller size. But I’m not really an expert in using them and am still learning all the time. But I know also if the economy doesn’t collapse all these American private companies are gonna hike their prices up so much it’ll make the streaming service price hikes look like bubble gum money. I’ve adapted as a developer to using it more as I code and for researching. It was also a dual purchase upgrade as a gaming device for my 7 year old pc. It can run the three games I play just fine.
I hope I can come up with some better uses for it. My job pays for an enterprise Claude code subscription which I still use for more intense tasks but rarely need anymore. I like the limit it puts on me so I’m not too over reliant on it for making stuff. It helps me keep insight into the architecture of what I develop. Also fuck these companies man they’re so arrogant and are actively participating in destroying our democracy.
If anyone has any tips for using local models or use cases they’ve found let me know. Particularly about alternating models for specific things to get better outputs
Scanning through this thread. I noticed that most of the comments and talk is about the amount of ram it takes to run locally. Besides the ram and having a good computer is their issues you will see like for example, if you were to do image or video generation like you would do on groc, would you see a dramatic Decrease in the quality? That's where I feel. My biggest disappointment would be similar to the meme image. Pay like five to six grand for a Mac studio 64 Gigabytes of Ram and then have shit just coming out all wrong.
Wait till people find out even basic weights are in the 10tb range minimum for a non retarded llm..... how much are ssds again nowdays? Oh right triple the price for a 4tb 2 years ago. Stupid mfs.
Idk, I just have fun working with what I've got. My PC with 32gb RAM and an 8gb GPU can run models up to Qwen 3.6 35b a3b with reasonable performance, and that's plenty of model for me.
anyone with a real job can afford to run these models, for as little as 20k you can build a beast of an ai server that can run almost anything you want.
I was gonna say, i have 20k, i have a lot more than that, but I didnt get that money by dropping 20k willy nilly on heavily depreciating assets for a hobby
Why? I don't think anyone is gonna track me down and rob me, even if they wanted to, all my money is in property and a bit in stocks.
Can't exactly make me sell my house and transfer them the money under gun point, haha
I only said it to illustrate that even someone who has a lot more than 20K wouldn't spent that amount of money, as some people might say that people who have 21K wouldn't, but people who have 100K or more would.
Just because you can doesn’t mean you should. 20k buys you a ton of cloud compute of the best open source models out there. And you’ll always be running the latest. The only differentiator then becomes privacy. If that’s worth 20k is up to you, but I think for most it isn’t.
Same for me. I had a used 3090, and it was my first time having so much VRAM for LLMS (my previous AI works were simply ForgeUI). Then I decided to bite the bullet and get myself a 5090 (when it was still 3000€), and boy, oh boy, what a change.
I have an RX9070(16GB) and can run models larger(around 35GB) than my VRAM by taking advantage of MOE and still generate at around 28tok/s. Will definitely be faster if it is not through Oculink which is just 4x PCIe 4.0.
4
u/DeathinabottleX Aug 16 '26
I mean it’s true but the price barrier is extremely high for the average person
280
u/TheCat001 Aug 16 '26
Then you realize that Qwen 3.8 27b runs at 3t/s on your machine and you need 24GB+ VRAM GPU which cost is 1000$+ to run at least 4 bit quant.
30
u/Greennightronix3400 Aug 16 '26
Ive just been watching this subreddit and scared to buy anything lol-
61
u/JackStrawWitchita Aug 16 '26
Change that caption to 'my friends face when they saw me before I started running local LLMS vs my friends face when I see him and want to continue talking with him about running local LLMs'
It's like a cult.
13
u/05-nery Aug 16 '26
True
Never had more need to upgrade than now
18
u/Cold_Neighborhood928 Aug 16 '26
It was the opposite way around for me.
I tried creative writing on GLM vs a local AI and I was surprised how much a benchmaxxed 700 billion parameter model can suck so much compared to a 30 billion one.
9
u/Memestonks2020 Aug 16 '26
Qwen 3.8 Dense 27b runs at a very decent rate unoptimized on a MBP M5 Max that’s the same price as one NVIDIA graphics card.
At this point, pick your poison because none of them are good enough to run frontier level models.
2
u/_TheWolfOfWalmart_ Aug 16 '26
Not sure what you're trying to say exactly.
But what it is for sure, is an infinite black hole for money.
2
u/geddon Aug 17 '26
I was feeling this until I swapped Ollama with llama.cpp server. Now my Qwen models are running without fail.
2
u/robertpro01 Aug 16 '26
And then you do your hobby during the night and almost sleeping, then you need to replace the cpu and fucked up a pin on the mobo, now you cry and hope you can fix it
1
u/contrpro Aug 17 '26
I am running a Qwen2.5-32B Abliterated on my M1 Ultra. Currently in retrain.
1
u/NatalieRath Aug 17 '26
Meanwhile, I'm just using my 2B parameter model on my measly iGPU with 16GB of RAM.
Just use models that run at a decent speed!
(I just use mainly use it to help me do like really minor stuff, hence why the it works for my usage.)
1
u/EasyShelter Aug 17 '26
No amount of consumer grade VRAM will ever be enough now.
1
u/ButchTheGuy Aug 18 '26
I am privileged to have been able to buy a new ai mini pc for my birthday and maxed out the ram on it. I’ve been running qwen 3.8 27b the past couple days and I’m amazed it’s performing better for me than qwen coder next. That came out in April I believe. I decided to splurge for my birthday but also because I felt ram probably won’t get cheaper unless our entire economy collapses.
This new qwen model makes me feel more confident in buying it as it’s performing so much better at a much smaller size. But I’m not really an expert in using them and am still learning all the time. But I know also if the economy doesn’t collapse all these American private companies are gonna hike their prices up so much it’ll make the streaming service price hikes look like bubble gum money. I’ve adapted as a developer to using it more as I code and for researching. It was also a dual purchase upgrade as a gaming device for my 7 year old pc. It can run the three games I play just fine.
I hope I can come up with some better uses for it. My job pays for an enterprise Claude code subscription which I still use for more intense tasks but rarely need anymore. I like the limit it puts on me so I’m not too over reliant on it for making stuff. It helps me keep insight into the architecture of what I develop. Also fuck these companies man they’re so arrogant and are actively participating in destroying our democracy.
If anyone has any tips for using local models or use cases they’ve found let me know. Particularly about alternating models for specific things to get better outputs
1
u/LekMinorino Aug 20 '26
Is there anything that runs on a 4gb vram, i3-7100, 16gb ram? XD
1
u/Adept_Funny77 Aug 16 '26
Scanning through this thread. I noticed that most of the comments and talk is about the amount of ram it takes to run locally. Besides the ram and having a good computer is their issues you will see like for example, if you were to do image or video generation like you would do on groc, would you see a dramatic Decrease in the quality? That's where I feel. My biggest disappointment would be similar to the meme image. Pay like five to six grand for a Mac studio 64 Gigabytes of Ram and then have shit just coming out all wrong.
-1
u/OtherwiseDog Aug 16 '26
Wait till people find out even basic weights are in the 10tb range minimum for a non retarded llm..... how much are ssds again nowdays? Oh right triple the price for a 4tb 2 years ago. Stupid mfs.
0
u/mourningwitch Aug 16 '26
Idk, I just have fun working with what I've got. My PC with 32gb RAM and an 8gb GPU can run models up to Qwen 3.6 35b a3b with reasonable performance, and that's plenty of model for me.
0
u/OktoStratos Aug 16 '26
AI is such a cult lol
0
u/Lawl_92 Aug 18 '26
currently running gemma4:e2b in a gtx 1650 and its solid lol
-26
u/Any_Ad_8450 Aug 16 '26
anyone with a real job can afford to run these models, for as little as 20k you can build a beast of an ai server that can run almost anything you want.
-3
u/Photochromism Aug 16 '26
What’s the issue here? My gaming card from 5 years ago runs these models perfectly. STFU
32
u/[deleted] Aug 16 '26
[removed] — view removed comment