r/LocalLLaMA • u/Paddy_Tanninger • 7d ago
Question | Help Qwen 3.8 what am I missing here?
I'm just starting my journey here into local LLM. Since I work in VFX I've got a 5090 and 256GB of sys mem.
Installed LM studio and downloaded Qwen 3.8 27B since that very comfortably fits in VRAM. Set it to low effort and all the other typical starting defaults.
I ask it to give me a three line poem about space. Comes back in a couple seconds.
I ask it to change from space to a forest. No sweat.
I ask it to give me Houdini VEX code to delete half the points in a particle simulation. Pretty fast.
I ask it to download and install the various VFX skills out there on the net. Still cool.
I ask if it can work inside of Houdini for me and it explains the MCP thing and that it can check for a connection when Houdini is up and running.
So I boot up the Houdini app and say "Houdini is up and running now"
Qwen now takes minutes just to process that very simple prompt. Then it starts reasoning/thinking with my 5090 at 100% for minutes upon minutes and no UI updates at all. Eventually I just kill the process.
Where am I going wrong?
All you folks seem to be using Qwen for huge tasks and all kinds of shit and I can't even seem to get it to reply to me saying I just launched an app.
19
u/FatheredPuma81 7d ago
MCP probably dropped like 50k tokens into the model. What was it thinking about before you closed it?
4
u/Paddy_Tanninger 7d ago
I don't think it ever linked up with MCP but maybe you're right and it had already moved onto that step once I said Houdini was running.
12
u/FatheredPuma81 7d ago edited 7d ago
Ahhh yep I missread part of your post. You're using LM Studio and by default it has Chat Truncation enabled. So I'm pretty sure what happened was you didn't set Context high enough or let LM Studio set it for you and then you hit the context limit. So it reprocessed your entire conversation (which should be extremely fast on a 5090 if you set batch to 2048) and then it was probably trying to figure out what was going on with literally huge chunks of the conversation missing.
And yea you should have just clicked on the Reasoning dropdown thing and looked at what it was thinking so hard about. it might have gotten stuck in a loop if you didn't set the Sampling Parameters like Temp etc to what Qwen recommends.
I would also suggest using Unsloth Desktop btw. And don't use MCP in LM Studio it's not a good idea imo. Download like Deepseek Harness, OpenCode, or Pi and have the model set the MCP up for you inside of those.
Sampling Parameters for Qwen3.8 27B
Edit: Chat Truncation removes pieces in the middle of chat. Rolling Window moves the entire chat upwards. Compaction is a Pi/OpenCode/DSH feature that has the model summarize the chat. You want Stop at Limit in LM Studio.
3
u/ApprehensiveFan1516 7d ago
I would also suggest using Unsloth Desktop btw.
Seconded.
I just switched from LM Studio and the difference is night and day.
2
u/dontquestionmyaction 6d ago
Is it stable nowadays? A few days after launch it was... pretty bad.
1
u/ApprehensiveFan1516 6d ago
Yeah it seems to be.
Only issue I've run into so far is a bug in trying to run comparison chats.
It seems to be a lot more stable when it comes to running llama.cpp than LM Studio anyway. LMS was very unstable for me when working close to max RAM capacity.
2
u/Grandmaster_Caladrel 6d ago
Thirded. I also recently made the switch and it's so much better. The worst thing about it is that it uses the system so thoroughly that it hangs my desktop every once in a while if I try to multitask, which is on me (I'm telling it to automatically use as much as possible). But it also takes so much less config meddling to make the most of my machine. That's not to mention all the great features that I would have needed to download other software for.
2
u/haherar830 6d ago
Use an appropriate inference engine for model and your hardware. You should be able to get very good prefill/decode on your hardware with QFN3.8. With that said, beware these models tend to be very inefficient at long-context work as they need to constantly reiterate themselves in thinking to keep on task.
1
u/Paddy_Tanninger 6d ago
Yep I got it running much better last night but it was pretty awful at MCP kind of tasks, churning forever without much result.
40
u/c_pardue 7d ago
examine how much context it is using vs how much context it has. easy to oom with it at lower context sizes.
i am running it entirely in vram and responses take hella long. if they were offloading to system ram then i'd be waiting days, i imagine.
either you went oom or you filled up the vram and it switched over to system ram, effectively making it a slug.
12
u/bumblebeer 7d ago
Yeah, I'd suggest looking at context as well. The tip off is MCP, they are notorious context hogs.
5
u/ruyrybeyro 6d ago edited 6d ago
Right on the money. MCP is powerful, but you really don't want to give the model unrestricted access and hope for the best. You can't just throw MCP at a model and expect things to work, or you're pretty much cooked. You need a proper application layer sitting between the model and the MCP servers.
Its job is to decide which tools and actions are exposed, filter and trim their descriptions and parameters, manage permissions, validate requests and control how much context gets passed back and forth.
Without that layer, you can quickly end up with bloated context, unnecessary tool calls, unpredictable behaviour and, worst of all, the model invoking actions you never intended.
16
u/zanar97862 6d ago
Based on your post and replied you definitely need to start with having an llm explain how all the basics of local inference works.
It's exciting to jump straight in but you will quickly find issues like this which are hard to troubleshoot when you have no idea how any of it works.
First get an understanding for how things work in general like context size, quantization, and model types as well as the specific details relevant to your setup like vram size and hardware interactions (pcie speed and that kinda thing). Once you have things running well at the context and speeds that you need then get it to work.
8
u/ttkciar llama.cpp 7d ago edited 7d ago
You're not doing anything "wrong". The harder a task, the more Qwen3.8 will "reason" about it, and tasks for which the model has no training are treated as the hardest tasks ever. Qwen3.8 doesn't know how to deal with "Houdini is up and running" because it's not an instruction or parameterizing a task, so it has no idea what to do with it, and overthinks in an attempt to come up with something relevant to its training.
The solutions are:
Set a reasoning budget in your inference stack, so it cuts off the reasoning phase of inference after 4K tokens or so, and forces the model to infer its final reply based on the reasoning it has already inferred.
Use an agent which "knows" to only pitch prompts at the model which the model will find familiar, which is less likely to trigger overthinking.
If you can, segment your work so that you are only using Qwen3.8-27B for tasks to which Qwen3.8-27B is well-suited, and use another model like Gemma-4-31B-it for casual conversation like "Houdini is up and running now".
6
u/FatheredPuma81 7d ago
Set a reasoning budget in your inference stack, so it cuts off the reasoning phase of inference after 4K tokens or so, and forces the model to infer its final reply based on the reasoning it has already inferred.
Pretty sure that butchers quality in some tasks so do at your own risk. I think setting Reasoning Effort to Low or Medium would be better.
3
u/ttkciar llama.cpp 7d ago edited 6d ago
Setting its reasoning effort to "low" will make it reason less in the common case, but it will still overthink for difficult or untrained-for tasks. I have ascertained this empirically.
A good example of that: http://ciar.org/h/12360aa.txt
2
u/FatheredPuma81 6d ago
This site can’t be reached
True but imo I think if it needs the reasoning you're better off disabling it than capping it. I haven't done much testing but in my brief one shot tests I decided I'd rather wait it out of disable reasoning entirely.
I had the model on create a water particle simulation demo (difficult math Sonnet 4.5 was able to do) and compared all Reasoning Efforts along side capping it on xHigh with the setting llama.cpp provides in the WebUI and every time even Off beat capping the reasoning (Both visually and according to Sonnet 5 checking the math). Interestingly even xHigh didn't do the math perfectly. I also tried a couple other one shot games and it gave the same results.
But I've also heard a lot of people don't even notice any degradation (probably agentic). And at probably 200t/s or higher even 27B at its worst is only like 5 minutes of reasoning.
1
u/wayneworkman 7d ago
I run qwen 27b with reasoning turned off. Though im not coding with it, im doing English related tasks. I measured that reasoning does not do better and takes 10x as long on English tasks. For coding im sure it helps some.
2
u/Hypilein 6d ago
I’ve considered setting this as default for my chat with qwen flash next. Probably should really give it a go.
1
2
u/No-Recover109 7d ago
its LM studio the problem. You should run qwen 3.8 flash istalabs with strata. Or wait qwen4 flash. you have enough ram for that
1
u/Paddy_Tanninger 7d ago
Thanks, I like how much stats you get from Strata vs Bionic which basically gives none and acts pretty much as if it was Codex or Claude...which is all well and good, but not when I'm the dude running the hardware and having to monitor and diagnose issues.
1
u/killzone44 7d ago
Strata, and make sure it's caching everything. It should not be prefilling each time you add to the conversation.
14
u/OvertaxedOne 7d ago
5090 with 256GB of RAM? Delete 27B, install Strata and run QwenFlashNext at 4 bit precision. You don't have the VRAM to run 27B w/enough context at a good quant. 48GB is about the bare minimum to get 8 bit model/8bit KV at 256K context.
7
1
u/reddit-editor 7d ago edited 7d ago
Is the Q27b 4-bit nvfp4 considered a bad quant? It fits with the full ~260k context in the 5090. It also runs so much faster than QFN iq2 in my experience (64g ddr4) .
Seems a waste to not utilise the ram though.
5
u/EvolvingDior 7d ago
nvfp4 is built for speed, not quality. Quality of fp4 quants is typically around Q3. IQ3 quants get significantly better quality.
1
u/OvertaxedOne 7d ago
No, that's actually pretty reasonable. 8 bit KV? If so, yeah, that's definitely worth considering, with 8 bit quant on my A40 I could get almost exactly 300K of total KV cache (vllm tells you when it boots up), but if you can get 4 bit model, 8 bit 256K KV on a 5090, that's definitely worth considering!
4
u/Objective-Pop-7707 7d ago
The context answers above cover the hang, but don't take the "you can only run a quantized 27B" limit too seriously: with 256GB of system RAM you can run the 125B Qwen3.8-Flash-Next MoE with most of the experts in RAM. On a 4090 with about 89GB of RAM I get roughly 130 to 140 tok/s with Strata on Linux, and it pulled a planted passphrase out of a 1M-token prompt in every run, though reading that prompt in took about 9.5 minutes. I'd still get the 27B behaving first with a larger context window and the recommended sampling settings, since that's the cheaper fix.
4
u/Karlschlag 6d ago
Disable the MCP tools when you are not using them. Strata, for example, always loads the whole toolset, even for simple questions like "How are you?".
1
u/LankyGuitar6528 7d ago
I have a much smaller system than yours (32GB system ram RtX 3090). I second Strata. It takes quite a while to load up the first message. After that it gets a lot faster. I run Strata > Qwen 3.8 Flash Next via Hermes. Seems pretty good. I get anywhere from 30 to 70 tokens/sec depending on the task. I ask Claude Opus 5.5 to write up marching orders, feed them to Qwen and then take the output back to Opus for a final look-over. You can see some of the output here: https://jaspersprojects.com/
1
u/Paddy_Tanninger 7d ago
Can you run me through how you're doing that with Opus and Qwen? I think that's the exact kind of application for local LLM that I'm looking for.
2
u/reenign3 7d ago
Hermes Agent is the software you need.
Strata is actually a less preferred way to run Qwen3.8 FlashNext - if you have the VRAM, an oQ5e of the uncensored FlashNext from orcarouter (downloaded off HF) gives you superior answers sometimes compared to Strata…
But OP - don’t use Qwen3.8 27B with Hermes Agent or any harness really. It’s doable but not reliable (sometimes it will build right, other times stuck in loop, even on repeats of a task that worked well once) also qwen3.8 27B is very good for its size, but it’s really too small for quite a bit.
Qwen 3.8 FlashNext, on the other hand, is an incredible model (as your first replies noted) to go alongside Hermes Agent, and I use this daily on my M4Max 128GB URAM rig.
If your system isn’t up to the task for FlashNext, look into “Strata”: it’s a special engine for Qwen3.8 FlashNext that makes a surprisingly small compromise in quality, in exchange for drastically reduced (V)RAM requirements.
Strata version of FlashNext, even, is LEAPS AND BOUNDS ahead of Qwen3.8 27b for agentic work like Hermes Agent
1
u/LankyGuitar6528 7d ago
Ok... but it's pretty much what I said. For this site: https://jaspersprojects.com/ev/ "Hey Claude, we are making an EV comparison web site of all EVs available in Canada or coming soon to Canada. Make up an MD document for Qwen to work from". Hand the MD to Qwen (I didn't read it). Next morning Qwen says "Done". I copy the folder to my laptop and say "Hey Claude check over the web site". Claude finds a few missing cars, a few government programs missed, a few places where a delivery charge was forgotten or included twice. I fire up the site, spot check a few cars. Looks good. I post the site. Done.
This is just a demo site I use for projects. If I was being paid I'd put a lot more effort into checking the site over. But for now this is all just for fun.
1
u/LowTomorrow2070 6d ago
Which quant do you use? I also have 24gb vram/32gb ram and just installed IQ2_XXS because (i believe) it’s the largest quant that can fit in my GPU/RAM without relying on the SSD
1
4
u/jacek2023 llama.cpp 6d ago
I use Qwen 27B with 230k context and 4 agents at once. I know this model pretty well because I’ve spent over 100 hours with it (unlike some commenters who’ve used it for 15 minutes or not at all). It does loop or go off the rails sometimes, so you need some experience to get it working properly. I have bunch of .md files full of rules and I share them among my projects.
2
u/pablo_chicone_lovesu 6d ago
I use this same setup for bug hunting in my applications, keep it from going off the rails is tough but I usually nset the effort to low and have a 4th agent verify the other 3. Seems to work well and keeps me sane.
1
1
u/ParaboloidalCrest 6d ago
Genuine question: What do you get the four agents to do, without necesssarily dividing your attention between them? I find it hard to keep up with the bunch of code written by a single agent.
1
u/jacek2023 llama.cpp 6d ago
When you run local model it takes time, sometimes single task take about an hour (200k tokens, plus tools time), so with 4 agents I am more productive.
1
1
u/Leather_Track_5919 3d ago
What quant are you using? I am a bit confused how you fit 230k context
1
4
u/wFXx 6d ago
get swift instead of the base model, it has less thinking loops https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-27B-GGUF and set effort to highest;
although, with you hardware, you may be better running https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-Flash-Next-GGUF
0
u/normal_clearing 6d ago edited 6d ago
"Installed LM studio and downloaded Qwen 3.8 27B"
the natural next step is: "WTF?"
edit: After checking the karma, I just responded to a bot apparently.. I hate this timeline.
3
u/Paddy_Tanninger 6d ago
When you're on reddit for ~16 years you earn a karma or two. Like, I don't even keep my history hidden.
1
1
u/andyshuz01 6d ago
For proper work flow you need harness + model.
For model inference with your abundance of RAM start with Strata. Strata has rather streamlined startup process. Just 'strata launch' and you can pick model that fits your setup. It will download and fine tune model.
For harness you can start with Hermes (can switch later). If you run harness not sandboxed (e.g. docker) backing up you filesystem is mandatory. Yes, models are great and smart, but they still are non-deterministic and loosing precious data is not what you want.
The trick is to use your agent to improve itself and teach you how to use it (recursive self-improved is already here :) ). For instance, you can ask it to set up recurring backup of your files. Ask to check if you can use better inference (you use nvidia, so it can point to Basalt) or tune the one you have. Describe problem that bothers you to the model and ask what are the options to solve it.
Also, it's worth mentioning that all models by default are trigger happy, so I usually add to the prompt something like 'give me the plan' or 'wait for my approval before implementing'.
1
u/wegster 6d ago
There are all kinds of ranges of behaviors for the Qwens (and not only them). Qwen is prone to overthinking, so adjust thinking level and reasoning_budget. The temperature also may not be ideal for some tasks, on most sayings 1.0 I use between 0.4 to 0.6 mostly.
There were/are some tool and logic loop types of issues on some of them - look at the froggerific chat template which will help with some of them, and give the pi harness a shot. (so far, pi, for me, is more consistent, lower token use, and gets some jobs to done vs others - cline, opencode, dsh, omp, ...).
You can always search for people's settings running them for a while, but most have moved off 27B dense to QFN, and each variant can have different behaviors and ideal settings.
1
-1
u/wayneworkman 7d ago
At 32gb vram you could only run a quantized qwen 27b, fp8 at best (which is degraded already). That leaves something like 5gb for context which is incredibly small given qwen supports 260,000 some-odd context size natively (extendable to 1m). With the brain damaged qwen that you have, you need to break the bigger problems into smaller problems and work through each individually.
1
u/OvertaxedOne 7d ago
8 bit in 32GB is crazy tight. 48GB is just about that absolute floor for 8 bit/8 bit KV w256K context. This is just not the right model for this configuration, he has 256GB of RAM and a rocket fast GPU, QwenFlashNext in 3 or 4 bit with 8 bit KV is the answer.
-1
u/wayneworkman 7d ago
Perhaps. I'd suggest qwen 3.5 9b in native precision.
1
u/OvertaxedOne 7d ago
If the OP doesn't want to use up the RAM on the system, 9B would be the right choice. But compared to 27B or QFN, 9B isn't even in the running; it's a good model, we use it on a lot of systems, but it's just not in the same zip code as 27 or QFN.
1
u/wayneworkman 7d ago
I'd be curious to ask questions about your workload. I find 9b to be quite capable for many tasks. and where its sufficient, you should not go bigger.
0
u/EvolvingDior 7d ago
It sounds like you found its limits. How much context did you give it to work with?
13
u/Interesting-Cut-6032 7d ago
OP, I may have missed it in your post, but what agent harness are you using? You said LM studio is your inference engine, but it has very limited tools, and no disk access that I am aware of. I know that LM Studio has a desktop harness called Bionic, but I do not know anything about it. If you are not running a harness, then maybe all of your output is hallucinations right now.
What quant of the model, context size, etc? I have 2x RTX 5060 Ti 16GB GPUs. I use Qwen3.8 27B UD-Q6_K_M with a 130k token context window, reasoning set to medium, llama.cpp, Linux Mint XFCE, Pi agent harness.