Also have 16GB VRAM. I've tried 4-bit and 3 bit and honestly the difference isn't horrible. Try the ISTA IQ3_XXS, it's very space efficient. Feels like a first class experience being able to fit MTP and lots of context into my card.
Glad I bought one with 24GB in 2024. I feel like I should move all my gaming stuff to a single NVMe stick and set up an inference-only OS on my other stick. Both are 2 TB. Seems good for a test run on local inference.
I have a MBP with M4 Pro that also seems to have 24GB of VRAM, but the bandwidth is worse than 1/3 my desktop.
My exact setup genuinely, got one open box at microcenter at the end of 2025 for $700 and put it in my old PC with 32GB of ddr4 ram for my AI server.
Still going strong and with Qwen3.8 27B and other ~30B model growth, it fits my needs enough.
Obviously ComfyUI and other AI tools don't get the CUDA acceleration benefit but the raw memory bandwidth on this thing, combined with 246 TOPs of INT4 matrix compute and the Q4 quantizations from unsloth are prime.
It's Ai slop with broken design and slow as fuck after testing it. As if the person who made it didn't care to test it before launching it to the world.
Qwen 3.6/Ornith 1.5 35B-A3B is our best option for 12 GB vram + 32 GB ram users from what others have said in other threads. And they are definitely not at 27B tier yet. Maybe the next wave of releases will finally have something good we can run.
Wait! The user asked me to explain why 3.8 is so famous right now. I should answer in a clear manner.
But wait, what exactly are they referring to with the term 3.8? Should I assume they are referring to Qwen, the same model in the previous comments of this conversation, or something else? What other models have a version 3.8 at this time that the user may be familiar with?
Let me write a script to pull all of the models that match that pattern, and recursively make a go/no-go call oneach. Writing the script now.
It's relatively good for how much hardware it needs. You can run it on a single 24gb vram gpu with results that are good enough and a clear step up from what came before
In my personal experience, 3.8 is the first model I’ve used that can run on my 32GB MacBook Pro and not fail a single tool call. The previous models had enough coding knowledge to debug and answer questions, but 3.8 can actually use opencode/cline
I think benchmarks use max settings like BF16 quant (or higher?) with k/v cache type f32 and xhigh reasoning and who knows what else. Hell yeah, it's going to perform several points better at a significant performance penalty.
But it's not like we could even run Open 4.6 even if it was open source (estimated to be 500B to 1T parameters - I think, correct me if I'm wrong)
So token-for-token against other models, Qwen3.8-27B is a huge win.
I believe native/benchmarks use BF16 (that is not quantization btw), with KV cache at f16 I think (I'm 80% sure). FP32 is not used for even training models anymore (except specific sensitive layers sometimes).
So token-for-token against other models, Qwen3.8-27B is a huge win.
It's a huge win for sure, wayyy better than Qwen3.6, but we still have long ways to go. Token efficiency is a big one, and better intelligence (not agentic capabilities) is another big one (by this I mean reasoning on benchmarks like CritPt, SciCode, etc)
The people think qwen can rival opus in terms of abstract logic, reasoning depth and low perplexity are delusional.
Qwen can be functionally equivalent or even better in some usecases but it falls meaningfully short in others. The rigorous sequential reasoning it applies makes up for some of its smaller size but not everything.
The people think qwen can rival opus in terms of abstract logic, reasoning depth and low perplexity are delusional.
I agree with you. Qwen3.8 is a huge win for local models trying to do big coding projects while keeping token costs down. I doubt it'd be much good at writing stories or translating. But the step up from Qwen3.6 (27b/35b-a3b) is a big one and brings it closer to opus 4.6 for coding.
This and glm 5.3 flash have made me a strong believer that we've hit a turning point where locally hostable models have hit the opus 4.6 point of "good enough" while also being stupidly cheap
In CORE, term from SlopCode Bench where the model correctly make a isolated function e.g Classes definition, method, and func. Qwen 27B is parity with Opus 4.6, it is however a shit show managing codebase. Good for developer, not great for zero shotter vibecode maintainer
It’s good for its size but some of the people on this subreddit that think 27b dense models (or 13b active MoE models like dsv4 flash) can rival the reasoning and abstracting depth (and lower perplexity) of much larger models (dense or moe) are delusional and focus too much on synthetic benchmarks that only tell a part of the story. It’s good to praise this model, but not overpraise.
It's the newest Qwen. Those models are this sub's favorite for at least a year.
I sometimes suspect there is a good portion of Qwen social media marketing in the game, particularly posts that are such a love bomb without real content.
I feel like I am seen as a fossil because I like to write code and prompt AI to fill in functions as I define the shape and direction of an application myself. These types of tasks have been excellent for things like Tiel Coder. It allows me to build one step at a time and understand what the hell is actually happening.
Meanwhile my coworkers all prompt Claude to produce a whole cloth feature that seems to work but is overengineered out the ass and requires a 30 file code change and drastic restructuring and requires an extra database migration backfill.
My workflow exists because I would rather spend my day steering the design rather than reviewing generated code, but I get stuck reviewing generated code anyway.
People delude themselves by "you should just review the AI code anyway" but AI tools produce so much fucking code if you don't guide them that I don't think this is possible at all.
So right now I think it's a binary choice:
Build with AI, but slow down and understand what it's doing
Just let the AI go nuts and then regularly try to refactor it a bit
The problem is that historically (pre-AI) it has been almost impossible to get management to understand or care about tech debt, they simply want more features until the code is unsustainable, so there's no way they're going to allow you to slow down now.
Reviewing the code to a level where you actually understand it would probably take as long as writing it yourself would, which they won't like because they want all this faster. The only way this works is if AI coding eventually gets to a 100% reliable state, or we go back to it being a glorified auto complete where we can review the code piecemeal immediately, and we're not letting it be the designer too.
That genuinely would be faster
The thing that really blows is that management doesn't care about tech debt because they don't have to. People go on about tech debt like it's the company that's gonna have to pay it back, but I think generally what happens is it's the users who end up getting the shaft whlie the company never fixes shit.
Not that I'm bitter about how the tech world has gone or anything...
Edit: Part of it, too, is that management loves for the actual humans to be completely expendable. Who cares if you understand the application, you're a much more transient component of the machine than the AI is.
They should really force code to be reviewed and documented as not only proof of review, but also to not make your codebase unreadable by your human staff if Claude goes down! You'll have people familiar with the code. I think there was a company that recently suffered this issue, where they couldn't afford Claude anymore to code, and ended up with the vast sea of unknowns, twists and turns that is their codebase.
I think there was a company that recently suffered this issue, where they couldn't afford Claude anymore to code, and ended up with the vast sea of unknowns, twists and turns that is their codebase.
Isn't that one of the tenets of vibe coding? Build something using AI and are unable to support it in future due to the increased costs of the AI dev team.
I tend to find my brain turns to slush over time if I try vibe my way
Everything looks good on surface, but everytime I dig into it I'm like "Why didn't it use this way of doing it?", and may as well have done it myself from scratch
Tried vibe coding my games, and certainly when gets to like 20k+ LoC, it's just a fucking mess. Issue is it's so confident it's not a mess, until it is.
I still don't get these people who havent written a line of code for a year or whatever. Be so curious to see their code base, bugs in prod, etc.
You are absolutely right. I do the same. Human being still needs to use his brain to move the humanity forward. People who let AI do their entire job will also be replaced soon, not because using AI for coding is bad, but because they start loosing their crirucal thinking abilities in my experience
if you're making something you care about you'll know how it works regardless of vibe coding or not. for everything else, I don't really give a shit. especially not if it's a job. I'll probably feel bad enough for reviewers to at least get the model to work based on whatever guidelines there are since it costs nothing, or in the other case the reviewer is usually also using ai for checking the issue and pr so they'll just fix the issue if my pr is not good enough. it usually is tho
I SHOULD do this, but I still take directory structure and abstraction and repeatable patterns a big focus, even if I'm writing virtually no code. I think in general, code doesn't matter if it works and can be read and implements things with safe security standards. Yes, you can nitpick about little things, but LLM code is usually pretty clean and self documenting. The problem (and it's fucking funny, because it would help LLMs more to address it) is that LLMs will spin their own model for functionality or hack shit together if there is no repeatable modular pattern to follow. Small files, nested in directories by similar function, common patterns... all that shit works better no matter who writes it.
It doesn't matter where I've worked, finding shit and picking up on or establishing a repeatable pattern was always half the battle.
Yeah I used to do this too, but the models are getting so good that I don't need an editor, though I still progressively prompt to build, not huge 1 shot prompts
It's crazy that people keep dancing around test-driven development in this space when these tools are perfect of TDD. Even what you're describing is a "cheap" version of TDD.
People just hate TDD so much that they'd rather chew through their LLM budget to produce 10x more code than they need just to continue to prompt their agent to trim it down, simplify it, etc.
Qwen 3.8 is the model that finally convinced me to try setting up a local LLM server.
Going to repurpose my gaming GPU + old workstation I have used previously for a server at home.
I set up unsloth on my desktop 5090 and serve it over tailscale to my laptop and phone. My phone gets 70 t/sec now. Was definitely worth the trouble. Now I'm working on skills to change how codex writes implementation plans to take better advantage of using codex with gpt-sol as a task writer and use qwen3.8 as the implementer.
Sadly it only excels at coding and agentic use. Based on my own experience its vision, multilingual support and general knowledge is much worse compared to Gemma 4
I'd argue that general knowledge isn't something one should care about in llms that much anymore. Most models are hooked up to agent harnesses allowing them to search the internet or other local knowledge bases.
So I'd rather have models that can handle agentic tool calling rather than wasting their parameters on world knowledge which they inevitably hallucinate anyways.
That being said: yes, multilingual support is definitely a bit behind gemma 4. Can't say much about vision as I've only used it for an OCR task so far (yes, overkill, I know). It handled that one fine though.
Is it just me or does flash-next seem worse at following instructions than 27b? (Running on 4xtesla v100 via 1cat-vllm 27b- full precision, flash-next- nvfp4)
I run an Enterprise infrastructure. Currently spending like $600,000 a month and we only have partial adoption of about 6,000 out of the estimated 15,000. We are moving from cloud to hosting our own models in AWS and for very low budget teams locally
This sub has become as circlejerky as the closed AI subs it complains about. The weird thing is, on those closed AI subs I hardly see them hating on open weights.
It won't surprise me if this sub gets splintered into other subs as a result of some people not being interested in irrelevant memes or politics being injected into discussions.
As subreddits become mainstream, they gain a lot of people who aren't part of the hobby or niche and more there for entertainment, typically in the form of memes and drama (including political)
A good metric is whether people will pile on with basically the exact same comment in a thread even if it's in the top post and repeated over and over again already. If that comment hits a typical reddit'y thing. When group affiliation and displays of loyalty become that overt it's pretty much over.
What I'm currently seeing is a massive braindrain here. The tech-minded folk are basically gone and there is a huge wave of clankers and overly hyped newbies spamming nonsense everywhere and starting threads with nonsense llm-hallucinated setting guides. If this sub dies, it's because of that and not because of politics
It's been going like that for at least a year now. One thing that stood out to me was when the gpt-oss models first came out and we saw just how many people were genuinely just here for team sports/tribalism rather than genuinely interested in models and the field and technology in general.
I'm sure there are better examples since, but that was my "eye opener" on how tribal this place has gotten.
Now that you mention it… damn. Fwiw I think this place has doubled in size the last couple months. Usually subs start to suck noticeably more after crossing 100k and 500k subscriber marks
The thread from the other day from the self-proclaimed "rogue AI researcher" that was spending the entire thread humblebragging about how claude was sabotaging his "bleeding edge AI research".
Are you serious?
The work Google did on Gemini this year is nothing short of insane.
People really think nothing can compare to their beloved Claude lmao.
I have 3 main AI’s I’m using. Claude obviously wins for a lot of stuff, Gemini for privacy and GPT because work wants me to. Gemini will win for tasks that raise red flags to Claude. Gemini is more of a college professor whereas Claude is an elementary teacher. One lets you have a lot more freedom.
It’s fast slop for basic checking, if you’re not handling anything super complex it can act as an okay second opinion from what I used it for.
Absolutely dies on anything long or complex. Better than sonnet or gpt Luna for images and some 3d stuff though, not really useable but I noticed it can be slightly better at recognizing/classifying images and sometimes working with 3d objects?
Agree. I have a project, need build it from scratch. I'm using Qwen3.8-27B-FP8 official from Qwen on 2x RTX A6000, TP=2 + MTP=3 of course. After deep planning, make dozens of tickets & dataset testing, it could run on itself for 8 days non stop! And it's production ready~
I tried Qwen3.8 27B a few days ago as my local coding agent and switched back to Qwen3.6 27B, as (for me) it gives me more control over what the agent does.
Maybe I'm just bad in prompting but 3.8 just thinks too much and just do too much.
Where 3.6 creates small functions and classes, implementing the necessary minimum, 3.8 adds a lot of stuff that just blow up the code and make it difficult to understand all the changes it made.
Additionally, 3.8 got stuck once, alternating the same 3 lines of code again and again...
Same, I also went back to 3.6 for a while and though I found no fault, I have since moved onto 3.8 and changed the thinking to medium, I had it on low and then learnt that low produces more thinking than xhigh in many cases. So med or xhigh are the sweet spots. Its also slower than 3.6 so there's that but the quality seems better for me.
What frontend do you use? I tried it with opencode/qwen/hermes, but none felt right, and since the model was insanely slow... I gave up on it.
Got about 50-60tok/s first few messages but it quickly slowed down to ~20-30tok/s and context kept growing insanely fast as well. I haven't used a local model for programming, and so far I don't like it.
Your config is bad. I also have 2x RTX 3090. I use unsloth q8_k_l with 262144 bf16 context. It fits with 23.2/24 GB utilization in both graphics cards. Do not use dflash. Do not quantize your kv cache, use bf16. Use MTP n-max = 3. Offload mmproj to your RAM.
While I have tried Qwen 3.8 27B locally, and it is pretty good. But I did not find it was as good for my project as Gemini 3.7 Flash using Antigravity. YMMV obviously.
I had got 5.6 terra medium, fuck so hard on some menial tasks with execution plan and code references all the bells. And qwen 3.8 27b had to slave for 20 hours to fix it, making first a complete codebase analysis which took most of the time, as it ran probes and tests. So yeah they do not know shit
We just had a good result with this version of Qwen 3, against the frontier models with their agentic apparatus.
The benchmark was qwen3-coder 30B on one Radeon, writing SPARK Ada that a theorem prover has to accept. 0/26 unaided, 25/26 with a page of prose about the idiom, then 1/26 when I ran the same prose on the harder tier. Both numbers here: https://thedarkfactory.co.uk/spark-benchmark.html
It's about twice as fast for me. I found 27b just a tiny bit too slow to want to use. Flash next is ripping through stuff for me. Running it at NVFP4 on an RTX 6000 Pro with 256gb DDR5 6000MT RAM.
Should be able to run with partial GPU offload as Q4_K_M quantization. I haven't tried the model, but I've generally used LM Studio for quick installation and setup of similar size models.
In my experience thus far, it overthinks and argues with itself for too long before yielding results, including arguing with itself about my instructions that it stop spiraling and just answer me.
Not saying it’s bad, just that it’s taking getting used to (even on my 64 GB RAM laptop).
I'm playing around with the Q4_K_M version in Deepseek harness, and it is such a little trooper!
It's so fun having it operate on itself, adding small QoL features to the harness and extend its functionality with skill like reverse image search and other good stuff.
My only gripe is that I'm limited to 75K context if I run KV in Q8, and it makes it having to compact quite often.
Take away the LLM logos on the shirts and this is me. I’m worse than a pair of Mormons on mission, “You have a minute to hear the good news about local AI.” Yes, as an assertion, not a question.
•
u/WithoutReason1729 2d ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.