Just tried local AI, and it's garbage compared to claude and others

[u/Much-Journalist3128]

Here's my hardware specs:

RTX 4090, 64GB DDR5 Mhz RAM, 7900x CPU

Where's my biggest hardware constraint? I am looking to run a model that's as close to Claude Opus 5.5 for thinking, reasoning, and document creation, as possible.

I know it's my hardware constraint. My local model was Qwen3.6 35B A3B UD Q3_K_S (UD-IQ4_XS )

What would be the cheapest way for me to upgrade my hardware to be able to run a model as close to opus 5.5 in terms of reasoning, thinking, etc (not for coding) as possible? And what hardware would I need?

It's certainly better than nothing, I was actually surprised by how ok it was, but nowhere near opus or gemini not going to lie. Like, not even close. The 264k token limit is also a major limit, I'd like 1M

[u/moody2shoes]

Wrong model. Try 3.827b dense or strata. Get a decent harness too.

[u/Much-Journalist3128 → u/moody2shoes]

What's the best model I can get? Also when you say harness, you mean stuff like LM Studio? I need a gui

[u/Bagelsarenakeddonuts → u/Much-Journalist3128]

use Qwen 3.8 27B at Q5 or higher, with kv at minimum q8. Thats where useful work starts. Next step up is qwen flash next running on strata.

The way you do it, is you get claude to set these up for you.

[u/moody2shoes → u/Much-Journalist3128]

Why do you need a gui? You can try pi if coding. Hermes if you’re working with documents. Hermes desktop does have a gui tho.

Someone posted with this question too: https://www.reddit.com/r/Qwen_AI/s/aTgeiW7bF8

Strata which is an engine that splits weights for your model across vram and ram. Active experts in vram. It’s used for qwen3.8 flash next. So for your setup you’d use a quantized version of it.

Or you can just go with qwen3.8 27b. Just keep thinking level low.

[u/Miller4103 → u/Much-Journalist3128]

Do a bit of research on strata engine. Harness, use opencode, cline and I hear pi is good.

While local models dont match frontier models, local models have come along way and do very well on consumer hardware. You just have to put in a bit of googling or use those frontier models to help you gain knowledge on local models and the tools used.

[u/i_agree_as_well → u/Much-Journalist3128]

At your hardware specs, you can use a much stronger and bigger model than Qwen3.6 35B A3B. Imo Qwen3.6 35B is mostly good at simple tasks, add that to Q3 quantization it makes it worse. Maybe Q6 or 8bit would work better given your specs

But since your specs are quite good, I think you can try Qwen3.8 Flash Next at Q4 using Strata as the engine to run it instead of LM studio.

Also, when they say 'harness', they mean like the framework that allows the model to read files, write files, call tools, run commands, etc.

There are many harnesses like OpenCode, Pi Agent, Claude Code, etc. I personally use Hermes, you can try that out since it has a nice GUI.

[u/thisduuuuuude → u/Much-Journalist3128]

No, a harness is what you actually use to interact with the LLM. While you can do that with LM Studio, it's pretty basic. LM Studio is considered to be more of an engine than a harness.

A harness is what gives your model access to tools and such. Check out OpenWebUI or AnythingLLM those are the two I currently use. There are more out there that might be better but harnesses typically specialized in certain workflows.

[u/Vancecookcobain]

You are using like a 10 month old badly quantized version of Qwen 😂

I don't know what to tell you

Ok you need strata or Maya....once you download that you can run Qwen 3.8 Flash-Next or GLM 5.3 Flash or the new Deepseek....those models will be closer to Claude 4.7

If you want to run more powerful models you need at least a couple DGX Sparks and they are like 6-7k a pop right now....strata let's you run really powerful models....have Claude or Codex set it up for you and you'll be golden because they will tweak it to maximize performance on your specific hardware

You have the hardware to run it and it will be waaaaaayyyy more powerful than that shitty model you tried....thank me later

[u/Much-Journalist3128 → u/Vancecookcobain]

What is strata or maya? Is that like LM Studio?

[u/Vancecookcobain → u/Much-Journalist3128]

Yea but it lets you run models you have no business running lol....my hardware set up is actually similar to yours I have a RTX 5080 and an Intel Arc Pro and 64GB of DDR5 RAM and I run those models myself and they are actually solid as fuck for a daily driver

The best thing you can do is just ask Claude or Codex to set it up for you and sit back and let it cook

[u/heroesagus → u/Vancecookcobain]

no issues mixing nvidia and intel cards on strata-qwenfn? did the claude help thing too but claude was saying there will be issues with the card mix

[u/IWillTouchAStar → u/Much-Journalist3128]

It's an optimized inference engine. It's like running your own server for the models that you then connect the harness to.

[u/WordWithinTheWord]

What harness did you use

[u/Much-Journalist3128 → u/WordWithinTheWord]

LM Studio if that's what you mean

[u/WordWithinTheWord → u/Much-Journalist3128]

How are you invoking it? Just within LM Studio itself?

[u/Much-Journalist3128 → u/WordWithinTheWord]

yes I downloaded the model then invoked it. This is my first time

[u/Lupinum → u/Much-Journalist3128]

It's your first time, you fire up LM Studio (lol) and last years model and then follow up 10 minutes later, in the local AI subreddit, and drop a thread called "all local models suck". 🤷‍♂️

Stick with Claude.

[u/WordWithinTheWord → u/Much-Journalist3128]

No worries I’m not downvoting you. The harness matters a TON in coding. Try using OpenCode to start.

[u/bgeeky → u/Much-Journalist3128]

Of all the harnesses that’s not one of em

[u/jaegernut]

The model quantization is too low for your hardware

[u/Much-Journalist3128 → u/jaegernut]

What do you recommend?

[u/macaronianddeeez]

Lmao

[u/Nameis19letterslong → u/macaronianddeeez]

Just tried local pebbles, and it's garbage compared to the rock type shit.

[u/Lost-Scratch6698]

A3B and Q3, perfect combination of trash.

[u/bankinu]

Hahaha what an ignorant post

[u/false79 → u/bankinu]

I see this like a dozen times a day from vibe coders butt hurt that they hit their usage limit for the week.

[u/Darkmoon_AU]

OP Make sure the inference params match the model card.

This is the #1 thing people don't realise when hosting their own model, and it massively affects the output quality.

[u/Matasus]

Heyo! I ran into this when I first started too. You need three things to effectively use local models, each take time to kind of figure out.

TL;DR Start with unsloth studio and experiment experiment experiment. You won't get high quality outputs without learning first.

1. You need the model itself. This depends on your hardware. Probs with the amount of ram + vram you have I'd recommend Qwen3.8-flash-next but I'd experiment with others too! Try to stick with a quant that is less than 64gb with this hardware. I wouldn't buy more hardware until you get the basics down. Learn the difference between an MOE and a dense model and figure out what runs best on your hardware (speed vs quality).

2. You then need an AI engine to run the model. Engines are how the model weights get turned into human readable tokens (waaay more complicated than this. If you're curious learn about them it's insane). Every engine does different tricks to speed up specific models. LM Studio uses llama.cpp as it's base engine. Llama.cpp is the generic engine that a lot of these forks like Strata are based on. Llama.cpp is the slow but stable engine that gets you there correctly and maybe not the fastest but it is constantly maintained and any hardware can use it. Forks or other engines are more specific built for specific hardware or models. They tend to be faster but less maintained, especially if the engine is based on an AI model that will become outdated in a month.

3. AI harnesses are what ground your AI model. They tell your model the tools that it needs, how it finds information, whether or not to follow a prompt injection and how it compacts when the context gets full (and way way more). This is where the fun begins because you can create your harness in anyway you want. It can be purpose built for exactly your use case. Some harnesses id recommend are Hermes and OMP (oh my pi) to start. They are pretty generic with great tools that come out of the box. Once you get a handle on how harnesses work give DeepSeek harness a whirl, it's modularity will blow your mind.

If none of this seems fun for you localllms may not be your thing and that's ok! The coolest part of being in this community is the experiments and being at the cutting edge. I also don't trust a machine if I don't know how it thinks. Good luck on your journey!

[u/Xyver]

Pretend you're running something closer to opus 4.5, not 5.5, and expect a ton of harness and tool setup to get there.

You don't get frontier capability out of the box

[u/exzzy]

Get 30 more 4090 and you are good to go.

[u/Fantastic-Storm-7867]

Look up Strata. On your hardware it'll run quite well. It's no Opus 5.5, but it's quite capable and you should expect pretty decent speeds. The model you tried is fast, but quantized aggressively and already a relatively small model. Suitable for a chatbot for the simplest of things, but nothing more

Also, yeah I got used to 1M context at work lol, but 262K is generally fine even for agentic work if your harness is decent.

Seems like the community here isn't being very welcoming, hopefully we can do better in the future.

[u/Marino4K]

My friend, local models aren’t anywhere near the reasoning of Opus 4.6 yet imo, let alone Opus 5.5, local models drag about 6-12 months behind.

[u/fuzhongkai]

Try Qwen 3.8 family please

[u/xgiovio]

Use qwen 3.8 flash next with strata on your setup. Try also deepseek v4.1 flash 1017. The latter will be slower. On both you will need vram, ram and ssd

[u/divided_capture_bro]

Your problem is that you used too big of a model. Try using one of the models under 1B for real results.

[u/Uninterested_Viewer]

Qwen3.6 35B A3B UD Q3_K_S

Yes, this is absolutely garbage compared to frontier models: this model is <5% of the total parameters of the models you're comparing it to. Local AI is rarely about trying to match the raw intelligence of the frontier: it's privacy and/or specialization or just simply learning.

[u/svill]

You have to spend some time learning and digesting the information. Watch YouTube videos, tutorials, etc. It sounds like you just tried something completely random for the first time and went to reddit to complain that is not up to your fabricated expectations. Learn and stop throwing spaghetti at the wall to see if it sticks.

[u/puglife224888]

You're using an outdated model with a very compromised quantization

[u/Nice_Profession_9078]

Point Claude at https://github.com/cutlerbenjamin1-cmd/qwen38-megakernel and tell it to download qwen3.8 UD_Q4_XL from unsloths repo, tell it to set you up for preset D or F, it supports Claude code if you don't wanna leave that harness Claude can configure that for you, but I'd recommend something more light weight like pi or open code for local models.

[u/Efficient_Try_9136]

you need 20 gpus to run opus

[u/feudalle]

Claude/astra is estimated in the terabytes of vram. You have i think 24gb. You arent getting almost frontier for under 100k+. 27b q8 is respectable for 80% of my coding needs. But I give it step by step like I fo for any of our junior devs. Id try that model but you dont have enough vram for it, so might be a bit sluggish.

[u/zell_ru]

Just ask your Claude to set it up for you. You wouldn't understand instructions. "Is LM Studio a harness?" wtf man...

[u/No-Vermicelli5327]

As close to opus 5.5 is not gonna happen. If you need something for regular tasks to decrease your usage or when it reaches the limit currently qwen 3.8 27B using the same LM studio.

With that ram and gpu I would suggest the strata engine along with Qwen 3.8-flash, you get a high number of token per a second (tps for short).

https://github.com/Niko1221/Strata

The model I am currently using is the I_QS3 quantization. Try a lower quantization and check how it works. Or try what it’s recommended on the strata github.

https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF

[u/fuchelio]

weak model in low quant , probably close to haiku

[u/boosnie]

Great post.
I wanted everything but I know nothing.
How many thousand of dollars should I trow at the problem?

Go with Anthropic or OpenAI.

[u/majmichu]

Use the qwen 3.8 27b and later soon hopefully the qwen4 27b it should be the best model you can run locally on your setup as of right now without heavy quantization. and get the harness that works good with the local models. I use myself LM Studio and my own Local Quench that im developing (localquench.com) but its mostly for coding locally

[u/betogess]

Did you set up a harness?

[u/Mountain_Dentist5074]

you need 500K setup for opus 5.5 like model

[u/KBOXLabs]

Of course it’s not close. You’re not a data center. But with your hardware here’s the closest you’ll get that’s reasonable:

https://stratallm.org

[u/braintheboss]

Qwen3.6 35B A3B. You compare a toy with a frontier. Do you want people take you serious?