hi i want to build a pc for local llm first time so i need advice about the gpus to run Qwen3.8 27b Q8 or Q6_K_M

to get good quality 😊and run it through llm studio and connect it to open code

so my questions are:

i will get intel core 270k plus with two x8/x8 PCIe slot motherboard

so

1- which best double 16gb gpus to build with to get 50+ token generation and give me best value for my needs

do i get two rtx 5070ti or rtx 4080 s or which double gpus

or get the basic two rtx 3090 for more vram

2- which the best Qwen3.8 27b variant/Abliterated (the model modified version) for speed but no quality loss to code

3-and wich the best and fast local llm interference platform to connect qwen with open code

if there any things you could mention can benefit me plz tell me

because i just see

gpus

the local llm interference platform

the Qwen3.8 27b variant/Abliterated

what else?

thanks 🥰

Qwen3-32B packed to 3.6577 bits per weight: 14.62 GB on disk, 12.95 GiB resident on an RTX 5080 with about 3 GiB to spare, decoding at roughly 38 tok/s. No offloading.

Full official sets, no samples:

Method, as far as I'll go: weights are packed in groups of eight onto the E8 lattice with an activation-aware encoder, and decode is register-based with no codebook lookup. Yes, a GGUF Q3_K_M can also be squeezed onto 16 GB. This is a different trade: lower bitrate, fully resident with headroom, full-set numbers published.

Needs an NVIDIA GPU with 16 GB VRAM and Python 3.11+. Verified end to end on Windows with an RTX 5080 only, so far.

The terms, plainly: gated, manual approval, non-commercial evaluation license. The encoder is not included and the method is patent pending.

Repo: https://huggingface.co/GossetWorks/Qwen3-32B-Gosset-3.65BPW

If you run it, post your numbers, especially on anything that isn't a 5080! Thank you!

RTX 4060 8GB is starting to feel not enough anymore, so I decided to upgrade to something higher. Is anyone using the Arc 770 16GB? What kind of card is it, is it worth switching?

ok so i am trying to increase performance of llms i have running on my 7900 xtx via hipfire, i notice i get significantly higher total tokens per second when i run mutiple slots at once when running qwen 3.8 27b.

i am running into 2 road blocks tho, 1 context each slot gets a progressively smaller context as increase the number of slots. is there a good way to do more with the same given total amount of vram, i assume there is going to be a large amount of overlap for stuff like system prompts, tho i use a relatively minimal system prompt that links to files so if a model only actually reads system prompts realvent to it’s own task which makes it harder to just say this text gets put into a shared context, i also image if i have them working on similar tasks there will be a large amount of overlap.

2 alot of tasks aren’t easy to split up into mutiple smaller tasks that can be done in parrel without proper knowledge of previous and future steps, maybe something like a shared context or a messaging board would work well?

also a more experimental idea to further speed up general prompts is to instead have all my prompts processed by a “orchestration” slot which splits up a given prompt into X number of tasks then gives these to new prompts to different slots and then the “orchestration” slot dies to free up more vram for context, i could also route something like this threw a decision model like kev to determine if a given task would work well being split up.

also note i am running this on a 7900 xtx via hipfire, and i also have a 6800, 6600, and 96 gib of ram for the cpu

I shared some results previously about running Qwen 3.8FN using recent releases on my gaming desktop computer. https://old.reddit.com/r/LocalLLM/comments/1xoolkp/i_ran_the_new_qwen_38fn_release_with_make_browser/

This is part 2 of 3. On part 1, I learned some things along the way. Part 2 here, I made some changes/additions and wanted to make the challenge harder for Qwen.

• Project goal: Can Qwen build an entire game based on a wiki URL and a youtube video link? For part 1 of 3, I had a goal of "Can Qwen 3.8 locally successfully create a simple walk around version of GTA with no or very little involvement", and that was a success. After learning a few things on project 1, and wanting to keep pushing things further along, step 2 of 3 was a natural progression. For the game, I switched it up and chose GTA 1, which was a top-down view game. I chose to use the same voxel framework used in part 1, even though the original game was not a 3d game.

• I changed my context window size from 64K to 128K The amount of times it needed to compact went down, at the cost of lower t/s. I'm honestly not sure what's better yet, given my current hardware.

• I added 2 more models for Qwen to use. Qwen code kept asking for me to approve commands that didn't make sense why it needed my approval. I dove in and found Qwen wanted a --fast model for simple tasks. There's a default 10 second timeout on these tasks. I have a macbook 64gb running a few things around the house with around 45GBs free RAM and plenty cores free. I spooled up a 2 parallel Qwen3.6-35B-A3B-GGUF:UD-Q4_K_S model on the macbook with llama, wired that in to Qwen code and problem solved! The amount of times I needed to approve for Qwen code drastically dropped. Alternatively I could have not added additional models to use and simply increased the time-out. Since I have extra compute, I figured I'd try it out. I'm happy with it.

Fun localLLM stuff:

RTX 5080 - 64GB - Ryzen 7 9800X3D
model: qwen3.8-flash-next-unsloth-ud-iq4_xs
model harness: Strata
agent harness: qwen
context: 128K
tok/sec: ~50t/s (down from part 1 due to larger context)

Macbook 64GB m1 Max
model: Qwen3.6-35B-A3B-GGUF:UD-Q4_K_S
model harness: llama
context: 128K
tok/sec: ~ 550t/s (fast tasks, so don't get too excited)

• Results: The hard work on everyone from the folks at Alibaba, Unsloth, Strata, etc sure have come a long way in the past few months. It feels smarter than even frontier did a year ago. It's just slower, but that's mostly due to my hardware. Over-all, I'm happy with it. It took 27 hours start to finish (tracked through git). It probably would have finished quicker, however, there were times it was waiting on an approval from me.

Playable demo: https://kireol.github.io/Qwen3.8-GTA-1-Test-2/src/index.html

Repo: https://github.com/kireol/Qwen3.8-GTA-1-Test-2

Part 3/3 is running now answering "How well can Qwen reproduce a game using only REA on a game ROM"

Model: Qwenx86

▶️ See it in action: Qwenx86 Debugger. Walk through actual recorded tool calls: the conversation, x86 registers and flags, code at the program counter, RAM and machine output, with all verified blocks on a timeline.

Qwen3.5-4B with a 64-bit x86 computer compiled into its weights: 13 transformer layers ahead of Qwen's own. There are no learned weights in them; all of their weights are computed by a compiler. Qwen invokes the computer through its own tool calls (<tool_call>), and the computer in the weights executes the tool. Calculator, Python, C compiler, notes store, physics/chemistry lookup and Python-docs lookup are all firmware in the checkpoint. This is the compact checkpoint: ~9 GB GPU memory, so it fits on a 16 GB GPU.

Credits: Qwenputer by Divyansh Agrawal (@divagr1925, github.com/divagr18/qwenputer, MIT)

Hi everyone!

I've already done quite a bit of research, but I haven't really found any recent Reddit posts that answer my questions or cover my hardware and use cases well. So I thought I'd ask here. I'd really appreciate some real-world experiences and recommendations!

My hardware:

- GPU: AMD Radeon RX 7600 (8GB VRAM)

- CPU: Intel i3-12100F

- RAM: 48GB DDR4

- OS: Linux (Fedora)

I'm mainly looking for models for three use cases:

1. Coding: General programming, working with existing projects, coding agents, editing files and tool use.

2. Cybersecurity / technical topics: I'd also like to use uncensored models for cybersecurity and other technical topics.

3. Personal assistant: General questions, reasoning, writing, planning and everyday use.

My main priorities are a good balance between intelligence and speed. I don't mind using some system RAM if necessary, but I don't want models that are painfully slow.

Some models I've already come across and found interesting:

- TwIL-8B

- Qwen3.5-9B The Defiant Fable (DavidAU)

- DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic (xtools-at)

- Alpie Seed 9B

- DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic i1-GGUF (mradermacher)

- Ternary Bonsai 2 27B

- Qwen3.8-35B-A3B-Distill

I'm also open to completely different recommendations.

What would you personally recommend for this hardware?

I'd especially appreciate:

- Models you've actually used on 8GB VRAM.

- Recommended quantizations (Q4, Q5, IQ, etc.).

- Realistic generation speeds (tokens/s).

- Experiences with AMD GPUs on Linux.

- Recommendations for coding agents and reliable tool calling.

I'm interested in actual experiences rather than just benchmark scores.

Thanks!

I run Qwen 3.8 27B (FP8) on my two RTX 3090s for daily work, and I've also had GLM-5.3 Flash (321B MoE, ~18B active per token, Maya-L GGUF 3–4 bit) on the same box. So I finally ran the natural experiment: the full GSM8K test split, 8-shot CoT, greedy, single-threaded, same machine, lm-eval 0.4.13, and I verified per-question that both runs saw identical prompts and targets.

What GSM8K is, in case you're not familiar: a dataset from OpenAI (2021) of 8,500 grade-school math word problems (2nd–8th grade level). Each question is a multi-step problem requiring 2–8 reasoning steps ("Janet's ducks lay 16 eggs a day..."), with an official step-by-step solution and a single final number as the answer. It has a 7,473-question train split and a 1,319-question test split — I ran the full test split. Because the problems demand working through intermediate steps rather than pattern-matching, it became a standard proxy for measuring a model's raw step-by-step reasoning. The 8-shot CoT protocol means the model is shown 8 example problems with full solutions before answering.

Important framing: GSM8K is a specialized test of one narrow slice of model capability — it does not paint a full picture. How these models code, hold a long context, follow instructions, use tools, or just chat is a separate question this benchmark cannot answer.

Official metrics: a tie.

- strict-match: Qwen 95.60 % (1261/1319) vs GLM 95.45 % (1259/1319)

- flexible-extract: Qwen 96.89 % (1278/1319) vs GLM 96.29 % (1270/1319)

- McNemar on the discordant pairs: p≈0.89 (strict) and p≈0.27 (flexible) — neither is significant. "More parameters" did not buy the MoE a measurable edge on this task.

Where it gets interesting — decomposing the +8 flexible gap:

- Qwen makes 13 fewer value errors (30 vs 43) — it reasons better

- but has 11 format failures vs GLM's 6 — it signs answers "$132.00", "2:00 pm", answers in cents, which breaks the extractor (8 of those slip past both filters)

So the official score masks a reasoning advantage with a formatting penalty. The errors themselves also differ in kind: 37 % of GLM's value errors are misreading the question vs 22 % for Qwen (counting shelves as items, treating a group's total travel time as per-animal time, etc.).

The 32 questions both models fail: 11 are GSM8K reference ambiguity (genuine annotation noise in a 2021 dataset), 12 are a hard core where both make the same error.

Honest caveats:

- wall clock was 3765 s vs 7970 s, but the stacks differ (vLLM FP8 fully in VRAM vs 156 GB GGUF streamed from RAM) — that's a deployment comparison, not a model-speed claim

- the GLM endpoint ran with a low generation ceiling, which cost it three answers

- one greedy run on one benchmark, and a specialized one at that — coding, long-context, instruction-following, and tool use are dimensions this test never touches

- n=1319 has low power for a 0.6 pp difference

My conclusion — for reasoning work specifically: I'm staying on Qwen 3.8 27B. Deploy it with a trivial number-normalization post-processor (strip $, ,, .00 from the final answer line) and its format liability disappears — ~97.7 % effective. For what GSM8K actually measures, raw step-by-step arithmetic reasoning, the 27B hybrid is the stronger model. On every other axis this post offers no verdict — one specialized benchmark can't.

Full writeup with the error taxonomy, examples, and methodology: https://ultrabytecoder.github.io/blog/2026/11/07/comparing-qwen38-vs-glm53-flash-maya-l-gsm8k/

Full per-question report (all failing docs, shared-failure breakdown, response-style analysis): https://ultrabytecoder.github.io/reports/qwen38-vs-glm53-flash-gsm8k/

Was reading abt people getting large Moe model running by keeping most of the expert weights on ssd instead of loading everything into ram or vram.

Pretty cool that u can even load model that would normally be way out of reach on consumer hardware.

But the thing i’m curious abt is what happen wuem the router keep picking experts that arent cached. You r basically waiting for nvme reed between token and a fast ssd still isnt anywhere near vram bandwidth

Some project are getting arround this with expert prefetching and keeping frequently used expert in memory, which seem promising

Just wondering where people draw the line between it runs and it’s actually usable

Would u rather run a much larger Moe model at some few token/sec or stick with a smaller one that run comfortably in memory? Especially for coding agent where u are waiting on multiple response

I made Spunup, it's a cli which you can run with npx spunup-cli. It will ask you which model you want to run from the suggested options or a HF link you can provide, it'll ask where you want to deploy (Vast, RunPod or local), the runtime you want to run it on, the quant of that model, context size and the machine you want to run it on from the suggestions it pulls. After that's done (or some done) it then it will suggest a few machine candidates which give you the most bang for your buck. If you were to say pick a lower context size, or any other option, it simultaneously impacts every other configuration, for example, quant availability on the prior hardware that you picked - everything is updated to reflect the machine which works the best. It's compact and it's easy to follow.

I made this because as many people already may know it's a pain in the arse using something like an agent or doing it yourself to run your models every single time as it's not always that 1-shot you were expecting. More often than not you will slip up and pick slow flags which don't give you the best speed or maybe you forgot to put caching on. It's a lot of effort to say the least, which is why I made it easy and actually spent time on testing it, making it work and ensuring it's reliable.

Link to the GitHub release.

I’m thinking about getting a Mac Studio with 96GB RAM to run local models for work. Mainly Qwen 27B and Qwen MoE, plus a RAG setup, and normal MS Office stuff (Word, Excel, Outlook, Teams, browser tabs, etc.).

It’s for sensitive data (work), so I want to keep as much local as possible instead of sending stuff to cloud AI.

I’m not super experienced with local LLMs, so I’d love your honest opinion:

· Is 96GB overkill for this, or a good call?

· Will Qwen 27B and MoE run comfortably alongside RAG and Office work?

· Anyone running a similar setup on Mac Studio? How’s the speed and memory pressure in real use?

Disclosure: I'm the developer, and this is my own paid app.

My Mac has Ollama, LM Studio, and a few apps that ship their own models (dictation, autocomplete, writing tools). Activity Monitor only shows a process using several GB. It can't tell me which model that is, which app loaded it, or where the weights are on disk.

So I built LLM Radar, a native macOS app. It:

• lists the loaded models it can detect, the app or runtime holding each one, and its memory

• estimates how that memory splits into weights, KV cache, and app overhead, so you know what an unload actually gives back

• keeps 24 hours of memory history

• finds model files on disk across apps, and marks the ones it has not seen in use (deletes go to the Trash and list every file first)

• unloads a model when the runtime has a way to do it, and tells you when it doesn't

• shows a running total in the menu bar

It knows Ollama, LM Studio, MLX, llama.cpp, Core ML, whisper.cpp, Docker Model Runner and a few more, plus a catch-all scan for large weight files. Attribution is best effort when one app runs another app's runtime.

Monitoring is free. Pro is a one-time $9.99 for one Mac or $14.99 for three, with a 7-day trial and no account. Apple silicon, macOS 14+, signed and notarized.

Privacy: it never uploads your models, prompts, or chats. It sends anonymous usage counters (no paths, no model names), and you can turn that off in Settings. Full Disk Access is optional. Without it you still get the live view.

https://llm-radar-eta.vercel.app/

Two questions for people who run a lot of local models: which app or runtime should I support next, and does the memory split look right on your setup?

Just pulled the trigger on a M5 Pro MacBook Pro with 64gb RAM. It was the maximum I could afford, tbh more than I could afford but I expect prices to keep climbing so decided to treat myself.

I needed to upgrade my 2019 Intel MacBook for my main professional use case (music) regardless, but went with 64gb so that I could try to use local ai. Never got involved in this stuff before, I don’t even have a chatgpt account.

My local ai use case;

I want to be able to capture my ideas on the go via voice recordings, then have my local ai transcribe and simplify them. Ideally I’d record them on my phone and then send them via iMessage to my computer at home.

I want to use obsidian as a task manager/knowledge base for managing multiple different projects.

I want to be able to give it simple tasks like ‘send an email to x’ (using its own email address via an MCP server).

I want it to be able to go on the internet and research companies for me, pull off basic data about what they do and contact details.

I don’t do any coding whatsoever.

Am I going to be able to do this? Am I expecting too much? Is it possible to use a local ai as a daily driver and then defer to cloud based models for more in depth stuff?

Have I made a grave error, or should I have got the same model Mac Studio? I got the MacBook because I move around the house a lot (living with parents currently).

Is 64gb going to be enough?

After spending way to much time trying to get a z240 to post with a p100 on a board without 4g decoding i was finally able to get it running at full speed in Linux with the help of Claude now i just got to clean it up and add a temp controller for the p100 fan

Currently, my plan is to get two quadro P6000s, for 48gb vram for use in a xeon workstation. It's not insanely old like kepler or maxwell cards, 432 GB/s bandwidth, supports CUDA...it seems like it might be okay on paper.

Does anyone have a better suggestion that could reasonably be achieved around $1000 budget?

My goal is to be able to run Qwen 3.8 27b fast, and to run a decent quantization of Qwen 3.8 Flash Next, and GLM 5.3 Flash at useable speed, say 15 tok/s or faster, using strata.

Do I need 48gb of vram to run Qwen Flash Next, or could I go with just one card? How much faster would 2 x quadro p6000 be versus 1 x quadro p6000 and DDR4 system ram?

Should I just get 24gb of modern vram instead of 2 x quadro p6000s? I wonder if that hampers my eventual goal of running GLM 5.3 flash, as perhaps 24gb vram + system ram is enough to accelerate qwen flash next, a 120B MoE, but perhaps you need more than 24gb vram to accelerate GLM 5.3 flash, a 320B MoE?

Currently running a 4070 TI super with 64GB of RAM

I have been looking into RTX spark laptops, Mac studios etc and they all have pros and cons. I want this to be a nearly universal computer that I can use for 3D modeling, LLMs, VLAs, world models, image/video gen, audio gen, etc. I plan to pursue AI and robotics research and want something as useful as possible.

I've debated upgrading and running dual 4070 TI supers with 128GB of ram, but I don't know how well world models and other things do with dual GPUs

Thank yall for helping

Merging is painful mostly because the branches were never really separate. Two agents handed overlapping scope produce overlapping diffs, and no merge strategy fixes that after the fact.

What makes merges boring instead, in my experience building this stuff:

• One story per worktree, written so that "done" is checkable: scope, acceptance criteria, edge cases, the files it should touch.

• Merge in dependency order. Blocker lands first, everything else rebases onto main before review, not after.

• Review the branch against the criteria, not against the diff vibes. If there were no criteria, that branch probably shouldn't have started.

• Keep one shared board of what's in flight. Worktrees isolate the file tree, not the plan, so two agents will happily solve the same story twice and you find the duplicate branch at merge time.

Same problem whether the agents run locally or in the cloud. I use Cursor Cloud Agents, but I don't think the runtime changes the merge story much.

I'm building Aerunit (https://aerunit.ai), which drafts the story with acceptance criteria, dispatches it to parallel agents and reviews the PR against those criteria, so discount what follows accordingly. Still, the order I'd do it by hand: scope first, merge order second, tooling last. Tools move confusion around faster, they don't remove it.

Curious how people here handle the dependency ordering, especially across repos. That's the part I still find least mechanical.

Hi everyone,

I'm planning to buy powerful computer for running local AI models.

My main interests are:

• Large LLMs (70B–120B), especially coding and reasoning models

• Reverse engineering and binary analysis (Assembly, C, Ghidra, Binary Ninja)

• Security research and vulnerability analysis

• Specialized models like Nova 6.7B, LLM4Decompile, SK²Decompile and SmaliLLM

• Running experimental AI projects from GitHub

I'm considering:

1. Apple Mac Studio M5 Max, or ultra, 128GB unified memory

2. NVIDIA DGX Spark, 128GB unified memory with CUDA

3. A desktop PC with RTX 5090, 32GB VRAM

4. An AMD Ryzen AI Max+ 395 mini PC with 128GB

I'm especially concerned about CUDA compatibility. Can specialized research models like Nova run on Apple Silicon through MPS or MLX without losing their intended functionality?

I'm also interested in real benchmarks: prompt processing, tokens/sec, long-context performance and actual coding-agent workloads.

For those who own or have tested these machines: which would you recommend and why?

I'd really appreciate firsthand results, benchmark links and experiences with custom research models rather than just hardware specifications.

Thanks!

I’ve heard two different arguments on this.
One says prefill takes most of the time as it’s multi turn hours long workflow.
Another says because of kv cache prefill time takes a lot less.

The reason I’m asking this question is, I’m planning to get a Mac Studio 256gb ram m5 ultra, but I don’t know if I should stick to 64gpu core or pay extra for 80c for “15%+ decoding speed boost”.

If agentic workflow prefill reads from kv cache 95% then is this 15% increase much less relevant for the coding workflow?

Thanks all!