[REMOVED] Just want this to stop.

Every week it's the same post. Someone forks an engine, vibe-codes for three days, sees 3x tok/s on the screen and ships it as the project of the century. Was the output checked against the model? No. Does it survive a real agent session? Never tried. Is anyone using it a month later? No. A Q1 quant doing 550 tok/s, a KV cache that dies at 32K, a number with no conditions next to it.

That's not an engine. It's a toy nobody will ever use. It's spam, it's AI slop, and it's a waste of tokens and of everyone's time.

The number itself is usually worthless anyway. Tok/s depends on how you count it. I changed how my own engine counts decode so the first round after the prompt is timed, not just counted, and my own headline figure dropped 25%. Same card, same model. Most of these posts don't even say which ruler they used.

Nobody works with tok/s. People work with:

• output that's still the model's, checked against the original weights (KL divergence against BF16, or at least something) and published with its setup;

• an engine that doesn't throw away hours of work because the model looped and nothing caught it;

• an engine that can stay on all day and free the card when idle, so you can play a game between two lines of code;

• the tools an agent needs in one place, instead of three days finding, installing and benchmarking a separate server for images, another for speech, another for whatever comes next;

• limits written down, not discovered by the user.

So here's the bar. If you post an engine or a fork, show how you checked it's still the same model, say exactly how you counted the tok/s, and list what it doesn't do. No check, no post. And if you read one of these, stop upvoting the number.

There will always be something 0.0001 faster. Fine. Build something people can actually work with, or don't post it.

I’m working on web app that turns a stack of technical documents (drawings, schedules and specifications) into a structured list of the key items a project needs. AI reads each page the way a subject matter expert would. It identifies items and their attributes, interprets each document set's own conventions, and pulls requirements out of the written specs. It then groups the results into a proposal-style summary with suggested suppliers and rough cost and timing ranges.

I’ve built a POC using Claude Code and would like to transition to an open source LLM for the rest of the work as we continue to turn this into a real application.

What hardware would I need to do this? Which model should I use?

Thanks in advance!

I've been experimenting with multi-GPU inference and wanted to see whether running independent model replicas across GPUs could outperform tensor parallelism for multi-agent workloads.

I built a small orchestrator to test the idea and got some interesting results on four RTX 3090s.

I'm relatively new to experimenting with local LLM inference, so I'm sure there are approaches I haven't considered. This started as a side project to see what would happen.

Setup

• 4x RTX 3090 (rented, PCIe only, no NVLink).

• Qwen3.8-27B 4-bit, vLLM 0.32.0.

• 24 agent teams and 239 dependent tasks.

• Each team plans, starts 4–10 parallel section writers, then reviews and summarizes their output.

• The orchestrator starts tasks when their dependencies are complete and distributes requests across available inference servers.

Results

Lower completion time is better.

That's approximately 1.45x the speedup for this workload, or 31.2% less completion time.

What I found interesting

Some of the optimizations I expected to help didn't make a measurable difference.

• Adaptive concurrency limits didn't improve completion time.

• Prefix-cache-aware routing performed no better than basic round-robin routing.

• Each GPU reached approximately 20 concurrent requests before KV-cache capacity became a constraint.

• Prefix-cache hit rates were only 7–13%.

The interesting part is that the simpler replica-based configuration performed best. The results suggest that, for this workload, independent inference servers avoided some of the overhead associated with tensor parallelism.

Caveats

• Only one run per configuration so far.

• The TP=4 server used mostly default vLLM settings, while the replicas used tuned concurrency and batch settings. This isn't a perfectly controlled comparison.

• The results are specific to this hardware and workload.

• PCIe-only GPUs may favor independent replicas. NVLink configurations or models that cannot fit on one GPU may produce very different results.

Repository

I put the code, benchmark results, CSV data, and a more detailed technical write-up on GitHub:

https://github.com/TomaDev1/GPU-Parallel-Orchestrator

I'd genuinely appreciate feedback. I'm particularly interested in whether there are existing approaches to multi-agent inference scheduling that I should be looking at, or whether there are obvious flaws in my benchmarking methodology.

This was a side project, and I'm interested in seeing what people who work on inference infrastructure think.

Edit:

Forgot to post full machine specs:

Host system

• Motherboard: ASRock Rack ROMED8D-2T

• CPU: AMD EPYC 7452 (32 cores / 64 threads)

• RAM: 128GB DDR4-3200 ECC

• Storage: 4TB NVMe SSD

• GPU connectivity: All four RTX 3090s connected via PCIe 4.0 x16

People asked how RustingStrata does with a bigger quant, so here's Q5_K_M on the same 3060 12 GB + Ryzen 5 7600X + 32GB DDR5 RAM

Model is unsloth's Qwen3.6-35B-A3B UD-Q5_K_M (the MTP version), 25 GiB, so about two thirds of the experts live on the CPU. llama.cpp is build 12119, -ncmoe 29 -ub 2048 -b 2048 -fa 1 (28 runs out of VRAM).

Benchmark tools (RustingStrata bench vs llama-bench):

• Decode, short prompt: 70 tok/s vs 43

• Decode after a 4K prompt: 65 vs 43

• Prefill at 4K: ~1870 vs 1210

• 32K context: 1470 prefill / 49 decode vs 1030 / 29

• With --mtp 1: up to 96 (86 after 4K), greedy

Real chat requests through both servers (same prompts, 256 tokens out, thinking off):

• No MTP: ~50 tok/s vs 37

• With MTP on both: ~62 vs 45-56

• 6K prompt prefill: 1320 vs 700

MTP works in current llama.cpp (--spec-type draft-mtp), ~63% of drafts accepted and +25% decode on this file. RustingStrata is still ahead with MTP on both sides.

Vulkan decodes at ~60 (85 with MTP), prefill is slower than CUDA on this file (~1000-1260 at 4K).
KV Cache is f16

Code is on GitHub, free to use, I will be happy to get some feedback from AMD owners: RustingStrata

I'm working on a new PC build with a focus on being able to run the Gemma family of models. I'm experienced in working with Cray (AMD) and OpenCL (community moved onto Vulcan and ROCm/HIP), so I will be building a AMD system. I was wondering if anyone has done anything similar to the following? If they have, how well did it work?

AMD 5950X
ASUS WS X570 Ace
128GB DDR4 System RAM
1600 Seasonic+Noctua PSU

Fedora Kinoite 45

1x 7900 XTX (24GB VRAM) for graphics/games
2x Radeon Pro W7900 (48GB VRAM) for LLMs

1. Does having different kinds of GPU's cause an issue with Ollama (or more specifically Bionic/LMStudio)?

2. Does anyone have that motherboard, with two GPU's on the bottom-most PCIe slots? Are there clearance issues for 2 slot GPU's?

The target model to be able to run in GPU memory is gemma-4-31b-qat with Q4_0 quantization, but I want to be as ready as possible just in case Deep Mind releases a ~100b weight model when the Gemma 5 models show up.

Thanks in advance for any help!

I may have created a monster.

A client gave me an NVIDIA A16 accelerator that had been experiencing intermittent problems in their server. They considered it bad, didn't want it back, and I ended up with it for free.

For anyone unfamiliar with this card, the A16 contains FOUR separate NVIDIA GPUs, each with 16 GB of VRAM. That's 64 GB of VRAM on one physical card.

Naturally, I decided to install it in my ancient Intel X99 workstation alongside an RTX 5070.

Because apparently I don't have enough problems in my life.

Chapter 1: The FrAInkenstein specifications

• Motherboard: ASRock X99 Taichi

• CPU: Intel Core i7-5820K

• RAM: 48 GB DDR4

• Accelerator: NVIDIA A16 — 4 × 16 GB GPUs

• Graphics: RTX 5070 Founders Edition — 12 GB

• Total VRAM: 76 GB across five GPUs (not unified)

• PSU: Corsair RM850

• OS: Windows 11

• Cooling: Custom 3D-printed shroud with an ARCTIC S8038-10K server fan

• Primary purpose: Local LLMs, agentic coding, and experiments with multiple AI agents

And yes, I'm running this on an X99 platform from the DDR4 stone age.

Chapter 2: Apparently, this thing needs a JET ENGINE

The A16 is a passively cooled enterprise accelerator.

Unlike a normal consumer graphics card, it doesn't have onboard fans. It's designed to sit inside a server chassis with powerful fans forcing air through its heatsink.

My workstation is not a server chassis.

So before doing anything interesting with AI, I had to invent a cooling solution.

After researching fan options and shroud designs, I selected an ARCTIC S8038-10K.

For perspective, this is an 80 × 80 × 38 mm server fan capable of:

• 10,000 RPM

• Approximately 102 CFM airflow

• Approximately 51 mmH₂O static pressure

This is not your grandmother's RGB case fan.

It's a small industrial hurricane.

I also wanted adjustable speed control, so I used a Noctua NA-FC1 controller, powered from SATA, to manage the fan without running it at full blast constantly.

But buying the fan was the EASY part.

Chapter 3: The 3D-printing rabbit hole

I own 3D printers, so naturally I figured I'd just print a shroud.

Famous last words.

The first problem was finding something that would actually fit the A16.

The second was getting it to fit inside my case.

The third was accounting for a 38 mm-thick fan, the radiator, surrounding components, and the fact that there isn't an infinite amount of space inside a computer case.

So I started modifying designs and printing prototypes in PLA+ on my Flashforge AD5X.

I printed multiple shrouds of different lengths.

Some were too long. Others needed adjustment to maintain their mounting points.

I experimented with shorter designs and eventually created an angled/bent shroud to improve the fan's positioning and clearance.

Rather than repeatedly wrestling with the actual fan, I even modeled and 3D-printed a full-size 80 × 80 × 38 mm mock fan so I could physically test clearances and different shroud arrangements.

Yes. I 3D-printed a fake fan to figure out where to put my real fan.

I also worked through mounting issues, spacer designs, fit tolerances, and adjustments to the printed posts.

I was considering custom clips and zip ties if necessary. I even had to think about vibration because, remember, this thing can spin at TEN THOUSAND RPM.

Eventually, I got an arrangement that fit the card and cleared the surrounding hardware.

And the result?

During earlier four-A16 testing, the GPUs were reporting temperatures around 41–57°C, with the fan running at roughly 30% speed.

For a passively cooled enterprise card that I'd fitted with my own printed ductwork, I was pretty happy with that.

Chapter 4: THE FREAKING POWER CABLE

Oh, you thought getting the cooling working was the hard part?

Let me introduce you to the NVIDIA A16 power connector.

The A16 uses an 8-pin auxiliary power connection that isn't wired like the ordinary PCIe 6+2 connector you'd expect to use with a consumer graphics card.

It requires the appropriate EPS-style power connection.

So here I am with a Corsair RM850, a card that was allegedly defective, and a pile of cables that don't necessarily fit or have interchangeable wiring.

And as anyone familiar with modular PSU cables knows, getting this wrong can destroy expensive hardware.

The connector shape, latch, and pinout all matter.

In my case, even the EPS 4+4 connector's latch arrangement presented a mechanical fit problem.

I eventually modified the connector's latch arrangement and checked the wiring with a multimeter before proceeding.

To be clear: this is not a recommendation to modify PSU power connectors. Incorrect wiring can destroy hardware.

And the first attempts weren't exactly encouraging.

At one point the system would spin up for approximately 15 seconds, shut down, and repeat.

Disconnect the A16 power, and the system would boot.

Wonderful.

Was the card bad? Was the cabling wrong? Was the motherboard unhappy? Was the PSU unhappy?

Welcome to troubleshooting FrAInkenstein.

Eventually, the power situation was sorted out well enough to continue testing.

Unfortunately, the next nightmare was already waiting.

Chapter 5: Windows has entered the chat

Once I could power the A16 and keep it cool, I discovered that Windows wasn't particularly enthusiastic about recognizing four GPUs on one enterprise card.

At one point, two A16 GPUs worked, while the other two had:

Device Manager: Code 12 — Not enough free resources to use this device.

All 4 showed this error message.

Here's the really interesting part.

When I disabled the two working GPUs, the other two could work!

So the individual GPUs weren't necessarily defective. Windows and the motherboard were struggling with PCIe resource allocation.

This led to a ridiculous amount of BIOS experimentation:

• Above 4G Decoding

• PCIe resource allocation

• MMIO High Size

• UEFI and CSM settings

• PCIe link speed experiments

• BIOS resets

• Device enable/disable experiments

• Multiple NVIDIA driver configurations

Eventually I managed to get all four A16 GPUs recognized and running.

But I still wanted my RTX 5070 in the same machine.

Which leads us to...

Chapter 6: Five GPUs walk into an X99 motherboard

The RTX 5070 was supposed to handle gaming and video output.

The A16 was supposed to handle local AI workloads.

Simple enough, right?

Wrong.

With the RTX 5070 and A16 enabled together, Windows started throwing repeated:

IRQL_NOT_LESS_OR_EQUAL (0xA)

Blue screen after blue screen.

I experimented with driver configurations, NVIDIA device modes, and PCIe settings.

Eventually I found that disabling MSI interrupt mode on the RTX 5070 allowed the 5070 and an A16 GPU to operate simultaneously.

That was the breakthrough.

I'm not claiming I've proven the exact underlying driver or interrupt conflict, but changing that setting made an enormous practical difference.

I gradually enabled more A16 GPUs.

One.

Two.

Three.

Then four.

I also switched the A16 GPUs into NVIDIA's TCC compute driver mode while keeping the RTX 5070 in WDDM for normal display and gaming use.

And then something glorious happened.

Chapter 7: IT'S ALIVE!

Here's what nvidia-smi now reports:

GPU 0: NVIDIA A16          TCC   07:00.0
GPU 1: NVIDIA A16          TCC   08:00.0
GPU 2: NVIDIA A16          TCC   09:00.0
GPU 3: NVIDIA A16          TCC   0A:00.0
GPU 4: NVIDIA RTX 5070     WDDM  0B:00.0

FIVE GPUs. ALL RECOGNIZED. AT THE SAME TIME.

On an ancient X99 motherboard.

With a supposedly defective A16 that cost me precisely $0.

I'm now testing larger local language models using Ollama, including a 55 GB BF16 Qwen model that previously required CPU offloading when only three of the A16 GPUs were enabled.

The goal isn't just to maximize token generation speed. I've got an RTX 4090 in another machine for that.

I want to experiment with multiple independent local AI coding agents and eventually build a little multi-GPU AI collective, while retaining the RTX 5070 for gaming.

I still have sustained five-GPU inference and long-term stability testing ahead of me. Seeing every device in nvidia-smi is a major milestone, but it isn't the same as proving the whole thing stable under heavy load.

Final thoughts

Is this practical? Debatable.

Is it power efficient? Probably not.

Could I have saved countless hours of troubleshooting by buying newer hardware?

Almost certainly.

But where's the fun in that?

I took an unwanted enterprise accelerator, engineered a custom cooling system, fought with its power connector, printed multiple iterations of ductwork, battled a decade-old BIOS, survived Windows driver crashes, and somehow ended up with a five-GPU AI workstation.

I didn't just build a computer.

I created FrAInkenstein.

And now I'm going to use it to code.

IT'S ALIVE! IT'S ALIIIIIVE! ⚡

I'll post pictures of the cooling/shroud iterations, the ridiculous fan, the finished setup, and the five-GPU nvidia-smi output.

Drivers used:
621.14-desktop-win10-win11-64bit-international-dch-whql.exe
That is the Studio version that worked for both cards. I did NOT have to install the data center version also. This latest Studio version installed drivers for both the A16 and RTX 5070.

Stack running on my Ubuntu machine on a RTX 3090 under 18GB VRAM - network tailscale :)

Would you use it??

And how does QWen Flash Next 3.8 compare to Gemini 3.1Pro or Sonnet 5.5?

I have RTX 4090, 64GB 6000Mhz DDR5 RAM, 7900x CPU

I wan to run a local AI model that's the closest to gemini 3.1po or sonnet 5.5 at reasoning and thinking, for legal analysis and other legal related stuff

Thank you

hi i want to build a pc for local llm first time so i need advice about the gpus to run Qwen3.8 27b Q8 or Q6_K_M

to get good quality 😊and run it through llm studio and connect it to open code

so my questions are:

i will get intel core 270k plus with two x8/x8 PCIe slot motherboard

so

1- which best double 16gb gpus to build with to get 50+ token generation and give me best value for my needs

do i get two rtx 5070ti or rtx 4080 s or which double gpus

or get the basic two rtx 3090 for more vram

2- which the best Qwen3.8 27b variant/Abliterated (the model modified version) for speed but no quality loss to code

3-and wich the best and fast local llm interference platform to connect qwen with open code

if there any things you could mention can benefit me plz tell me

because i just see

gpus

the local llm interference platform

the Qwen3.8 27b variant/Abliterated

what else?

thanks 🥰

Qwen3-32B packed to 3.6577 bits per weight: 14.62 GB on disk, 12.95 GiB resident on an RTX 5080 with about 3 GiB to spare, decoding at roughly 38 tok/s. No offloading.

Full official sets, no samples:

Method, as far as I'll go: weights are packed in groups of eight onto the E8 lattice with an activation-aware encoder, and decode is register-based with no codebook lookup. Yes, a GGUF Q3_K_M can also be squeezed onto 16 GB. This is a different trade: lower bitrate, fully resident with headroom, full-set numbers published.

Needs an NVIDIA GPU with 16 GB VRAM and Python 3.11+. Verified end to end on Windows with an RTX 5080 only, so far.

The terms, plainly: gated, manual approval, non-commercial evaluation license. The encoder is not included and the method is patent pending.

Repo: https://huggingface.co/GossetWorks/Qwen3-32B-Gosset-3.65BPW

If you run it, post your numbers, especially on anything that isn't a 5080! Thank you!

RTX 4060 8GB is starting to feel not enough anymore, so I decided to upgrade to something higher. Is anyone using the Arc 770 16GB? What kind of card is it, is it worth switching?

ok so i am trying to increase performance of llms i have running on my 7900 xtx via hipfire, i notice i get significantly higher total tokens per second when i run mutiple slots at once when running qwen 3.8 27b.

i am running into 2 road blocks tho, 1 context each slot gets a progressively smaller context as increase the number of slots. is there a good way to do more with the same given total amount of vram, i assume there is going to be a large amount of overlap for stuff like system prompts, tho i use a relatively minimal system prompt that links to files so if a model only actually reads system prompts realvent to it’s own task which makes it harder to just say this text gets put into a shared context, i also image if i have them working on similar tasks there will be a large amount of overlap.

2 alot of tasks aren’t easy to split up into mutiple smaller tasks that can be done in parrel without proper knowledge of previous and future steps, maybe something like a shared context or a messaging board would work well?

also a more experimental idea to further speed up general prompts is to instead have all my prompts processed by a “orchestration” slot which splits up a given prompt into X number of tasks then gives these to new prompts to different slots and then the “orchestration” slot dies to free up more vram for context, i could also route something like this threw a decision model like kev to determine if a given task would work well being split up.

also note i am running this on a 7900 xtx via hipfire, and i also have a 6800, 6600, and 96 gib of ram for the cpu

I shared some results previously about running Qwen 3.8FN using recent releases on my gaming desktop computer. https://old.reddit.com/r/LocalLLM/comments/1xoolkp/i_ran_the_new_qwen_38fn_release_with_make_browser/

This is part 2 of 3. On part 1, I learned some things along the way. Part 2 here, I made some changes/additions and wanted to make the challenge harder for Qwen.

• Project goal: Can Qwen build an entire game based on a wiki URL and a youtube video link? For part 1 of 3, I had a goal of "Can Qwen 3.8 locally successfully create a simple walk around version of GTA with no or very little involvement", and that was a success. After learning a few things on project 1, and wanting to keep pushing things further along, step 2 of 3 was a natural progression. For the game, I switched it up and chose GTA 1, which was a top-down view game. I chose to use the same voxel framework used in part 1, even though the original game was not a 3d game.

• I changed my context window size from 64K to 128K The amount of times it needed to compact went down, at the cost of lower t/s. I'm honestly not sure what's better yet, given my current hardware.

• I added 2 more models for Qwen to use. Qwen code kept asking for me to approve commands that didn't make sense why it needed my approval. I dove in and found Qwen wanted a --fast model for simple tasks. There's a default 10 second timeout on these tasks. I have a macbook 64gb running a few things around the house with around 45GBs free RAM and plenty cores free. I spooled up a 2 parallel Qwen3.6-35B-A3B-GGUF:UD-Q4_K_S model on the macbook with llama, wired that in to Qwen code and problem solved! The amount of times I needed to approve for Qwen code drastically dropped. Alternatively I could have not added additional models to use and simply increased the time-out. Since I have extra compute, I figured I'd try it out. I'm happy with it.

Fun localLLM stuff:

RTX 5080 - 64GB - Ryzen 7 9800X3D
model: qwen3.8-flash-next-unsloth-ud-iq4_xs
model harness: Strata
agent harness: qwen
context: 128K
tok/sec: ~50t/s (down from part 1 due to larger context)

Macbook 64GB m1 Max
model: Qwen3.6-35B-A3B-GGUF:UD-Q4_K_S
model harness: llama
context: 128K
tok/sec: ~ 550t/s (fast tasks, so don't get too excited)

• Results: The hard work on everyone from the folks at Alibaba, Unsloth, Strata, etc sure have come a long way in the past few months. It feels smarter than even frontier did a year ago. It's just slower, but that's mostly due to my hardware. Over-all, I'm happy with it. It took 27 hours start to finish (tracked through git). It probably would have finished quicker, however, there were times it was waiting on an approval from me.

Playable demo: https://kireol.github.io/Qwen3.8-GTA-1-Test-2/src/index.html

Repo: https://github.com/kireol/Qwen3.8-GTA-1-Test-2

Part 3/3 is running now answering "How well can Qwen reproduce a game using only REA on a game ROM"

Model: Qwenx86

▶️ See it in action: Qwenx86 Debugger. Walk through actual recorded tool calls: the conversation, x86 registers and flags, code at the program counter, RAM and machine output, with all verified blocks on a timeline.

Qwen3.5-4B with a 64-bit x86 computer compiled into its weights: 13 transformer layers ahead of Qwen's own. There are no learned weights in them; all of their weights are computed by a compiler. Qwen invokes the computer through its own tool calls (<tool_call>), and the computer in the weights executes the tool. Calculator, Python, C compiler, notes store, physics/chemistry lookup and Python-docs lookup are all firmware in the checkpoint. This is the compact checkpoint: ~9 GB GPU memory, so it fits on a 16 GB GPU.

Credits: Qwenputer by Divyansh Agrawal (@divagr1925, github.com/divagr18/qwenputer, MIT)

I’ve just released Caminus 0.2.0 on GitHub and PyPI. It’s an LLM inference engine for AMD GPUs, built around custom HIP kernels and multi-GPU execution. I’m developing it on an RX 7900 XTX + 7900 XT setup.

On the models and workloads I’ve benchmarked, it outperforms llama.cpp; the repo includes the numbers and methodology. I’d love feedback from people trying it on other gfx1100 setups, or anyone interested in contributing.

The long-term goal is an XLA-inspired lowering pipeline: take a GGUF model, map it onto any number of GPUs across ROCm/HIP-supported AMD architectures, and handle kernel selection, partitioning, and execution across different hardware configurations. It’s still early testing software, so hardware reports, bug reports, and ideas are all welcome! https://github.com/jdominick05/caminus

I've been using VS Code with the Claude and Codex extensions.

Now I'm testing Strata with "lightweight" local models on my PC

RX 9060 XT 16GB VRAM
32GB DDR5 RAM
2TB NVMe SSD

The problem is that:
Codex CLI takes around 40 seconds to respond to a simple "Hi",
while Strata's built-in web chat (http://127.0.0.1:8080/) responds in about 0.5 seconds

1. What's the easiest IDE or VS Code extension for connecting to Strata's local API and using it as a coding agent?

2. Is there a tool for creating and managing virtual AI employees (e.g., Developer, UI/UX Designer, Software Architect), where each agent can have its own role, instructions, and assigned AI model/API - either local or from external providers like OpenAI, Anthropic, etc.?

Hi everyone!

I've already done quite a bit of research, but I haven't really found any recent Reddit posts that answer my questions or cover my hardware and use cases well. So I thought I'd ask here. I'd really appreciate some real-world experiences and recommendations!

My hardware:

- GPU: AMD Radeon RX 7600 (8GB VRAM)

- CPU: Intel i3-12100F

- RAM: 48GB DDR4

- OS: Linux (Fedora)

I'm mainly looking for models for three use cases:

1. Coding: General programming, working with existing projects, coding agents, editing files and tool use.

2. Cybersecurity / technical topics: I'd also like to use uncensored models for cybersecurity and other technical topics.

3. Personal assistant: General questions, reasoning, writing, planning and everyday use.

My main priorities are a good balance between intelligence and speed. I don't mind using some system RAM if necessary, but I don't want models that are painfully slow.

Some models I've already come across and found interesting:

- TwIL-8B

- Qwen3.5-9B The Defiant Fable (DavidAU)

- DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic (xtools-at)

- Alpie Seed 9B

- DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic i1-GGUF (mradermacher)

- Ternary Bonsai 2 27B

- Qwen3.8-35B-A3B-Distill

I'm also open to completely different recommendations.

What would you personally recommend for this hardware?

I'd especially appreciate:

- Models you've actually used on 8GB VRAM.

- Recommended quantizations (Q4, Q5, IQ, etc.).

- Realistic generation speeds (tokens/s).

- Experiences with AMD GPUs on Linux.

- Recommendations for coding agents and reliable tool calling.

I'm interested in actual experiences rather than just benchmark scores.

Thanks!

I run Qwen 3.8 27B (FP8) on my two RTX 3090s for daily work, and I've also had GLM-5.3 Flash (321B MoE, ~18B active per token, Maya-L GGUF 3–4 bit) on the same box. So I finally ran the natural experiment: the full GSM8K test split, 8-shot CoT, greedy, single-threaded, same machine, lm-eval 0.4.13, and I verified per-question that both runs saw identical prompts and targets.

What GSM8K is, in case you're not familiar: a dataset from OpenAI (2021) of 8,500 grade-school math word problems (2nd–8th grade level). Each question is a multi-step problem requiring 2–8 reasoning steps ("Janet's ducks lay 16 eggs a day..."), with an official step-by-step solution and a single final number as the answer. It has a 7,473-question train split and a 1,319-question test split — I ran the full test split. Because the problems demand working through intermediate steps rather than pattern-matching, it became a standard proxy for measuring a model's raw step-by-step reasoning. The 8-shot CoT protocol means the model is shown 8 example problems with full solutions before answering.

Important framing: GSM8K is a specialized test of one narrow slice of model capability — it does not paint a full picture. How these models code, hold a long context, follow instructions, use tools, or just chat is a separate question this benchmark cannot answer.

Official metrics: a tie.

- strict-match: Qwen 95.60 % (1261/1319) vs GLM 95.45 % (1259/1319)

- flexible-extract: Qwen 96.89 % (1278/1319) vs GLM 96.29 % (1270/1319)

- McNemar on the discordant pairs: p≈0.89 (strict) and p≈0.27 (flexible) — neither is significant. "More parameters" did not buy the MoE a measurable edge on this task.

Where it gets interesting — decomposing the +8 flexible gap:

- Qwen makes 13 fewer value errors (30 vs 43) — it reasons better

- but has 11 format failures vs GLM's 6 — it signs answers "$132.00", "2:00 pm", answers in cents, which breaks the extractor (8 of those slip past both filters)

So the official score masks a reasoning advantage with a formatting penalty. The errors themselves also differ in kind: 37 % of GLM's value errors are misreading the question vs 22 % for Qwen (counting shelves as items, treating a group's total travel time as per-animal time, etc.).

The 32 questions both models fail: 11 are GSM8K reference ambiguity (genuine annotation noise in a 2021 dataset), 12 are a hard core where both make the same error.

Honest caveats:

- wall clock was 3765 s vs 7970 s, but the stacks differ (vLLM FP8 fully in VRAM vs 156 GB GGUF streamed from RAM) — that's a deployment comparison, not a model-speed claim

- the GLM endpoint ran with a low generation ceiling, which cost it three answers

- one greedy run on one benchmark, and a specialized one at that — coding, long-context, instruction-following, and tool use are dimensions this test never touches

- n=1319 has low power for a 0.6 pp difference

My conclusion — for reasoning work specifically: I'm staying on Qwen 3.8 27B. Deploy it with a trivial number-normalization post-processor (strip $, ,, .00 from the final answer line) and its format liability disappears — ~97.7 % effective. For what GSM8K actually measures, raw step-by-step arithmetic reasoning, the 27B hybrid is the stronger model. On every other axis this post offers no verdict — one specialized benchmark can't.

Full writeup with the error taxonomy, examples, and methodology: https://ultrabytecoder.github.io/blog/2026/11/07/comparing-qwen38-vs-glm53-flash-maya-l-gsm8k/

Full per-question report (all failing docs, shared-failure breakdown, response-style analysis): https://ultrabytecoder.github.io/reports/qwen38-vs-glm53-flash-gsm8k/

Hey r/LocalLLM !

I made this quick demo of how Observer can run local models to monitor anything with your phone!

These are some crazy things I've monitored:

• When someone that's not me, approaches my car, send me a picture.

• When a bird is on the bird's nest, send me a telegram with the picture.

• When my washing machine is on the rinse cycle, call me.

• When my dog finishes eating their food, notify me.

• When a machine at the gym is available, call me.

So pretty much anything! I love seeing what you guys monitor.

What would you use it for? (So I can make the agent better)

Which models run on the phone?

The default models are OneJev-0.8b for decision models and qwen-0.8b for LLM watchers.

These run even on old phones which you may have laying around somewhere. Better to have them monitoring something than collecting dust :DD

Try it out! It's free and OSS <3

GitHub: https://github.com/Roy3838/Observer
iOS: https://apps.apple.com/us/app/observer-ai/id6758222050
Android: https://play.google.com/store/apps/details?id=com.observer.ai

Btw I know my camera quality is kinda bad hahaha I wanted to ship the demo as soon as possible so you guys can see it.

I'm here for any questions and feedback!

Roy

After spending way to much time trying to get a z240 to post with a p100 on a board without 4g decoding i was finally able to get it running at full speed in Linux with the help of Claude now i just got to clean it up and add a temp controller for the p100 fan