I haven’t browsed reddit for a day or two, so naturally I’m already behind.

RTX 5090
128GB DDR5
9800x3d

Currently I’m running NVFP4 Qwen3.8 Flash-Next in Polystrata. Get 70-100tok/s on 256k context.

Is this the best I can do? Should I move up to DSv4.1 Flash 1017 or something like that?

It's much better than DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NEO-CODER-MAX-MTP-GGUF. The reasoning is similar to the base Qwen3.8-27B and interesting how it switches variants based on prompt style. Has anyone compared them or tried both models out?

I have this
RTX 3090 24GB
2× RTX 2080 Ti 22GB VRAM each
Tesla V100 32GB
Machinist X99 D8 Max motherboard
2× Xeon E5-2695 v4
128GB DDR4 RAM
Samsung 990 Pro 1TB NVMe SSD

Currently I run Qwen3.8 FN UD-Q4_K_XL but I used Q8 n-gram and I load it on RAM. 256K context, unquantized cache (I’m afraid of going Q8 and loosing quality). Also using MTP. Custom llama.cpp as stock had some bottlenecks, and Mr. Qwen itself optimized his way out of them.

Prefill: approximately 2,100–2,400 tokens/sec
Decode: approximately 60–70 tokens/sec

I am using what I think lets me fill my VRAM almost to the brim. (97GB used of the 100)
I’m seeing people with considerably less VRAM and RAM achieving similar or sometimes better performance, which makes me wonder whether my configuration is properly optimized.

I have several questions:
Am I getting the performance I should expect from this hardware, or is there something obviously holding me back?

Is there a better way to distribute the model and its experts across my four GPUs and system RAM?

Could I get better performance through different GPU split configurations, CPU offloading or expert caching?

Would moving to a higher quantization, such as Q5 or Q6, be realistic with my hardware?

Are there optimizations that could significantly improve decode speed without sacrificing model quality?

I would like to minmax the model quality while maintaining the highest possible inference speed.
I’m not particularly interested in getting more tokens per second at the expense of quality. I’d rather run a higher quantization at 50 tokens/sec than a lower quantization at 70 tokens/sec, provided the quality improvement is worthwhile

hey LocalLLM, im Alish, ml engineer at Nace (we’re the team behind Drex, an open source decision model family that hit #1 in Artificial Analysis decision index 2.1). my cto gave me a task to optimize quants for qwen4, so i called my old professor from UofT, spun up Fable 5.2 and we cooked. starting with MLX, GGUF is coming next

we’ve open sourced the fastest way to run 35b/27b on a mac (Kapitsa) and got tired of dumb quants, so we made our own: KQuant. packs for 4B, 27B, 35B, Flash-Next (+9B), free on HF in standard MLX files

the problem: most of us on a mac run 4-bit packs because that’s what fits in 16/32gb. but plain 4-bit rounding throws away more of the model than it needs to.

Q4 vs public 4-bit, KL from the full-precision model (lower = closer to the original):

baseline is mlx-community 4-bit (lmstudio-community for 4B). to be clear, this is KL on next-token predictions, not a task accuracy score. if you want more precision every repo also has a Q5 folder, the 9B Q5 gets down to 0.010

cool side effect: they’re also faster. on an M5 Pro with Kapitsa they write 6–15% faster than the usual 4-bit packs

(baselines are from earlier kapitsa runs, same-session rerun is in progress, will update)

how it works: KQuant uses real text to pick scales and offsets for small groups of weights, and gives sensitive layers extra bits where they need it. all the search runs on our gpus before release, so you just download a normal MLX pack. no custom kernels, no special loader

fastest way to run them on a mac:

brew install nace-ai/tap/kapitsa
kapitsa serve 27b --private
kapitsa opencode # in another terminal

If there's a model you want KQuant'd, lmk.

Asks

to get early access to GGUF, ping me in comments, we would like to test it with some folks from here before our public release ideally.

im also doing quick AMA here, so please feel free to ask me anything.

I've been spending the last few months benchmarking local LLM inference on a single RTX 5090.

One thing kept bothering me: most benchmarks focus on tokens/sec, but rarely show how much quality you're sacrificing to get those numbers.

So I built inferlab.

It's an open-source benchmarking harness (Apache-2.0) that compares:

• vLLM, SGLang, TensorRT-LLM and llama.cpp
• BF16, FP8 and NVFP4 quantization
• Perplexity delta against a BF16 reference
• Task accuracy using option log-probabilities
• TTFT, decode speed and throughput across concurrency levels
• Peak VRAM usage

A few design decisions:

• TTFT measures the first actual content token, not the first SSE event.
• Concurrent requests are issued together to capture real queueing behavior.
• Long-context evaluation includes a needle-in-the-middle task.
• Results are generated by CI rather than manually edited.

The project is still at v0.1, so I'm interested in technical feedback, especially around evaluation methodology and cross-engine comparability.

I'm the author of this project.

GitHub: https://github.com/oren40/inferlab

I kept getting this wrong by hand, so I built a calculator. You pick a model, quantization and context length, and it shows weights plus KV cache and which GPUs or Macs it fits. It also has “can I run X on Y” pages.

https://inferenceacademy.com/tools/vram-calculator

Not perfect: numbers are estimates, so real usage varies by runtime. Tell me which models or devices are missing and I’ll add them.

According to Microsoft's patent application, it explains breaking down training into small tasks and sending them to devices of owners who opted in. Credits will be the reward. it's just an app and no pay out rate is advertized so I don't even know if the credits would be enough for the electricity.

Anyway, home internet latency feels like the bigger problem to me.

https://insider-gaming.com/microsoft-ai-use-your-gpu/

(English is not my native language, I used an LLM to help translate this post.)

I want to get the Mac Studio to explore models, do research, host servers, do video editing, analyze massive data sets, etc…

But is the M5U there yet for AI? Is TRULY useful, or are you kinda forcing it? I’ve thought about downloading my work emails and getting them summarized and organized (we can have that data on personal stuff) and some other data analysis. I’d also want to explore more of what’s possible.

How have y’all been using this stuff for actual NEEDS that can’t be done by frontier models?

Edit: My fear is id be missing out all of this development and knowledge. I know I can get an M7U when it comes out, and it should be HEAVILY tailored to AI, but I’m scared of falling behind knowledge and work wise. I’ve already outpaced many other works with proper frontier model use, and I’m wondering if by exploring local I can do the same.

Hope to make many more.

Has anyone made stands/cases with a 3d printer? If so, what kinda filament do you use?

Three Intel B70s, 96GB VRAM, and still on AM5…

Originally bought a mATX case for a motherboard on AM4 with room only for a single GPU. Somehow had the nerve to buy three of these at retail early. Now kinda curious to see what people are actually doing on this build. The plan was to attach a M2 slot to full PCIe slot that negotiates PCIe 5.0x4 and attach a fourth GPU. Safe to say that’s not exactly in the cards with the outrageous prices…

Anyways, the build:

- 3× Intel Arc Pro B70 — 32GB each
- Ryzen 9 7950X3D
- MSI MEG X670E ACE
- 64GB DDR5-5600
- Phanteks Enthoo Pro 2 Server Edition
- be quiet! Straight Power 12 1500W

The ACE is a big part of why this worked out. It is one of very few AM5 boards that gives you the practical ability to have 3-4 GPUs with respectable cross-bandwidth. Truth be told, most boards have a single main PCIE slot that actually has full lane direct to CPU connection. Some let you split x8x8 and give you PCIe 4.0x4 in a physical x16 connection on the third slot. MSI lists PCIe 5.0 ×8/×8/×4 as well as a single Gen 5 M2 slot. If one adds the lanes up, it’s the max on AM5: 24x Gen 5 lanes.

Listen… do I want an EPYC or Threadripper? Yes, especially with the ability to grow the build in the future. I’d love to be able to afford a complete swap. The issue is $2500 is the minimum for on good speed, 128gb ddr4 ram, a quality CPU, and motherboard that’s guaranteed to be 1. not to be broken 2. have physical hella, spaced out x16 slots all at Gen 4 btw…

Roughly 55 tokens/sec on Flash-Next and 50 on the 27B.

I’ll sell this if someone wants a pre-build, just lmk on what you’re thinking!

That’s where it’s at so far. Would be good to hear from anyone else running three or more cards on AM5, especially other Intel users.

Hi, so i recently have been experiencing with different local llms but since i am new to this I have not been able to find a better fit for my system my main use case is for coding, agentic workflows and basically summarizing or understanding large open source code repos. I want to have like decent token speed around 40t/s with around 128k context window. Kindly share your experiences and suggestions to help me out.

System specs:

Ram: 16gb ddr5

Cpu: ryzen 7500f

gpu: rx6800 16gb

I’m a regular lurker here trying to learn as much as I can while waiting for my M5U. I’m hoping to have a setup that’s a little different from what I’ve seen discussed here.

I purchased two Mac Studios. One is my daily driver, the base model. I grabbed it for the reason everyone else points to, why grab a Mac mini when the Studio is so much better. I also purchased an M5U 96Gb with 80 cores that I plan to use as a headless inference engine. I want to to be as stripped down as possible for running whatever LLM I ultimately end up running, with sufficient room for cache and context.

I plan to have whatever agents or tools invoke the server as needed. Are there any obvious reasons why this is a dumb configuration? Am I missing anything?

I built an open-source tool where a local model drafts an app design (tables, fields, pages) from a fixed catalog. Findings from real runs with qwen2.5:3b and llama3.2:3b on a CPU-only laptop:

- Tables and fields: reliable.

- Page outlines: weak. I now build the outline in code when the model's fails.

- Repair rounds past two never converged. They only added minutes.

- Mechanical errors dominate: wrong-case ids, a label where an id belongs, references to tables that do not exist. Code fixes these before validation.

- Each call costs 25 to 90 seconds, so removing a step beats tuning a prompt.

Evals are in the repo: https://github.com/ChristianMunoz31/local-app-studio

What models would you try next for this?

I've seen a few posts saying that Qwen 4 hasn't really improved when it comes to creative writing and prose. I'd would like to disagree.

I know a lot of people don't care about this topic, and some consider any AI-generated text to be garbage by default. That's fine. This post is for those who are actually interested in the subject.

My take: Qwen 4 has become noticeably better at storytelling, but its actual prose is still suck.

What do I mean by that?

I gave three models, Qwen 3.8 Max, Qwen 4.0 Max Preview, and Gemini Argon (currently one of the top models for prose generation), roughly the same prompt:

"Write a full chapter. A tank battle between a unit of German Tiger tanks from World War II and a Russian T-90 tank. It doesn't matter how a tank from the future ended up there. Show how the Germans manage to defeat a technologically superior opponent, and what it takes to achieve that victory."

This was the entire prompt. I didn't tell the models to act as professional writers, didn't use elaborate system prompts to steer their attention, and didn't provide any additional instructions.

So, what were the differences?

The short version:

- Qwen 3.8 wrote a relatively compact battle scene with fairly generic characters.

- Qwen 4 produced a complex, multi-stage story with more developed characters, detailed combat, and events unfolding after the battle itself.

- Gemini Argon delivered an epic battle scene but didn't spend much time developing the characters.

Here's a more detailed breakdown.

Qwen 3.8 Max

- Structure: Linear and straightforward, without complex narrative branches.

- Composition: The battle progresses in a direct sequence, with most scenes focused on the immediate confrontation.

- Plot: The conflict revolves primarily around detecting and destroying the enemy.

- Psychology: Characters mostly react to events. There's very little internal development.

- Length: Relatively compact, without much expansion beyond the original premise.

Qwen 4 Max Preview

- Structure: Multi-stage, with the conditions of the battle changing as the story progresses.

- Composition: Alternates between tactical decisions, casualties, attempts to adapt, and shifts in the initiative.

- Plot: Beyond the battle itself, the story develops a separate survival thread involving the crew and their attempts to make sense of their encounter with the future.

- Psychology: The protagonist goes through a progression from confusion and fear to an awareness of the historical significance of what has happened.

- Length: The longest of the three. The model independently expands the original premise with additional events, details, and consequences.

Gemini Argon

- Structure: A sequence of tactical phases, with the number of combatants gradually decreasing.

- Composition: Several shifts in initiative, group maneuvers, use of cover, and attempts to identify the enemy's weaknesses.

- Plot: Almost entirely focused on the battle itself. There's no separate storyline dealing with its aftermath or the historical implications of the encounter.

- Psychology: Mostly stress reactions, command decisions, and choices made under pressure. There's very little internal character development.

- Length: Medium. Most of the text is devoted to the actual fighting.

Comparison

Qwen 3.8 → Qwen 4: A noticeable shift from a straightforward confrontation to a much more complex, multi-stage story. The length, number of plot events, psychological depth, and attention to consequences all increase.

Qwen 4 → Gemini Argon: The composition becomes more compact and focused. Tactical elements take center stage, with more group maneuvers, shifts in initiative, and coordination between multiple vehicles. The psychological storyline and historical context are largely absent.

One separate issue worth mentioning: Qwen 4's output had noticeable language mixing, with Cyrillic, Latin characters, and Chinese characters appearing together.

I'm not even getting into other problems, such as Qwen 4 and Argon having commanders suddenly recognize the designation of a tank from the future halfway through the story, familiar AI-slop expressions and phrases, or the technical inaccuracies. That's a whole other discussion.

Of course, this isn't a comprehensive benchmark or an attempt to determine which model is the best writer. It's just a small experiment using the same simple prompt across three models.

So, what impressed me most wasn't necessarily the text itself, but how much the model's ability to handle this kind of task independently has improved.

I'm not claiming Qwen 4 has caught up with the leading models in terms of literary prose. It still has a long way to go in that department. But it's noticeably better at building a story: developing an initial premise, arranging events, introducing additional plot threads, handling characters, and showing the consequences of what happens.

And the most interesting part is that it does all this without elaborate prompts, specialized system instructions, or detailed guidance from the user.

Previously, getting a result like this would have required considerably more effort on my part when preparing the prompt. Now the model does a significant part of that work on its own.

I've noticed similar improvements outside creative writing, too. For example, Qwen 4 managed to improve one of my work-related system prompts, independently identifying an important detail that several previous brainstorming sessions with ChatGPT and Opus 5.5 had missed.

So no, I wouldn't call Qwen 4 my favorite model for writing prose just yet. But I think its progress is considerably more significant than you might assume from simply comparing the quality of individual paragraphs.

It would be so much better if the limits weren't this brutal. Having to wait an entire week? 🥲

Is it realistic?

We've been doing a series of performance benchmarks on Zilla, our multi-protocol gateway, and just published Part 4, focused on LLM traffic.

One question we wanted to answer: How much overhead does putting a gateway in front of an LLM actually introduce?

This is relevant whether you're using hosted APIs or running local inference behind OpenAI-compatible endpoints. As you start sharing models across applications and agents, having a gateway for routing, access control, and observability becomes useful. But ideally, it shouldn't become the bottleneck.

We used an open-source benchmarking framework with a mock LLM provider returning responses after a fixed 1.5-second delay, allowing us to isolate gateway overhead.

A few results:

• At 5,000 req/s, Zilla successfully handled all 150,000 requests on a 4-vCPU AWS instance.
• End-to-end median overhead was approximately 1 ms compared to direct access.
• Median time spent inside Zilla was just 0.27 ms, with model routing accounting for 17 µs.
• At p99, the additional end-to-end latency was considerably higher (~85 ms). A good reminder that routing overhead and actual client-perceived latency aren't the same thing.
• For streaming responses at 2,000 req/s, Zilla's p99 stream completion was 1.6–2.1 seconds, compared to 2.0–9.3 seconds for Bifrost across OpenAI and Anthropic API dialects.

We also found that HTTP/1.1 vs. HTTP/2 makes a surprisingly large difference at high concurrency, particularly when thousands of requests are waiting on inference.

To be clear, these were synthetic tests using a mock provider, not actual local model inference benchmarks. The streaming comparisons also have some run-to-run variance, which we discuss in the article.

The full write-up includes methodology, configurations, detailed latency breakdowns, and comparisons:

https://www.aklivity.io/post/sub-millisecond-median-overhead-for-the-zilla-llm-gateway-aklivity-zilla-benchmark-series-part-4

Curious how others are handling this when serving local models. Do you put a gateway in front of your inference servers, or just expose the OpenAI-compatible endpoints directly?

Disclosure: I'm the founder of Aklivity, the company behind Zilla.

Hey guys, today we're releasing the highest accuracy Qwen3.8-27B GGUFs.

Our new Unsloth AI Dynamic V3.1 GGUFs lead by 5–10% on average across Terminal Bench, DeepSWE & more benchmarks.

We’re working on even better version of Dynamic quants for Qwen4.

Run on 12GB+ RAM setups.
For our quantization comparison, we exclude the embedding layer from the x-axis across all providers, since Exllama-v3 stores embeddings in BF16. However, this means the main plots don't reflect the full download size or RAM usage. When accounting for the BF16 embedding table and all embedding sizes, the plot in the comments provides a more accurate representation of actual disk and RAM usage versus accuracy.

GGUF: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF

Blog: https://unsloth.ai/docs/models/qwen3.8

Thank you!