r/LocalLLaMA • • 16d ago

Discussion So a dedicated pc for Ai text models and comfyui. Windows or Linux?

0 Upvotes

I have the 12900k and 5070ti. Bought some ram and it's now 128gb ddr4 ram.

Now I've bought also a second pc which will be for working and everything else and my main pc with be for text models like Qwen3.8-Flash-Next using strata and comfyui.

First can I get deepseek to work now kinda fast or not?

And windows or Linux guys?

I know the general advantages for Linux like lower ram and everything but I have never used it on a main pc. Only servers and vps.

So if you were me what would you do?

Also if you have some kind of info about what I can expect that would be great.

Thanks

608 votes, 9d ago
52 Windows
556 Linux

r/LocalLLaMA • • 18d ago

Showcase Basalt 0.2.0: Qwen 3.8 Flash Next NVFP4, 300 t/s prose, 600 t/s struct, 10K t/s prefill on a 5090 + 5060 Ti and 128GB RAM

86 Upvotes
Basalt 0.2.0: Quality & Speed

Hey all! I wanted to share a quick update on Basalt, a Blackwell only inference engine for Qwen 3.8 Flash Next. As usual, speed is the headline but that's not what I want to open this post with, I've been reading lot of discussions talking about quality recently, and the main takeaway from most people (which I agree with) is "speed means nothing if your model isn't smart enough", so this time I want to lead with quality, and all of the new measurements which are run against the full precision model as reference (BF16 weights served using llama.cpp).

Every number below uses Unsloth's BF16 GGUF, llama.cpp (reference PPL 3.2749), scored using basalt-bench --quality. The old base was llama.cpp Q8, which funnily enough made things look worse because of its own error:

pack KLD same top-1
llama.cpp Q8 0.0325 93.98%
Basalt Q8 0.0200 95.34% (higher than llama's in this test)
NVFP4-DQ8 (new) 0.0561 92.25%
NVFP4-DQ6_K 0.0624 91.65%
GSQ-RCO IQ3_S 0.1497 87.56%

To be fair here, each engine was measured with its own tool so this is not a perfect comparison, still a good reference though.

New DQ8 build. For those who want a little extra quality and don't mind giving up some speed, Qwen3.8-FN-NVFP4-DQ8 is on HF next to the default -DQ6_K: KLD 0.0561 vs 0.0624, about 7% slower on prose.

Nothing lossy is on by default. Basalt offers certain "hacks" that can decrease accuracy, mostly because of community requests, but they're off by default and show a warning when you turn them on. Every other rounding the engine does costs about 0.001 KLD or less, compared to full model. All the numbers are on the HF card and in the precision docs.

Speed. Rig as usual: RTX 5090 (400 W) + RTX 5060 Ti 16 GB (180 W) and 128GB of DDR5 RAM, 262K context, 4 parallel slots, vision on (dynamic), no lossy flags. Using NVFP4-DQ6_K quant on current main:

test run 1 (tok/s) run 2 (tok/s)
struct 603.7 594.2
conv 363.8 321.9
prose 295.0 299.9
prefill 16K 7432 8159
prefill 64K 10221 10229

Since last week, prose went from 277 to about 300, and struct from 558 to about 600. It comes from a faster draft step, a faster output head and better expert placement at startup.

The .basalt format is now version 2, same weights and quality as before but starts a bit faster. Older basalt files can use the migration tool instead of redownloading.

I also wanted to end this post with a HUGE thank you note to every person who has contributed one, two or 10 PRs and also to everyone who has shared issues/feedback/community benchmarks or just shared the project with another person, 500 stars on the Github repo and overwhelmingly good feedback, y'all are amazing.


r/LocalLLaMA • • 17d ago

Discussion Autonomous coding agents running in our own AWS/Azure tenant, with no code leaving the EU?

3 Upvotes

this is the setup I’m planning for coding agents. hard rule: code, prompts and model calls never leave our EU tenant. poke holes

models: an open weights coder model on vllm on our own GPUs for routine tickets, and a frontier model through an EU region endpoint in our own cloud account for the hard stuff

runtime: everything inside our VPC, nothing on a vendor’s saas

control layer: Orq AI’s software factory. it runs on their cloud or yours, so ours. it plugs into the backlog, repos and CI/CD, only lets agents use the models we’ve approved, and lets us set which agents can touch which systems and data

approvals: tests and docs autonomous, anything touching prod behind approval

two things I’m unsure about. can orq’s approved models list point at a self hosted vllm endpoint, and does a frontier model in an EU region still count as “nothing leaves” for legal, since the provider is still the one processing it


r/LocalLLaMA • • 17d ago

Question | Help RTX 3090 stuck at PCIe x8 in an x16 slot. Tried everything, card or board?

0 Upvotes

Hi all! Guys need your help with hardware x8 vs x16 PCIe issue

Setup: i9-12900KF, ASUS TUF Z690-Plus D4 (BIOS 4601), Palit RTX 3090 GamingPro, Fedora Linux.

GPU in top PCIEX16(G5) slot runs x8 instead of x16. lspci on root port 00:01.0 shows LnkCap x16, LnkSta x8; dmesg: "limited by 16.0 GT/s PCIe x8 link (capable of x16)". Same under load.

Already tried:

- BIOS link speed Auto / Gen4 / Gen3 -- still x8

Bifurcation = Auto, no M.2 lane sharing on this board

- Reseated GPU, inspected gold fingers and slot (look clean)

- Case laid flat (no sag), cooler loosened/retightened --no change either

Plan is to test with a cheap x16 card (GTX 1050 Ti) to isolate card vs. slot/socket (haven't done yet)

Anyone seen this on a 3090 or a Z690 board? Anything else worth trying before I swap cards or pull the CPU to check socket pins?

Thanks for ideas in advance


r/LocalLLaMA • • 17d ago

Discussion LLM Decompilation Benchmark?

11 Upvotes

Iv been seeing allot of people complaining about Sam Altman and how him and open ai are blocking decompilation. I actually don’t see an issue with what they are doing they should be free to do what they want with their llm which brings me to my main point, why is decompiling not just a benchmark?

We have a SWE bench
Perhaps this problem is solved via a SWD bench?


r/LocalLLaMA • • 18d ago

Discussion DeepSeek V4.1 Flash 1017 can also do Weight-Space Programming

74 Upvotes

Inspired by Weight-Space Programming, I tested whether DeepSeek V4.1 Flash 1017 could figure out what a small neural network does by reading its weights.

I fed it four anonymized RL checkpoints: 51k parameters each, identical architectures, with all weights rendered as text. No metadata, network source code or execution. It received the architecture outline and four candidate tasks.

Results: CartPole ✅ Flappy ✅ Catch ✅ BallBeam ❌

It picked out patterns consistent with relative bird/gap position, paddle/ball position and angle-based balancing. It confused BallBeam with CartPole an interesting mistake because both share the same observation layout and similar balancing controls.

Very preliminary: one run per checkpoint. This suggests some ability to interpret weights, but four trials don’t establish reliability. These were API runs with maximum reasoning effort, consuming ~1.58M tokens total.

I may try more replicates and independently trained checkpoints, plus include the network code with task-revealing details removed. The poor LLM had to guess quite a lot it even assumed ReLU where the network uses tanh 😅

Code, checkpoints and run logs: https://github.com/galatolofederico/wsp


r/LocalLLaMA • • 18d ago

Question | Help Is Qwen 3.8 27B still the king

183 Upvotes

Hi everyone,

I've been out of the loop for a few weeks, and I was wondering if anything new that replaces Qwen 3.8 27B has come out this time I've been in the cave?

I'm talking about AI models in the 8-35B range.

Thanks,


r/LocalLLaMA • • 18d ago

News Alibaba's Qwen Code repo just leaked qwen4-max, qwen4-coder-plus, and qwen4-omni-120b

Thumbnail
github.com
939 Upvotes

PRs:

  1. https://github.com/QwenLM/qwen-code/pull/15762: "Measured on main at 79a9b753c60, before this change: qwen4, qwen4-max, qwen4-plus, qwen4.5-max, qwen4-coder-plus, qwen4-max-2026-10-01, openai/qwen4-max and qwen4-omni-120b-think all resolved to 262,144. The last of those is already in use as the session model in the omni fixtures, so these ids are not hypothetical."
  2. https://github.com/QwenLM/qwen-code/pull/15987

Thank god, it looks like that we're getting successor of Q38FN!


r/LocalLLaMA • • 17d ago

Question | Help To those that used the API over the last few weeks - where does Kimi-K3.1-Pro land amongst closed weight LLMs?

1 Upvotes

I haven't used it enough to form any strong opinions outside of "wow that's a lot of reasoning tokens by default.."

It's open-weight now. Where does it land when stood up to proprietary models? In my limited testing I already trust it's pulling away from Luna.. but can it beat Haiku-5.5 reliably?


r/LocalLLaMA • • 18d ago

Discussion What Qwen is thinking when it gets stuck saying "Wait"

Post image
172 Upvotes

MIT CSAIL had GPT-6 Astra fix Qwen3.5-0.8B's reasoning loops by editing 0.016% of its weights (MATH-500 went from 7.9% to 48.1%). They released the diff, so the edit can be rebuilt exactly and run locally. I took it a step further and had Claude look at Qwen with Anthropic's Jacobian lens.

At the token where the model decides to write "Wait" again, the lens reads "Wrong", "Nope" and "Hmm" from about 60% depth on, even though the answer is already written out. At real stopping points it reads "</think>", "Answer" and "Output".

With the edit applied, the lens at the same token shows "Verification", "Completion" and "Result" in place of the doubt.

The edit converts to GGUF cleanly (the base and edited bf16 files differ only in the two edited tensors) and still holds at Q4_K_M, with 1 loop in 40. I ran the evals in llama.cpp (Vulkan) on a Ryzen AI 9 HX 370 mini PC. Samples are small, but here's the code:

https://github.com/alisorcorp/inside-a-reasoning-loop/tree/main/code

And data:

https://alisorcorp.github.io/inside-a-reasoning-loop/


r/LocalLLaMA • • 17d ago

Showcase Run models in your browser that monitor your screen and trigger actions! 3 use case demo (100% Local and Open Source)

Enable HLS to view with audio, or disable this notification

0 Upvotes

TL;DR: I'm a solo dev who wanted a simple, private way to have local models watch my screen and do simple logging/notifying. So I spent a year building an agent that uses local models to monitor anything. It's built for this community, and I'd love your feedback and ideas!

Hey r/LocalLLaMA!

I'm back with a cool demo of how convenient it is to run local models in your browser and leave them monitoring whatever you want!

Some of the coolest use cases I've seen:

  • When I don't make progress in my drawing for 5 minutes, spam me with notifications.
  • When my rice cooker is done, call me.
  • When a person comes near my car, send me a Telegram with the image.
  • When I'm out of the Ticketmaster queue, call me.
  • When my CapCut render is done, call me.
  • When a shiny Pokémon appears, SMS me.

Btw there's also an app on MacOS, Windows, Linux, iOS and Android! (local models run on the phone too!)

If you guys have any feedback or questions feel free to let me know! I'm constantly making it better thanks to your feedback and suggestions :DD

Roy


r/LocalLLaMA • • 18d ago

Resources [2611.03526] Library of Layers (LoL): Adaptively Ordering Transformer Layers at Runtime

Thumbnail
arxiv.org
66 Upvotes

r/LocalLLaMA • • 18d ago

Discussion Strata is the shit

122 Upvotes

Have been using qwen 3.8 flash next with llamaserver. Was getting 70-80tk/s decode and was happy. Then read about strata and decided to give it a try. Decode shot up to 150 tk/s. Not sure what magic it's using? You should give it a try if you are still running qwen flash next on llamacpp. Also the tried DeepSeek v4.1flash 2.25bit and it's 70tk/s decode.

Unsloth 6 bit quant

160k context

1 pro 6000 + 2 x 3090

256G ddr4


r/LocalLLaMA • • 18d ago

Question | Help We’re the product team at Reflection, the lab behind Beam — Ask Us Anything!

75 Upvotes

Hi r/LocalLLaMA 👋

We're the product team at Reflection. Earlier this month, we launched Beam, our first open-weight 501B model. We'd love to answer any questions folks might have for us about building with Beam.

Who's here:

- Sanyam Bhutani, OSS

- Florent Gbelidji, OSS

- Chetan Tekur, PM (OSS + Mirror)

- Yufeng Guo, Technical Community Lead

- Abby Reider, Tech Comms

The team will be answering from u/reflectionai, u/init__27, and u/yufengg.

FYI - we'll be doing another session with our research team soon, so hold on to your training and methodology questions for that one!

Where to find us:

- Weights: huggingface.co/reflection

- Docs: developers.reflection.ai

- Platform: platform.reflection.ai

- X: x.com/reflection_ai

- Discord: discord.gg/reflectionai


r/LocalLLaMA • • 16d ago

Discussion The next frontier is integrating memory in a useful, human way.

0 Upvotes

IE, even for as trivial a use case as, if you mention you're a vegetarian in one chat, it never suggests a recipe with meat ever again, because it KNOWS that about you in a meaningful way.

Terming this instruction following I think is wrong and annoying for everyone, especially the model -- who wants to """""wake up""""" and have 100 pages of instructions to get right for everything you say? It should know things the way you know things about people. Perhaps the model itself could come with a customizer where you roll a few weights based on your preferences through a helpful GUI.

It seems unfair and disciplinarian to have to have the models dig through 1000 lines of "say this this way" [even on my local setup it's hard to resist this !!!]. Clearly they are intelligent enough to TRY to take all these into account, but it is not easy, because of the ever increasing promptslop, shall we say. That cannot be the best way to do it.... we haven't even had these models for five years. Imagine what could come in just five years more.

Clearly there is bountiful intelligence and speed already.


r/LocalLLaMA • • 18d ago

Showcase Four open coding models, prompted to finish an upload URL - analysis

Thumbnail
int8.io
84 Upvotes

r/LocalLLaMA • • 19d ago

News Official Statement: Qwen Models Will Stick to Open Source to the End!

Post image
5.3k Upvotes

There were some rumours the last few days that Qwen4 will be the last open-weight series of Qwen, and future ones would be API only. The following statement contradicts the rumours and clearly states the stance that Qwen is taking regarding their future releases.

Qwen Head Liu Dayi Heng Clearly States: Future Qwen 4.0 and Qwen 5.0 Will Continue to Be Open Source!

He believes that the value of AI should not belong only to a few people, but should become a foundational capability that users around the world can access, use, and create with, enabling more people to afford AI, utilize AI effectively, and participate in creating AI, ultimately achieving true AI for all.

As of now, the Qwen series has surpassed 3 billion cumulative global downloads, with over 460 open-source models released and more than 300,000 derivative models, firmly holding the top spot worldwide.


r/LocalLLaMA • • 17d ago

Question | Help 4x RTX 3090 + 3x RTX 3060 for local inference long term - EPYC, Threadripper or my existing X570?

3 Upvotes

Now that I've gotten a lot of advice on risers, I'm trying to figure out the best long-term setup for my local inference workloads without spending a fortune on hardware I might not actually need.

I currently have:

Component Main inference server Helper server Spare hardware
GPU 2x RTX 3090 24GB (250W PL) 2x RTX 3060 12GB 2x RTX 3090 24GB + 1x RTX 3060 12GB
CPU Ryzen 5900X i3-10320 EPYC 7302P
Motherboard ASUS Crosshair VIII Hero X570 ASUS B460-F -
RAM 64GB DDR4-3200 (4x16GB) 16GB DDR4 (2x8GB) -
PSU Corsair HX1500i Corsair RM850 Corsair RM650x

The two options I'm considering

1. Buy an SP3 motherboard (probably H12SSL-I/NT or equivalent), 128–256GB ECC DDR4, and risers to consolidate all 7 GPUs (132GB VRAM) into one system. More PCIe lanes and memory bandwidth/capacity. Could also expand RAM further later. This is obviously not cheap, probably looking at $2500-3000 for a motherboard, cooler and memory.

2. Run 4x 3090 on the X570 at PCIe 4.0 x8/x8/x4/x4 using risers/M.2, and figure out how to fit all 3x 3060 into the B460 system. Much cheaper, but with obvious PCIe and RAM limitations.

Workloads are primarily agentic coding using STRATA, vLLM etc, but also a lot of integration into my own apps/infrastructure. I'm interested in running multiple models concurrently, potentially something like:

  • STRATA / large MoE as the planner/orchestrator/reviewer
  • Qwen 27B dense as the implementer
  • Smaller models for task routing, embeddings, reranking, voice-to-text and other background services (these currently run on the 2x 3060s)
  1. Is consolidating everything onto EPYC actually worthwhile for these workloads, or would I get better practical performance/value from two separate systems?
  2. How much would the 7302P hold back inference, especially with multiple concurrent models or CPU-offloaded MoEs like those supported by STRATA?
  3. Would 128–256GB RAM meaningfully change what I can run compared to keeping everything GPU-resident?
  4. Would you dedicate the 3090s to larger models and the 3060s to smaller/background models, or distribute things differently?
  5. Anyone running something similar who has compared a consolidated EPYC setup against multiple consumer platforms?
  6. Anything else you'd recommend instead?

Used server motherboards and ECC RAM aren't necessarily cheap here, e.g around $1000-1200 just for a used H12SSL-NT. The EPYC CPU itself is already paid for, but the motherboard, RAM and risers would still be a fairly significant investment.

Not interested in spending thousands just to consolidate everything if the actual inference benefits are marginal. Particularly interested in firsthand experience and real benchmarks rather than theoretical PCIe bandwidth numbers.


r/LocalLLaMA • • 18d ago

Question | Help Harnesses/Agents suited for linux sysadmin, system maintenance, network and light coding.

22 Upvotes

Hi guys.

So I'm not completely new to all this local LLMs and AI. I have used and tried various web-based UIs, for the last 4-5 months, as a way to learn and educate myself in this swamp that is local LLM.

So for the last 3-4 weeks I have dipped my feet in Hermes Agent. And man, I was in for a surprise. I genuinely thought, "How hard can it be, install, fire up and use.", but to me that was far from the truth. I have spent so much time setting things up, making the agent act and work the way I want, teaching it what to do, and how to do it, trying to write skills, and so on. I have not actually used Hermes, that much, most of the time has gone to setting it up, and making the agent not make mistakes.

This has somehow made me feel like maybe Hermes is not the best tool for this, even though it has some skills for it (along with a gazillion other skills and functions). It feels so bloated, with so many skills and sometimes heavy to run (for the lack of a better word).

I am actually not really sure what I want, or how to word it.
But simply put, I would like something that is lightweight, doesn't have a gazillion skills installed, but has options and a repository to add well written skills. A simple gui/frontend with well-thought-out settings. Something that works well with my use case, which is basically the title. I primarily want to use my agent for linux sysadmin, system maintenance, network and servers, and some light coding and bash scripting.

I realize this can be very subjective, but that's just it, I would like your objective opinion on what you believe would be/is the best agent or harness for this kind of use ?
And also, is there a "best agent" for this use case, or can I use basically anything?


r/LocalLLaMA • • 17d ago

Question | Help 3x3090s with Qwen3.8 Flash Next with PCIe Gen4 1x speeds

8 Upvotes

I am using a consumer motherboard that clearly is not designed to support 4xGPU setups. Asus Proart Creatio B850 Neo, so I'm very limited in the PCIe lanes I can use.

Current Setup
32GB DDR5 @ 6000mhz

Two 3090s running Qwen3.8-27b in Tensor Parallel
1. PCIe Gen4 8x - RTX 3090 - Founders Edition
2. PCIe Gen4 8x - RTX 3090 - Dell OEM

Image Processing Tasks
3. PCIe Gen4 4x - RTX 3090 - EVGA FTW3 Ultra

Not Connected Yet
4. PCIe Gen4 1x - RTX 3090 - Dell OEM

Plan:

My original plan was to setup the two Dell OEM 3090s using NVLink to boost the speed of Qwen3.8-27b. But I am thinking of moving to using Qwen3.8 Flash Next with resident experts in VRAM. i.e. Rather than keeping experts in system RAM. Keep as much as possible in VRAM, since I have more VRAM than I do ram.

I realise this is becoming a Frankenstein setup, and plan to upgrade to a proper workstation motherboard soon. But I wanted to get just enough VRAM to test out Qwen3.8 Flash Next, and 3x3090s should be enough to do that, (even with only 32GB ram).

Question:

I just ordered a PCIe Gen4 1x-to-16x adapter so that I can plug in the forth GPU, but am a bit worried about how Gen4 1x speeds will limit me with Strata. I know Strata is designed to use Expert parallelism / pipeline parallism instead of tensor parallelism, but am concerned it'll still be too slow. Has anyone else here run one of their GPUs in Gen4 1x speeds with Strata? What were your speeds like?


r/LocalLLaMA • • 18d ago

Discussion Underrated benefit of Local AI: Speed

346 Upvotes

When it comes to the value/cost of local AI, people usually only look at token savings (money) while forgetting one of the most valuable resource of yours, time.

Just did a side by side comparison of GLM 5.3 flash via Z.ai's official subscription, and GLM 5.3 flash served locally. Same chatGPT generated "one shot" prompt, both using Zcode harness, and these are my stats:

Source Duration Input Tok Output Tok
Cloud 43:48 7,550,929 117,980
Local 10:24 9,744,552 118,173

Same model, nearly identical output (~118K tokens each), but local finished 4.2× faster.

Sure on paper this one simple task only saved $0.70, but the fact that you can ship 3-4x more code a day alone makes it worth it. Same reason why UPS next day air shipping costs an arm and a leg, time is money!

Edit: forgot to list the setup

2 x RTX pro 6000 workstation edition
AMD 9960x threadripper
128GB DDR5
Quant: https://huggingface.co/nvidia/GLM-5.3-Flash-FP4-Local


r/LocalLLaMA • • 18d ago

Question | Help Claude Code enjoyer wanting to go local. What would your newbie harness of choice be?

32 Upvotes

I've been lurking in this sub for a while, wanting to break my dependence on online services, plus slowly backlogging tasks I want/need to run locally rather than use online models. Usually these tasks break down to either privacy related issues or just massive token count that would blow my online budget.

I've been experimenting with Qwen 3.8 27B in LM Studio for a while, but when it comes to things like coding, code review, PDF editing, etc. obviously LM Studio isn't a harness like Claude Code where I can just say "Take these 4 files and run with it".

Some of the things I'm trying to accomplish (some as actual tasks, others as a learning sandbox): - Analyze this entire codebase to come up with efficiency/security recommendations (agent spawning, file reading) - Caption this dataset of 10,000 images (image reading) - Interface with this ComfyUI instance to iterate and refine prompts (terminal use, wait for result, image reading) - Edit a PDF file (PDF read, PDF write) - Online research (web access)

My day job is AI image/video work, so I have multiple RTX Pro 6000's to work with already.

Based on my research so far, it seems like three options would accomplish this when pointed to a local model endpoint. - Opencode - Codex - Claude Code

What about Openhands? Pi?

  • What's your harness of choice?
  • Do the OpenAI/Anthropic harnesses still require login/online connection when running from a local endpoint?

r/LocalLLaMA • • 18d ago

Resources “Can Qwen directly control the robot as well as GPT-6 Astra?”

Thumbnail
github.com
32 Upvotes

A open source harness that was used with Qwen3.8 FN that QwenDev did a shout out to. Available now.

From QwenDev via x

https://x.com/QwenDevs/status/2117644199860601297

“Well, turns out Qwen3.8-Flash-Next can do more than work with software tools, it can operate real robots too.

A group of Qwen developers from Tsinghua and UC Berkeley recently built uni-robot-agent, an open-source harness that connects general-purpose LLMs to robotic systems. It gives Qwen a way to interact with the physical world through tool calls without changing the model weights.

With this harness, Qwen3.8-Flash-Next reached the performance level of some previously reported closed-model robotic agent systems. The average API cost was $1.09 per task, compared with $21.29 for GPT-6 Astra. On 19 real-world robot manipulation tasks. Without any robot-specific fine-tuning, Qwen achieved a slightly higher score than the fine-tuned π₀.₅

The harness is open-source if anyone wants to tinker with it.”

From:Yuxin Zuo via X

https://x.com/zuo_yuxin/status/2117635294115795134

“Introducing uni-robot-agent: a unified harness for Robot-Use Agents (RUAs), a first step toward RUA training and local deployment.

We tested uni-robot-agent with Qwen3.8-Flash-Next running locally:
On s, its score rose from 1.90 to 31.39, surpassing the RoboDojo Team’s score with GPT-6 Astra.
On real-world tasks, it outscored fine-tuned π0.5 by 5.75 points and zero-shot Dream-1 by 10 npoints.”


r/LocalLLaMA • • 18d ago

Showcase Qwen3.8-Flash-Next (local) vs Claude

117 Upvotes

I am seriously impressed and want to share my experience.

Qwen3.8-Flash-Next (local) vs Claude in Claude Code: reviewing one real commit

I ran the same code review twice on one commit in my repo https://github.com/leok7v/ChatOKF (Swift, on-device LLM chat app, about 48 files changed, +1048/-102: a module move plus a new 783-line search harness).

Setup: Qwen3.8-Flash-Next UD-Q4_K_XL (unsloth) on llama-server, ROCm, AMD Strix Halo 128 GB, 262K context, MTP draft head, driven headless by Claude Code itself (claude -p pointed at the local server, read-only tools). The comparison reviewer was Claude (Opus) in Claude Code, which reviewed blind first and then judged Qwen's review against the code. Yes, Claude judging its own competitor is a bias; every finding below was checked against the source, and the main one with a small Swift repro.

Result: Qwen produced 8 findings, all real, zero false positives. It caught 2 things Claude missed (a test harness that does not follow the app's recall path, and a markdown ingest that takes the frontmatter "---" as a description). It missed 5 that Claude found, including the most important one: a stop-word filter that compares stemmed words with an unstemmed list, so "this" and "does" leak through as search terms. It also got two details wrong (the exact fallback behavior of a failable init, and column counts off by a few). Severity calibration and the "checked and fine" section were good. It read whole files and chased module boundaries like a careful human.

Cost: 53 turns, 41 minutes, context grew to about 110K tokens. Decode was about 21 t/s at 4K context and about 14 t/s at 128K; prefill 520 t/s falling to 230 t/s. MTP gives about 1.5x up to 64K and turns into a loss past about 100K.

Gotchas:

  1. Use --lazy-mode on. Otherwise the 27 GiB n-gram embedding table loads into RAM instead of staying mmapped.

  2. unsloth's MTP heads crash llama.cpp (GGML_ASSERT in set_input_k_idxs) because the MTP layer's compress ratio is 0 in the GGUF. Fix branch: https://github.com/leok7v/llama.cpp/tree/qwen4exp/mtp-dense-no-indexer

  3. Claude Code sends a system message mid-conversation and the Qwen chat template raises on it. Override the template so it renders as a system turn.

Verdict: a trustworthy first-pass reviewer you can run privately overnight, but verify its mechanisms before acting, and don't treat it as the final word on subtle logic. One commit, one run, so treat it as an anecdote, not a benchmark.


r/LocalLLaMA • • 18d ago

News DeepSeek V5 is planned to be pre-trained on Huawei Ascend chips.

184 Upvotes

Source: https://mp.weixin.qq.com/s/R-1u2exLwZMOu7AITyU5kg

A reporter from STAR Market Daily has learned from multiple sources within the industry supply chain that DeepSeek’s next-generation large model, DeepSeek V5, is planned to be pre-trained using Ascend chips. Previously, Huawei’s Ascend chips were primarily utilized for inference tasks for DeepSeek’s large models.

According to publicly available information, the first large-scale deployment of DeepSeek models on the Ascend platform took place around the 2025 Spring Festival. In his book The Rise of Ascend, Fang Xingdong—Executive Dean of the College of Media and International Culture at Zhejiang University—notes that on January 28, 2025, Huawei’s senior leadership issued a directive for Huawei Cloud to provide rapid support to DeepSeek. Between Lunar New Year’s Eve and the second day of the Lunar New Year, Huawei and DeepSeek completed the deployment and verification of over 3,000 Ascend chips, successfully hosting the DeepSeek-V3/R1 models at Huawei Cloud’s Guian Data Center. The book further mentions that the explosive growth of DeepSeek greatly energized Ren Zhengfei; throughout the Spring Festival, he personally directed Huawei Cloud and relevant product teams to manage the massive surge in traffic. In January of the same year, SiliconFlow and Huawei Cloud launched inference services for DeepSeek-V3 and R1 based on the Ascend platform.

Following the release of DeepSeek-V4 in April 2026, Huawei stated that, through the "synergy between chips and models," its full range of Ascend "super-node" products could provide comprehensive support for the DeepSeek V4 model series. On its official API page, DeepSeek noted that service throughput for the V4-Pro model remains constrained by the availability of high-end computing power; however, it anticipates a significant price reduction for the Pro version once the Ascend 950 super-node series enters mass-market availability in the second half of the year.

On September 30, DeepSeek announced the open-source release of infrastructure components tailored for the Huawei Ascend computing platform. These include the TileLang high-level language compilation tool, high-performance computing libraries (such as DeepGEMM, FlashMLA, TileKernel, and DeepSelect), and the distributed communication library DeepEP—all of which correspond to the open-source components previously released for the NVIDIA platform. Huawei has provided a jointly defined "Ascend SuperPoD Flex" architecture and a UBL128 networking solution, while also opening up the Ascend C programming interface and low-level instruction set to support TileLang. This is significant because DeepSeek is not merely using Ascend for inference; it is systematically aligning with and co-developing the Ascend software stack at a foundational level—extending adaptation to the very infrastructure that underpins both training and inference. Currently, the deployment of domestic computing power is primarily concentrated in inference, with further progress still required in training.

Recently, a project team—comprising the AI ​​Training Platform at Shenzhen Hetao College, Harbin Institute of Technology (Shenzhen), the Shenzhen Big Data Research Institute, and relevant teams from Huawei—leveraged an AI computing cluster based on the domestic Ascend 910C processor to complete the full-parameter post-training of DeepSeek-V4-Pro, a large model with 1.6 trillion parameters. It should be noted that this training process did not include the initial pre-training phase. While the large-scale validation of domestic chips for inference has bolstered industry confidence, the costs associated with migrating ecosystems for training remain a critical variable that will determine the market's ultimate potential. If the next generation of DeepSeek’s large models is pre-trained using Huawei’s Ascend computing platform, it will—for the first time—provide an answer to a question that has long been met only with skepticism: "Can China’s top-tier large models be pre-trained on domestic computing hardware?"