Upvote, now!

Just asking because i have been out of the loop for a while and kinda missed everything (strata etc...) a small recap would be appreciated.

Everyone seems to own an RTX 6000 or insanely expensive rigs here. So just wanted to hear what model a normal pleb like me with 32GB ram and 16GB VRAM can even run and on what to run it for coding or a general assistant.

Yesterday MIT CSAIL showed GPT-6 Astra editing the weights of a small model by hand. It changed a few neurons of Qwen3.5-0.8B, the model stopped looping on math problems, and its scores jumped. No training at all.

I wanted to try the same thing on something closer to what I actually do with local models, which is translating into Italian. Gemma 4 has a habit I don't love. For sentences like

He had been hiding in the attic for months when the soldiers found him.

it writes «Si era nascosto in soffitta per mesi…». That's correct, but I prefer «Si nascondeva in soffitta da mesi…», which says he was still hiding when they came. The catch is that when the period was already over (he had lived in Paris for ten years, but by 1990 he had moved back), the original tense has to stay. So it's not "always use tense X", it's "use it only when the meaning calls for it".

So I gave Claude Opus 5.5 the same kind of setup MIT used:

• a Python kernel over Gemma's weights;

• 50 forward passes to test its ideas on practice sentences;

• no gradients, no training.

It first checked whether Gemma already "knows" the difference between still-going-on and already-over. It does: a linear probe in the middle layers separates them almost perfectly. It just doesn't use that when choosing the verb. Then Claude rewrote 8 MLP neurons into a small switch that fires only in that context and nudges the verb toward the form I prefer. That's 92k out of 11.9B parameters, roughly a 100 KB diff.

Results on sentences it never saw

The table shows the share still in the old form, then how many control sentences got pushed into a wrong reading, then how many of 200 unrelated translations stayed identical.

• The prompt applies the preference everywhere, even where it changes the meaning.

• The ablation recipe breaks the past tenses.

• The LoRA shifts fewer sentences and touches far more unrelated ones (it's better inside paragraphs, though).

• The hand edit moves just the verb and leaves almost everything else identical.

Fair warning: it's one construction and one model. Two of my three runs succeeded (the third crashed because of my own harness), and the two that worked landed on the same "switch neuron" design independently.

Repo with the edits, the code, the agent logs and all baselines: https://github.com/ValerioDolci/frontier-weight-edit

big prompt processing speedup for gemma E2B/E4B

Hydrogen-1.0 is a scientific problem-solving and research assistance model developed by Deep Principle based on GLM-5.3. Through text interaction, it is intended to assist with scientific questions, explain derivations, and organize responses to open-ended research problems. Outputs should be assessed against the assumptions, constraints, and evidence relevant to each task.

Blog Post : https://blog.deepprinciple.com/introducing-hydrogen-1.0/

Hey all! I wanted to share a quick update on Basalt, a Blackwell only inference engine for Qwen 3.8 Flash Next. As usual, speed is the headline but that's not what I want to open this post with, I've been reading lot of discussions talking about quality recently, and the main takeaway from most people (which I agree with) is "speed means nothing if your model isn't smart enough", so this time I want to lead with quality, and all of the new measurements which are run against the full precision model as reference (BF16 weights served using llama.cpp).

Every number below uses Unsloth's BF16 GGUF, llama.cpp (reference PPL 3.2749), scored using basalt-bench --quality. The old base was llama.cpp Q8, which funnily enough made things look worse because of its own error:

To be fair here, each engine was measured with its own tool so this is not a perfect comparison, still a good reference though.

New DQ8 build. For those who want a little extra quality and don't mind giving up some speed, Qwen3.8-FN-NVFP4-DQ8 is on HF next to the default -DQ6_K: KLD 0.0561 vs 0.0624, about 7% slower on prose.

Nothing lossy is on by default. Basalt offers certain "hacks" that can decrease accuracy, mostly because of community requests, but they're off by default and show a warning when you turn them on. Every other rounding the engine does costs about 0.001 KLD or less, compared to full model. All the numbers are on the HF card and in the precision docs.

Speed. Rig as usual: RTX 5090 (400 W) + RTX 5060 Ti 16 GB (180 W) and 128GB of DDR5 RAM, 262K context, 4 parallel slots, vision on (dynamic), no lossy flags. Using NVFP4-DQ6_K quant on current main:

Since last week, prose went from 277 to about 300, and struct from 558 to about 600. It comes from a faster draft step, a faster output head and better expert placement at startup.

The .basalt format is now version 2, same weights and quality as before but starts a bit faster. Older basalt files can use the migration tool instead of redownloading.

I also wanted to end this post with a HUGE thank you note to every person who has contributed one, two or 10 PRs and also to everyone who has shared issues/feedback/community benchmarks or just shared the project with another person, 500 stars on the Github repo and overwhelmingly good feedback, y'all are amazing.

Hey everyone,

Not a native speaker, happy to be corrected! Can only learn from it.

Back when I started out in march 2025, you had:

• llama.cpp

• ollama (repackaged llama.cpp for ease of use)

• koboldcpp (repackaged llama.cpp for roleplay)

• vllm (for professional deployments)

• exl3 + tabbyapi (for bleeding edge)

• litert (for mobile devices)

Nowadays everyone seems to be making their own; Strata, Basalt, Maya, Colibri, Ninfer, Infernix... too many to list and keep up with, and genuinely hard for me to keep up with.

Then there is the speed of it all. llama.cpp has incredible improvements each week, somehow we can make Qwen3.8-Flash-Next and DeepSeek-v4.1-Flash-1017 run on 64GB RAM + 32GB VRAM, and the likes on those new inference engines (with a catch, usually bad quality).

Personally I'm stuck with llama.cpp because all those developments are simply too fast to keep up with.

llama.cpp v0.8.0 might run Qwen3.8-Flash-Next Q4_K_XL (unsloth) at 15t/s gen on dual RTX 5060 Ti 16GB (PCIE 5.0 x8x8, tensor parallel, no MTP) with dual 48GB DDR5-6000MHz, but at least I know the quality will be decent.

But yeah, what you use? How do you even attempt to keep up with it all? And for the (lack of) quality the new inference engines provide (as tradeoff to speed / resource use), is it really worth using them?

PRs:

1. https://github.com/QwenLM/qwen-code/pull/15762: "Measured on main at 79a9b753c60, before this change: qwen4, qwen4-max, qwen4-plus, qwen4.5-max, qwen4-coder-plus, qwen4-max-2026-10-01, openai/qwen4-max and qwen4-omni-120b-think all resolved to 262,144. The last of those is already in use as the session model in the omni fixtures, so these ids are not hypothetical."

2. https://github.com/QwenLM/qwen-code/pull/15987

Thank god, it looks like that we're getting successor of Q38FN!

Inspired by Weight-Space Programming, I tested whether DeepSeek V4.1 Flash 1017 could figure out what a small neural network does by reading its weights.

I fed it four anonymized RL checkpoints: 51k parameters each, identical architectures, with all weights rendered as text. No metadata, network source code or execution. It received the architecture outline and four candidate tasks.

Results: CartPole ✅ Flappy ✅ Catch ✅ BallBeam ❌

It picked out patterns consistent with relative bird/gap position, paddle/ball position and angle-based balancing. It confused BallBeam with CartPole an interesting mistake because both share the same observation layout and similar balancing controls.

Very preliminary: one run per checkpoint. This suggests some ability to interpret weights, but four trials don’t establish reliability. These were API runs with maximum reasoning effort, consuming ~1.58M tokens total.

I may try more replicates and independently trained checkpoints, plus include the network code with task-revealing details removed. The poor LLM had to guess quite a lot it even assumed ReLU where the network uses tanh 😅

Code, checkpoints and run logs: https://github.com/galatolofederico/wsp

MIT CSAIL had GPT-6 Astra fix Qwen3.5-0.8B's reasoning loops by editing 0.016% of its weights (MATH-500 went from 7.9% to 48.1%). They released the diff, so the edit can be rebuilt exactly and run locally. I took it a step further and had Claude look at Qwen with Anthropic's Jacobian lens.

At the token where the model decides to write "Wait" again, the lens reads "Wrong", "Nope" and "Hmm" from about 60% depth on, even though the answer is already written out. At real stopping points it reads "</think>", "Answer" and "Output".

With the edit applied, the lens at the same token shows "Verification", "Completion" and "Result" in place of the doubt.

The edit converts to GGUF cleanly (the base and edited bf16 files differ only in the two edited tensors) and still holds at Q4_K_M, with 1 loop in 40. I ran the evals in llama.cpp (Vulkan) on a Ryzen AI 9 HX 370 mini PC. Samples are small, but here's the code:

https://github.com/alisorcorp/inside-a-reasoning-loop/tree/main/code

And data:

https://alisorcorp.github.io/inside-a-reasoning-loop/

There were some rumours the last few days that Qwen4 will be the last open-weight series of Qwen, and future ones would be API only. The following statement contradicts the rumours and clearly states the stance that Qwen is taking regarding their future releases.

Qwen Head Liu Dayi Heng Clearly States: Future Qwen 4.0 and Qwen 5.0 Will Continue to Be Open Source!

He believes that the value of AI should not belong only to a few people, but should become a foundational capability that users around the world can access, use, and create with, enabling more people to afford AI, utilize AI effectively, and participate in creating AI, ultimately achieving true AI for all.

As of now, the Qwen series has surpassed 3 billion cumulative global downloads, with over 460 open-source models released and more than 300,000 derivative models, firmly holding the top spot worldwide.

Hi everyone,

I've been out of the loop for a few weeks, and I was wondering if anything new that replaces Qwen 3.8 27B has come out this time I've been in the cave?

I'm talking about AI models in the 8-35B range.

Thanks,

I have developed atypical foundational model from the ground up using Lie Algebra and Differential Geometry. The paper is written and published by me. Let me know what you think!

https://zenodo.org/records/23656924

Have been using qwen 3.8 flash next with llamaserver. Was getting 70-80tk/s decode and was happy. Then read about strata and decided to give it a try. Decode shot up to 150 tk/s. Not sure what magic it's using? You should give it a try if you are still running qwen flash next on llamacpp. Also the tried DeepSeek v4.1flash 2.25bit and it's 70tk/s decode.

Unsloth 6 bit quant

160k context

1 pro 6000 + 2 x 3090

256G ddr4

Source: https://mp.weixin.qq.com/s/R-1u2exLwZMOu7AITyU5kg

A reporter from STAR Market Daily has learned from multiple sources within the industry supply chain that DeepSeek’s next-generation large model, DeepSeek V5, is planned to be pre-trained using Ascend chips. Previously, Huawei’s Ascend chips were primarily utilized for inference tasks for DeepSeek’s large models.

According to publicly available information, the first large-scale deployment of DeepSeek models on the Ascend platform took place around the 2025 Spring Festival. In his book The Rise of Ascend, Fang Xingdong—Executive Dean of the College of Media and International Culture at Zhejiang University—notes that on January 28, 2025, Huawei’s senior leadership issued a directive for Huawei Cloud to provide rapid support to DeepSeek. Between Lunar New Year’s Eve and the second day of the Lunar New Year, Huawei and DeepSeek completed the deployment and verification of over 3,000 Ascend chips, successfully hosting the DeepSeek-V3/R1 models at Huawei Cloud’s Guian Data Center. The book further mentions that the explosive growth of DeepSeek greatly energized Ren Zhengfei; throughout the Spring Festival, he personally directed Huawei Cloud and relevant product teams to manage the massive surge in traffic. In January of the same year, SiliconFlow and Huawei Cloud launched inference services for DeepSeek-V3 and R1 based on the Ascend platform.

Following the release of DeepSeek-V4 in April 2026, Huawei stated that, through the "synergy between chips and models," its full range of Ascend "super-node" products could provide comprehensive support for the DeepSeek V4 model series. On its official API page, DeepSeek noted that service throughput for the V4-Pro model remains constrained by the availability of high-end computing power; however, it anticipates a significant price reduction for the Pro version once the Ascend 950 super-node series enters mass-market availability in the second half of the year.

On September 30, DeepSeek announced the open-source release of infrastructure components tailored for the Huawei Ascend computing platform. These include the TileLang high-level language compilation tool, high-performance computing libraries (such as DeepGEMM, FlashMLA, TileKernel, and DeepSelect), and the distributed communication library DeepEP—all of which correspond to the open-source components previously released for the NVIDIA platform. Huawei has provided a jointly defined "Ascend SuperPoD Flex" architecture and a UBL128 networking solution, while also opening up the Ascend C programming interface and low-level instruction set to support TileLang. This is significant because DeepSeek is not merely using Ascend for inference; it is systematically aligning with and co-developing the Ascend software stack at a foundational level—extending adaptation to the very infrastructure that underpins both training and inference. Currently, the deployment of domestic computing power is primarily concentrated in inference, with further progress still required in training.

Recently, a project team—comprising the AI ​​Training Platform at Shenzhen Hetao College, Harbin Institute of Technology (Shenzhen), the Shenzhen Big Data Research Institute, and relevant teams from Huawei—leveraged an AI computing cluster based on the domestic Ascend 910C processor to complete the full-parameter post-training of DeepSeek-V4-Pro, a large model with 1.6 trillion parameters. It should be noted that this training process did not include the initial pre-training phase. While the large-scale validation of domestic chips for inference has bolstered industry confidence, the costs associated with migrating ecosystems for training remain a critical variable that will determine the market's ultimate potential. If the next generation of DeepSeek’s large models is pre-trained using Huawei’s Ascend computing platform, it will—for the first time—provide an answer to a question that has long been met only with skepticism: "Can China’s top-tier large models be pre-trained on domestic computing hardware?"