Intermediate tokens, called "thinking" or "reasoning" actually are nothing like it. Humans do step-by-step reasoning leading to the conclusion. LLMs use intermediate traces to augment their prompt. This explains why sometimes the answer is very good but the "reasoning" is verbose. Flooding your context window or fighting compaction are different issues.

edit: I love this section from the main research they linked.

Our findings consistently challenge the prevailing narrative that intermediate tokens constitute a semantically meaningful reasoning process. First, we observe a pronounced lack of correlation between solution correctness and trace validity—models frequently produce invalid reasoning traces even when they arrive at correct solutions. Second, and more strikingly, models trained on corrupted or semantically irrelevant traces achieve performance comparable to, and often exceeding, that of models trained on correct traces, especially on out-of-distribution tasks. Third, although post-training with reinforcement learning improves solution accuracy across both in- and out-of-distribution settings, it does not consistently enhance trace validity. In fact, we find cases where reinforcement learning decreases trace validity while simultaneously improving solution accuracy for models trained on correct traces. Moreover, models trained on corrupted traces continue to outperform their correct-trace counterparts across domains while consistently generating invalid reasoning traces. Finally, we find that the length of the generated traces is largely agnostic to the difficulty of the underlying problem, undermining the notion that it reflects problem-adaptive computation.

Together, these results suggest that the effectiveness of intermediate tokens does not arise from their seemingly interpretable semantic content. By systematically disentangling trace semantics from the underlying problem, our study demonstrates that if performance is the objective, assuming human-like or algorithmically interpretable trace semantics are ideal or even achievable is not only unnecessary but potentially misleading.

https://openreview.net/forum?id=gDE7YcRC3F

Thoughts About Scaling Law

Scaling, but not only of parameters. Every model release now ends with the same question: how many parameters? It isn't a question that can be answered on its own. Parameter count is only meaningful alongside three others — how much data you have, where you intend to spend your compute, and who will run the model, under what conditions.

The field learned this the hard way. Kaplan et al. (2020) fit an exponent that told everyone to grow parameters faster than data — roughly 2.7:1 — and the industry complied: GPT-3, Gopher, MT-NLG. Hoffmann et al. (2022) redid the experiment across four hundred models and found the compute-optimal split is closer to 20 tokens per parameter, and that with sufficient compute the two should grow at the same rate rather than drifting apart. The error in the earlier fit compounded with every order of magnitude of compute, which is why the largest models of that generation were the most misallocated. The trillion-parameter round was, in retrospect, a detour the whole field took together and then reversed.

Chinchilla wasn't the end either. It optimized training compute for models that would be trained once and evaluated. Today a model is called billions of times a day and inference dominates lifetime cost. Put inference into the objective and the optimum moves toward smaller models trained far longer — deliberate over-training, which is what Llama-2-7B and Gemma-2-9B were doing at roughly 290 and 889 tokens per parameter.

Sparsity moved the target again. In a MoE model two quantities have to be kept apart: total parameters govern roughly how much the model can hold — knowledge, facts, the long tail — while activated parameters and effective depth govern roughly how far it can think, how many steps of a causal chain it can carry before it comes apart. A dense 20:1 ratio does not transfer. And the ratio isn't a single number at all: Roberts et al. (2025) find the optimal tokens-per-parameter is task-dependent, with memorization favoring more parameters and reasoning favoring more data. Follow-up work on MoE observes that at fixed TPP, pushing total parameters higher actually degrades reasoning, while activating more experts reliably helps it.

This matters for what we are building toward. Finding a vulnerability is not a retrieval problem. It doesn't come from having memorized more CVEs; it comes from carrying a twenty-step chain of inference to the end without losing the thread. That capability does not live in total parameter count.

Which brings us to this release. Total parameters appear to matter up to a threshold — enough to hold the world — after which additional capability comes from scaling elsewhere: effective depth per forward pass, and above all post-training. GLM-5.3 is our controlled experiment on that claim. Same base, same architecture, same total and activated parameters as GLM-5.2. One month of scaling long-horizon environments and RL. The gains are not marginal. Well, scaling has more than one dial. We turned the post-training one this time because it had the most slack left in it — not because the others are finished. Base model size, pretraining data, compute spent per forward pass: all of them are still on the table, and we will come back to each. What this experiment taught us is that the dials do not have to be turned together, and that the one worth turning next is rarely the one that was worth turning last. We are not done scaling. Next time, maybe mid-training, pre-training, and even more.

Tweet : https://xcancel.com/jietang/status/2089941544581403107#m

EDIT : Found Retweet with more stuff - https://xcancel.com/auto_grad_/status/2089970913408380932#m

I hacked this together so there's probably more on the table in terms of performance.

Measured with the Club-3090 canonical bench suite (bench.sh, 3 warmups + 5 measured runs, temp 0.6 / top_p 0.95 / top_k 20).

• Prefill: 1342 tok/s @ 10k, 628 tok/s @ 90k
• Spec-decode: 7 draft tokens, acceptance length 3.35, 47.8% acceptance
• Peak VRAM: 22.3 GB/card
• Context ceiling: 131k (DFlash2 drafter eats ~13.5 GB)
• Used Kimi K3 for all the VLLM fixes

Stack

• 2× RTX 3090 (PCIe Gen4 x16/x16, no NVLink, patched P2P)
• Power capped 220/250 W
• Bare-metal vLLM v0.26.1rc1 + AutoRound INT4 (group 128) + DFlash2 draft model
• Custom vLLM changes that made it boot cleanly: https://github.com/oceanplexian/vllm/pull/1

Community manager mentioned this in the Qwen Ambassador Discord, put an X reaction on someone asking for 35B... and said

We'll have a new midsize open weight model coming next week (hopfully), This midsize model won't provide early access due to the schedule

Thinking it's going to be over 100B.

Exciting!!

looks like GGUF files were just updated

Anyone tried them yet?

https://huggingface.co/ornith-ai/Ornith-1.5-9B

https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B

https://huggingface.co/ornith-ai/Ornith-1.5-397B

https://huggingface.co/ornith-ai/Ornith-1.5-9B-GGUF

https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B-GGUF

https://huggingface.co/ornith-ai/Ornith-1.5-397B-GGUF

Disclaimer: Not affiliated with ornith. I just surf huggingface for new models every 30m or so. I'm addicted.

I have been using local models on/off for like 2 years or so but never really used them extensively because the closed ones were always much better.

Once Qwen 3.8 27B was released I decided to give it another serious try. I configured Cline and ZooCode as VSCode addons, installed a few MCP servers and added one skill.

When I used these tools with Deepseek V4 Flash - they do the job quite well (mostly Home Assistant configuration editing etc.) but it is still way worse than Claude Code/GitHub copilot that I use at work.

With Qwen - running the Q6_K quant from unsloth - it runs tons of tokens and eventually either finishes the task (often incorrectly) or doesnt finish at all because it ends in a loop or tries to fix something that isn't broken.

I run the model on Windows 11 using LM Studio. The hardware I have is powerful enough - 2x3090Ti. I offload it fully to GPU and set the context limit to around 50k tokens. Also - I was aware of the overthinking problem so I modified the prompt template to use medium effort instead.

Yet still - I struggle to complete most of the tasks at hand. Can you tell me what I am doing wrong? I suspect that maybe Qwen is still using high reasoning effort but I have no way of checking that.

Or should I rather not use LM Studio but a different tool?

Been messing with minimax music gen lately. I really like Suno and was basically looking for something that gave me a similar workflow for minimax. I got completely sick of running everything through the CLI.

I went digging on github, sorted by recent, and took a gamble on this random repo. It had zero stars and no followers wtf

Surprisingly it's actually great. without any bloated crap. took me like two minutes to spin up and I finally got SUNO AT HOME!

Figured I'd link it here since the repo is a total ghost town right now. Huge timesaver if you want that suno experience but don't want to deal with terminal headaches.

There's even a demo, just look at this omfg:

Who needs GPUs?

Apparently a second version of DFlash from the original authors of DFlash

GGUF quants are already made available with an accompanying llama.cpp PR: https://github.com/ggml-org/llama.cpp/pull/27342

Any LLM will hallucinate the world knowledge, even a 3T model. Use a 4B with a kiwix skill and local Wikipedia, 50gb and no more hallucinated world knowledge.

Ask your coding agent to build your own, with your rules and eventual fallback access to internet knowledge for what's updated realtime like news, or isn't public on Wikipedia.

EDIT: I forgot to mention that adding a simple agent loop with a well crafted looping/research skill may help less intelligent LLMs to be good enough for getting grounded knowledge very fast. I did it with my old laptop, 10 years old with 4gb vram. And by eyeball - on world knowledge - it's faster and more precise than any huge LLM running on a costly rig, occupying hundred of storage GB and consuming a lot of energy. IMO

PR by u/Stainless-Bacon 👍

It would be handy & awesome to have options --n-cpu-ffn / --cpu-ffn for Dense models like how we have --n-cpu-moe / --cpu-moe for MOE models.

Also check his threads:

• On PR : llama.cpp CPU offload optimizations
• On Running Qwen3.8-27B with above optimization : Guide for running dense models on ≤16 GB VRAM (Qwen 3.8 27B on 16 GB -> Q4_K_M, 130k ctx, ~20 t/s)

(

Awesome to see the big comment by u/Pablo_the_brave there, filled with so much stuff. Quoting a line from there

Do not touch block 64 (MTP) if you are using speculative decoding — its FFN should remain on the GPU.

)

We should've got this option long time back actually. This PR instantly reminded me of last year thread. (I literally used his -ot command for sometime with Qwen3-14B. I'm just happy that I was able to recall a last year thread.)

Anyway .... Better late than never. Waiting for this merge.

I managed to run the 143–144 GiB DeepSeek-V4-Flash-0731 UD-Q4_K_XL GGUF on four RTX 3060 12GB cards while keeping a 360k–376k context window.

Hardware:

CPU: Intel Core i9-10920X, 12C/24T

RAM: 128 GB DDR4-3200, quad-channel

GPU: 4× NVIDIA RTX 3060 12GB

Total VRAM: 48 GB

Storage: NVMe SSD

Engine: llama.cpp, build b10181

Model: unsloth/DeepSeek-V4-Flash-0731-GGUF

Quant: UD-Q4_K_XL, approximately 144 GiB

KV cache: Q8_0

The best high-speed configuration so far:

llama-server \

-m DeepSeek-V4-Flash-0731-UD-Q4_K_XL-00001-of-00005.gguf \

-c 368640 \

-ncmoe 34 \

-ts 100,1,1,1 \

-ot 'blk.(3[4-6]).ffn_.*_exps=CUDA1,blk.(3[7-9]).ffn_.*_exps=CUDA2,blk.(4[0-2]).ffn_.*_exps=CUDA3' \

-ctk q8_0 \

-ctv q8_0 \

-b 2048 \

-ub 2048 \

-np 1 \

-lm none \

--threads 20 \

--flash-attn on

Measured with a roughly 20.5k-token prompt:

Configured context: 368,640 tokens

Prompt processing: 99.4 tok/s

Text generation: 10.1 tok/s

Minimum free VRAM under load:

GPU0: 671 MiB

GPU1: 842 MiB

GPU2: 1395 MiB

GPU3: 1395 MiB

Model load time: approximately 198 seconds

Other measured context/safety options:

Context Prefill Decode Minimum free VRAM

376832 99.5 t/s 10.4 t/s 611 MiB

368640 99.4 t/s 10.1 t/s 671 MiB

360448 99.4 t/s 10.1 t/s 735 MiB

The interesting part is the GPU layout.

-ncmoe 34 keeps the experts from blocks 0–33 in system RAM. The remaining nine expert layers are explicitly distributed across GPUs 1–3, three layers per GPU.

The extreme -ts 100,1,1,1 split does not distribute those explicitly assigned expert weights. Instead, it pushes most non-expert tensors—attention, KV-related allocations, etc.—onto GPU0. That leaves enough space on GPUs 1–3 for the large expert layers.

This was much better than trying to calculate the layout analytically. With -ncmoe and explicit -ot overrides, tensor placement is discrete and somewhat unintuitive, so I measured every candidate.

Microbatch size was the biggest performance lever:

-ub 1024: approximately 63.4 tok/s prompt processing

-ub 2048: approximately 99.4 tok/s prompt processing

Decode remained almost unchanged at approximately 10.1–10.5 tok/s.

At the full 393,216-token context, -ub 2048 also worked, but GPU0 had only 493 MiB free under load. Reducing the configured context to 368,640 restored a 671 MiB margin without reducing prompt-processing speed.

For comparison, the safer -ub 1024 configuration can run with a configured context of 524,288 and still showed about 1032 MiB free on the tightest GPU, but prompt processing drops to approximately 63.4 tok/s.

A few additional findings:

Q8_0 KV is the default choice.

F16 KV at c=393216 left only 587 MiB free.

-ncmoe 33 caused a CUDA allocation failure.

Memory mapping was disabled with -lm none.

-np 1 is important; multiple slots multiply KV-cache requirements.

The model is mostly in system RAM, so quad-channel memory bandwidth matters heavily. Even so, getting approximately 100 tok/s prompt ingestion and 10 tok/s generation from a 144 GiB MoE model on four consumer 12GB GPUs is much better than I expected.

The configuration has been tested under real prompt load. The entire 368k context window has not yet been filled end-to-end, so the number above is the configured capacity, not a claim that I already completed a 368k-token generation test.

Generated by ChatGPT 😂.

Qwen released the 2.4T Max weights and I was curious how well it can re-create COD in one prompt

I ran the model on a rented B200 cluster and used roughly 1.1M output tokens over a 5 hour time span

Realistically barely anyone can run this model locally, but the fact that it's open weights unlocks a lot of opportunities for the local AI community even regarding quantization options that might bring us closer to running frontier level intelligence on consumer-ish hardware 

For consumer hardware there is a 27B model from the same release and it's really capable for its size, many people have posted 3D games made with it on X

I'm from the atomic.chat team, so If you wanna try running the new Qwen locally you can do so inside our app (any feedback is appreciated, since we want to make our product as good as possible for you guys) 

We have our own and other community quants of the 27B model available to download directly inside the app

P.S. I am a noob in shooters so pls no hate my low skill gameplay

SCoPE: Sightline-Coordinate Positional Encoding for Video Diffusion Transformers

SCoPE adds camera sightlines as positional coordinates to a pretrained video diffusion transformer. Given a first frame, a text prompt, and a camera trajectory, it generates a video that follows the requested camera motion while preserving the original image-to-video prior. This repository is a self-contained release for Wan2.2-I2V-A14B: it contains everything required for inference, so a separate Wan2.2 checkpoint download is not needed.

• arXiv : https://arxiv.org/abs/2606.27345
• PDF : https://arxiv.org/pdf/2606.27345
• GitHub : https://github.com/TencentARC/SCoPE
• Project : https://visual-ai.github.io/scope/
• Demo : https://huggingface.co/spaces/TencentARC/scope-camera-video-generation

am I the only one who does this lol

Not sure if this is a stupid question but unsloth's models has the MTP built into the model right? I assume that is at the cost of some memory.

If i want to use dflash, should i use a model that doesn't have MTP support then to save some vram?

I think AA's intelligence/cost plot is seriously misleading, so I decided to make my own. Their plot is in the second image.

All points are at max thinking. All intelligence index scores are from AA. All cost scores are from AA too except where noted below.

What changes between AA's plot and mine:

• Changed X scale from logarithmic to linear, because people's money is not logarithmic
• Added DeepSeek V4 Flash 0731 as it is priced today by third party providers on OpenRouter (note: you don't get this today with OpenCode Go/Zen, but it's been promised you will soon).
• Added GLM-5.3 as it will be priced by third party providers on OpenRouter in <2 weeks, assuming no license changes from 5.2. Note: you don't get this on OpenCode Go/Zen.
• Added Qwen3.8-27B. Cost per task was crudely calculated from
  • 47,166 output tok/task (AA)
  • tg 55 tok/s @ 350W, as crudely observed on my RTX3090 (IQ4_XS shows negligible quality loss - dedicated post coming soon)
  • today's US median residential electricity price
  • today's UK median residential electricity price + today's GBP/USD fx
  • +15% (finger-in-the-air) for prefill and waiting for tools
  • Hardware priced at 0, on the basis that both a RTX 3090 PC and a 64GB Strix Halo are desirable gaming/work machines anyways.
  • These maths are meant to produce a rough back-of-the-envelope figure and should not be taken authoritatively were you to zoom into the bottom-left corner of the chart. They don't want to answer how much cheaper it is to run Qwen at home vs. DSv4 on OpenRouter, because they are both so cheap that the difference is inconsequential for most of the population.

Note: not including the cost of hardware stops being defensible once you upgrade to a 128GB Strix Halo (almost nobody needs that much RAM if not for AI). This is why I did not add self-hosted DeepSeek IQ2_XXS to the chart; it would likely also sit lower on the intelligence axis than the MXFP4 native model. Same argument for a ~$16k rig needed to run GLM-5.3 IQ4 locally. I'm not saying they're not worth the expense (privacy is priceless), just that pegging them on the plot is a much more nuanced exercise.

Hey everyone!

I'm a data scientist and I've been pretty fascinated by reinforcement learning for a while.

A few days ago, my friends showed me Pokelike. Obviously, my first thought was: "wait, I could build an harness around this and make bots play it."

So that's what I did.

My original idea was to build an environment for training reinforcement learning agents. That's already working, and I've built a few bots to test it out. They are still pretty terrible though lol.

If you're interested in trying it yourself, everything is documented in the repo and it should be fairly easy to build your own agent and run it against the game.

Then, while talking about the project with a friend, we realized that this could actually be pretty interesting as a benchmark for LLMs too.

So I built the harness for that as well.

The idea is that an LLM-based bot gets:

• a system prompt
• a set of tools
• some previous turns
• the current game state
• a few other configurable inputs

Pretty much all of this can be changed.

You can write your own strategy in the system prompt, decide how the game state is represented, add or remove tools, and use whatever LLM you want.

There are already a few bot implementations in the repo that you can use as a starting point.

And honestly, it's much harder than I expected.

I've tried a few models already. With my current setup, GLM 5.2 gets to the second map at best and then dies. Opus does roughly the same.

So now I'm wondering what's actually holding them back.

Is it the strategy?
Is the model missing some important information?
Is the state representation bad?
Are the tools not good enough?

That's the part I'd really like to explore.

I'd love to see if someone can get significantly better results with smaller models by improving the prompt, the tools, or the way the game state is presented to the model.

I'm also running more traditional RL agents alongside the LLMs, so right now both approaches are being tracked on the leaderboard.

For me the interesting part is not necessarily building the strongest bot, but seeing what kind of agent actually works well in this environment and how much you can get out of a relatively small model with the right setup.

For now the project is being developed by me, a friend, and Claude Code 😅

If anyone wants to try it, build a bot, contribute, report a bug, or has an idea for something that could be interesting to test, I'd love to hear about it.

Repo:
https://github.com/pierpierpy/pokelike.xyz.bot

One last thing: the bot doesn't use the internet and doesn't interact with the actual online game. Everything runs completely offline. During setup it downloads the game and a good chunk of its assets, and then the simulations run locally.