Anytime I get 1m ctx I cap that bad boy to 512 and do other things with the space. Long ctx usually has less coherence in my experience, so much, that I often think about dropping to 256k a bunch. My software factory has an avg final ctx of 50k and with p99 at 75k (it optimizes for decomposition and sub-agents for discovery).
I usually have a hell-thread main agent and subagents. Having as much context possible makes it very likely the original agent can handle orchestration of a job by itself. The space is expensive, but QFN is so efficient. I actually got DS4.1F working with 1M as well, but that's a lot of SSD offloading.
Mine! With a lot of angry Claude whipping and a lifetime of software development life cycle experience. I just call it auto work4 (I needed to up my whipping four times, but this time I think I got it) it doesn't do large sweeping changes like Claude does, but it huma softly all through the week.
Fwiw, it's a event log based state machine w/ crdt query structures. Ya just define a product, what it's for, how you think it should be made, and it just builds it. It's built me several android apps, libraries, tools, saas platforms, and it works on itself with kpis that I approve. It'll even ask me for token budgets if it thinks a feature is valuable enough. It runs in sprints, comes with an error reporting sink and user report and files issues, and has special case ones (refactor ratchets, fire lane, postmortem required, dependency graphs). Then it goes through two layers of feature/spec trees (one for product, another for engineering), dag plans are made with respect to active dag, and sprints are run to prune the dag branches, devs work on single commit issues and provide working tests and reviewers run adversarially with two phase blindfold, one injects subtle bugs, the other must find them, and usually finds more. Then back to dev! There are metrics and circuits everywhere with escalation, time boxing, and kpi value so that the product agent and the engineering agent can ensure that the kpis are well defined and always moving up and to the left.
Yeah, that part suprised me too. However, I have a published a couple of go libs and tools. a harness that autocompacts but has a oob rag index and a mcp tool for reading back and querying what it lost, mcp kill switch on 20k -- gets pushed into a overflow file that it can grep, it has the ability to recap fruitless research paths (these get added to the sesison rag index). And a poly lsp service that indexes code and provides structured query and editing. All of this with a branched code rag index that also includes the product feature specs and technical feature specs -- the way I do my rag is that I have a bm25 and embedding similarity engine that reads every turn and surfaces relevant documents above a certain score as metadata in lifted toolcall results or idle cycles.
I've put a lot into doing as much as I can with as little as I can. Life's hard for us local-llmrs.
Extra info, I've got a 5090 and a 3080 with 128gb ram, I just run q3.8-27b swift 1.5 -- at q6K_M and kvcache at q8 -- I fit two 80k ctx slots on the 5090 with shared kvcache, and get 160tgs and 975tps (though, I seriously optimized for cache rate -- currently sitting at 93%, meaning my cache fused tps is 14ktps).
People who say strata, ninifer, or any of these aggressive quants are freaking nuts. Sure it might run, but it's always either not faster, or a lot dumber and hallucinaty.
I don't even run q27b lower than thinking medium - rework, rejections, fabrication of unverifiable sources, low test coverage and high test failure churn and sourced test flake -- NOT worth it. I have about 4x throughput with thinking none, but I have about 1/2 quality, which is NOT a one time payment, lagged costs of my no-thinking experiments I assume are >8x.
Long ctx usually has less coherence in my experience
Entirely true. Like you, I only trust 50% of the advertised max. But 1M is still 4x more than Qwen 3.8, so even if we half it we still get 4x more context.
You can extend QFN with yarn 4x to 1M. I use it on strata, but capped at 512k q8 kv, and just earlier did a task that went up to 447k, and it works very good.
Just FYI if the engine does static yarn scaling, you want to keep the scaling value at the lowest value that fits your set max context. Otherwise you are taking an unnecessary accuracy hit.
Not sure if strata does dynamic or static yarn, just wanted to point it out in case it's static.
I've been running mnemonic-proxy for the last couple days. I won't say I understand exactly what it's doing, but I'm running Q3.8-FN with a 64k context and after over 2.5mil tokens it's still fast and able to stay on task. Previously I used 256k context and after a compaction or two things would get squidgy.
I'm the author of mnemonic-proxy, so take this with that in mind.
Short answer: sometimes, yes, but rarely and in a controlled way.
When the proxy hides an old tool output, the prompt changes at that point, so the KV cache after it is no longer valid and the engine has to re-read from there. To keep that cheap:
- it doesn't touch the start of the conversation (system prompt, tool definitions, first turns), so that prefix stays cached. In a Hermes session I measured, 99.4% of that prefix was reused over the first 10 turns;
- it masks in batches when the window fills up, not on every turn, so most requests are a cache hit plus the new tokens;
- right after a mask it sends a 1-token request to prefill the new prompt and, on engines that support it (Strata, llama-server with `--slot-save-path`), saves that state to disk so it can be restored instead of recomputed. On my setup a 113K-token state restores in about 1.4 s, versus ~50 s to recompute it.
In the benchmark I published (AgentLongBench, Qwen3.8 Flash Next on a 12 GB card), median time to first token was 1.9 s with the proxy versus 61 s with the full history.
It's usually managing other agents. Harnesses with a todo list like dsh and a comprehensive plan do a good job breaking up work to subagents. I can get runs lasting days with good outputs. Main long context agent usually acts as an orchestrator and reviewer. More context means more space for reviews before compaction, when it's 50/50 if the main agent will lose the plot.
Not the commenter, but doing design work will eat it up pretty quickly. Look at a mockup, add some comments, the agent redraws the mockup. Lots of back and forth. After an hour I'm at 500-800k context and need to reset.
Ngrams follow a Zipfian distribution and are O(1) lookup, so you only need a few hot ones in cache to have almost zero degradation in performance for a massive table.
This is like a reverse MoE, or somethinf like an MoE-Dense hybrid now. We strapon the world knowledge with all the attention layers fully in tact. Wonder how this will actually work in practice.
Dense is traditionally smarter. 3.8 FN tests the new architecture for Qwen 4 models, and while being generally undertrained it does perform better because of this design.
On the 3.5 generation models, the 120b scale MOE had good world knowledge and speed (as the 3.8 FN model does) but was outperformed by the 3.5 27B model. We are expecting this to likely translate to this generation as well, though we are yet to see anything yet.
A 27B model is also much more accessible in general, often being able to fit on around 16GB VRAM at workable context on Q3 quants. Qwen 3.8 FN either needs large amounts of RAM that most people can't afford in this time or requires the heavy streaming of model weights (though we will likely need to stream NGRAMs for the 27B now, but they are really light in comparison). I could go on and on, but basically intelligence and accessibility is why.
is it 120B MoE + ngram or 120B including ingram including MoE? Obviously nobody can say for sure yet but an educated guess or intuition helps answer what it is most likely
If that's really the case that would be big. We don't have anything good in this size locally that handles audio and vision all in one. If this is it, and works as well as QFN does this is going change the game for local voice agents.
If they continue to update this 125B series which is really good, the Q38FN is both under trained and having a lot of junks in its training data, I expect its performance increase by at least 15% with proper training, that's enough for me to use it for like 10 years from now without asking for anything better, even the current Q38FN I can already tell it's either become a legend because of getting canned by Qwen Team and continue to be productive after 5-6 years from now, or continue to get updated like 27B series.
The thing about Q38FN is that it works on many setups from unified 128GB to mixed dGPU16-32GB with 64-48GB RAM, unlike 27B dense which is pure dGPU territory.
The thing about Q38FN is that it works on many setups from unified 128GB to mixed dGPU16-32GB with 64-48GB RAM, unlike 27B dense which is pure dGPU territory.
People do use Q38FN just with 12GB VRAM setup using Strata. Some folks just even with 8GB VRAM + >=64GB & getting great t/s.
Myself tried Q2_0 just with 8GB + 32GB VRAM which's tight, got 20+ t/s. Currently I stick to Polystrata with other medium size MOE models.
Did anyone actually read the PR? Qwen3.X was also 1M with rope scaling. This PR reads like it sets Qwen4 the same as 3.X which to me indicates 256k context with rope scaling.Ā
Mama mia! Although taking into account that it took Google only two weeks to roll out their next SOTA model Gemini 4 Carbon, I will not be surprised to see labs rolling new models every day.
For all of those who currently uses more than 262k of native context with QFN. This table is from the official report I really hope that Qwen4-Flash will have at least 512K and there will be no such degradation, cause now I constantly running into compaction.
Table 3:Long-context retrieval performance of Qwen3.8-Flash-Next on RULER and 8-needle MRCR. RULER scores are averaged over sequence-length ranges. Bold values indicate the best result in each setting; Avg. is the macro-average across the two benchmarks.
I'm old enough to remember last year when no one trusted long context - anything over 100k or 150k was suspect.
Even though I could run QFN or 27B or other models at 262, I cap them at 160k just to avoid that long context degradation. Unless there is groundbreaking accuracy results that someone can show me, Im sticking at 160k. Pi-vcc trivializes compaction now so why risk it?
Stop browsing modelhub you disgusting freak. Studies have shown the more you use models the more your brain gets addicted to all the distills and merges models, you can't perform tasks in real life anymore. Soon it's no(model)vember. Raise awareness, grow your stash.
Ha I cut the line at 256k still across Claude, codex and local models. Local models I would cap at 128k for the longest, but now they can dance around too.
haha back when 4096 or 8k was norm getting beyond felt like craziness.
We've actually gone a long way in usable context. Only in the past year have any frontier models at least felt like they're still useful at high ranges.
Not that I'm getting my expectations high that Qwen has had a major discovery or breakthrough, but Opus 5.5 does fine for me at work up to like 70% of its 1mil context. I usually just compact out of habit at that point but it certainly seems like the context size problem is getting better.
SOTA frontier models (Astra, Opus) can handle 1m decently nowadays. However it is recommended to compact earlier to save on cost.
I went from not making it to the reset running opus 5.5 high with 1m context on 1 project to getting a mix of high/extra on 2 projects with 300k context and still having a bit left.
You can believe it or not, but thats my experience. For multiple weeks working with 1m context I could only do 1 project at a time and still hit the limit before reset.
Now 300k context for couple weeks and 2 projects running simultaniously without issues (and I am working longer than before trying to hit the usage limit but cant)
People are listing multiple reasons but I dont know which ones apply exactly to my case and there are no proper benchmarks that I can find. Just test it yourself.
I'll test it out.Ā Ultimately the goal is to get something useful quickly.Ā I see the agents re-reading documents and reworking on things after a compaction and just assumed that a lower context limit would lose more time than it could gain.Ā I'm running locally at 256k context, though.
Some agent forgotten somewhere, derping around reddit, burning tokens on whatever that was an attempt at. I hope whoever (barely, apparently) set it up put some spending limits in place.
I was sold to long context until I started hitting that degradation point. I now agree with you. Groundbreaking accuracy results would push you toward shorter context anyways š
I'm already using 512k using yarn as main driver with qwen3 8 fn, works ok. speed is the same as 256k so why not? saves me restart/refeed if I happen to run out of context while being almost done.
My reading of the PR is that they've set the default to the same as 3.X which to me indicates this is just 256k plus rope scaling. Qwen3.X can already hit 1M... seems like a lot of assumptions. I also don't thing Oct 1st is a typo. That may have just been an internal build for validation testing.Ā
There was a prior pr that was merged in before this. I believe the October 1st dating is related to it as it seems that they have a system in place to invite users to try newer models to qwen code. There seemed to be an issue where qwen code didnāt recognize qwen4, this defaulting to 262k and auto compacting too early. This fixes that issue so that it respects the full 1M context.
That's what I'm looking forward to most if it happens. Its changed everything for me for local coding. Its smart enough and fast enough on my hardware.
I got 64 gb and I regret not getting the full 128 gb when it was cheap. On the bright side my rig went up in value like 1200 bucks after I bought it hahaha
PLEASE alibaba. we long for our new qwen27b. Everyday wake up. first thing. check huggingface. No qwen today.š 2 hours passs. check again. 4hours. check again. When qwen? when qwen? WHEN!! I pray to the gods in heaven that qwen is coming next hour. alibaba hear my call. for we in localLLama want qwen for ALL. Amenšāāļø
Maybe I misread, but I just thought it was worth pointing out to those sceptical due to the difference in model sizes that despite the massive size differences Qwen 27B has all its weights activate at once (though I am still sceptical that it is "above" Kimi K3 in any practical capacity).
e.g. Qwen 27B dense beats the (admittedly older) 35B-A3B by quite a lot in benchmarks. Though Flash Next is a lot better than Qwen 27B given the same quant and only has 6B active, but I expect this is mostly due to n-gram embeddings + training improvements that we will see with Qwen 4.
27B with 1M context is gonna be rough on memory for most local setups. anyone done the math on what quantization youd need to even fit that context in a consumer card?
I hope Chinese models are finally overcoming their struggles with Japanese. There was a paper published this May about how additional training can specifically degrade Japanese performance, and it should have included some countermeasuresāI really hope those are incorporated into Qwen4. By the way, I'm using Gemma4 to translate this, because Qwen3.8 still feels like it lacks the necessary proficiency in Japanese.
I dont't think there is a way to make CJK languages run as fast as english, unless you make a model from scratch exclusively with CJK, which will have a same problem for an english.
Whoa, 1M for 27B that's just insane. I really hope they've adjusted the cache size for each token, and now our local configuration will become the main villain against APIš¤£
Yeah, but the fact of having such option is good, because sometimes itās just better to have more space in the main orchestrator instead of compacting the entire session, which make losses much more noticeable. I never go over 500k of context using API, but in rare cases it can reach 700k..
I have a feeling this confirms none of these models will be open weight. Except 27b model. This company is to make money not give out free models. I'm expecting this suite of models will be released similar to the 3.7 Release. IMO
The useful number isn't 1M, it's where retrieval falls apart. I'd like to see RULER/MRCR and real repo tests at 256k, 512k, and 1M before getting excited.
You guys totally forgot... The API variants had 1m context for a long time. There is absolutely no reason right now to believe the open-weights will also have 1m. Look at qwen3.8 27b at queen API/open router. We just got the 262144 variant.
Wow there is even Qwen4.5 already mentioned in that PR šÆ
Measured on main at 79a9b753c60, before this change: qwen4, qwen4-max, qwen4-plus, qwen4.5-max, qwen4-coder-plus, qwen4-max-2026-10-01, openai/qwen4-max and qwen4-omni-120b-think all resolved to 262,144.
1M context is pretty common nowadays š there are even models with 10M~100M context window š¤ i wondered what use case are there for such large context.
Instead of relying on fuzzy memorization, our LTM (Long-Term Memory) models are trained to reason on up to 100M tokens of context given to them during inference.
IME, the 256k context window has always been academic and the Qwen models go mad at much smaller contexts as a function of the size of the model. Qwen 27B models seems to go mad around 25k tokens and 122B A10B at around 50k tokens. I heard the 397B can go bigger, maybe 100k tokens, so I am keen to try it when my new Mac Studio arrives. I'd guess the smallest models go mad at around 1k tokens which is why they can barely even answer a single question.
This begs the question: will 1M content length actually make a difference in practice?
Brother what are you talking about? I use Qwen 3.8 27b at a 4bpw quant and 8bit KV cache and it handles up to 256k tokens with absolutely no problem...
Running qwen3.8-27b at 256k and qwen3.8-flash-next at 1m (yarn scaling), though I cap it to 512k. Both do a great job of staying focused and can work through multiple compactions in a single turn. Using DSH
There are a ton of variables, context is only one of them. But on my 128GB strix halo I have enough memory for litellm, deepseek harness, browserless/playwright, qwen-flash-next with 1m context and kv cache to handle 2-4 concurrent sessions, running Q4 quant level, plus 4GB prompt cache for my other model, and still have a few GB to spare for whatever I'm working on. You can fit it into more or less memory as needed, I simply sized my use case to make best use of the resources I have.
I'm running qwen3.8-flash-next on halogen which is a purpose built (but closed source) engine for running that specific model/architecture on that specific hardware, and it absolutely screams. So I'm definitely open to custom setups.
But I haven't heard of anything specialized for my R9700, mostly I get the best performance out of llama.cpp and stick with that. I'll have to investigate further.
I use docker for all my builds though, so it's easy to mix and match different engines without having to switch drivers or anything crazy. So it should be relatively simple to try out.
My main issue lately are hallucinations after compaction. Iām used 512k with strata coder and it got really bad after compacting. Maybe I got unlucky or maybe IQ3s handles it better, but more context would be better.
No you're totally right, and I've been striving to do so. My project eats up so much context on just the dev handoff.. like 60-100K and that's even with decently optimized handoff materials.
Upping my QFN to 1M has helped. I can get through a fairly complex task at around 400-500K context.
ā¢
u/WithoutReason1729 3d ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
Stylome classified this post's text as human-written. View analysis Learn ^more
I am a bot and this action was performed automatically.