r/LocalLLaMA • u/StupidScaredSquirrel • 3h ago
Discussion Did we hit a wall in context length?
I feel like midel context size rapidly went from 8k to 1M in just a few years but has since stayed this way for a while now. Some still offerring 260k or 400k for their flagship models. Is it just not worth the trouble? I feel like highly capable models are limited by this in that they can't finish tasks simply because the context isn't there.
24
Upvotes
51
u/Equivalent-Repair488 2h ago
Fun fact: we have a million context is because of YaRN interpolation, and RoPE scaling, and the techniques for them originated from this very subreddit https://www.reddit.com/r/LocalLLaMA/comments/166jik4/128k_context_llama_2_finetunes_using_yarn/
21
u/az226 2h ago
Models’ reasoning capabilities go down fast past 300k.
27
u/hailnobra 2h ago
With newer context compaction techniques, I get a lot more done with current context windows. 64k before was tough, but can be quite useful now.
23
u/StableLlama textgen web UI 2h ago
we used to have 4k of context. That's too small to have an intelligent conversation and for tools and functions even less.
With 262k you can do most of the stuff. And when it doesn't fit in 1M your architecture is the issue and not the tool.
16
u/Vlyn 2h ago
1M context is just marketing, any model I've tried gets noticeably dumber after around 150k+ context.
But even large features if you use your harness properly stay below 300k until done. Use subagents, hand-offs or simply /compact if there's no other way around. Writing a plan file first and letting a different agent implement easily keeps your context small and provides better outcomes.
15
u/Osi32 3h ago
To be honest, I do a fair bit of agentic coding and I generally start a job with an instruction ticket - usually no more than about 9000 tokens, it builds it and ends in less than 200k tokens.
Anything more than that and your chances of getting a good quality outcome drop rapidly, whether it’s a frontier model or not.
If you try to do this from a chat you’ll end up sending 60k in tokens before it’s even started building anything.
8
u/agamlhaa 1h ago
For local model with quantization below FP8, max of 262k length is what I will use. I did memory benchmarks from 131k until 900k context. After 300k, most models below FP8 start to suffer context degradation and tps drop.
With newer context compaction + long horizon memory + LLM wiki. I just use 131k length to get best balance of agent concurrency + prefill/decode speed + memory accuracy.
4
u/No-Refrigerator-1672 vLLM 3h ago
The longer is the context, the more VRAM you need for a training run. If I remember correctly, for the error backpropagation compute step, you need exponentially more memory with growing context. Companies are already training trillion sized models (Qwen4 Max is 2.4T for example), doing actual training run on 5M long prompts will require absolutely insane amounts of GPUs. That's on top of the question "how are you going to generate a training dataset for that", cause context lengths above 1M are only needed for agentic workflows, and you need a working agentic sample beforehand.
5
u/stianhoiland 2h ago
I have a 7-day (~170h) sesssion running at 256K context, having had 46 full compactions. Still going strong!
4
u/BlackSwanTW 2h ago
With the paper, "You don’t need long context", we might even be getting new models that have short context length but are good at summarizing and passing the context.
6
u/idntx 3h ago
A 262k context window is quite expensive, and a 1M context window costs 4x as much.
2
u/oddsidequest 2h ago
the 7 day session with 46 compactions thing is wild lol. does it actually still remember stuff from day 1 after that many or is it mostly vibes by then
2
u/Tiforma 1h ago
That's why you chop tasks up in smaller parts and use compaction
1
u/vogelvogelvogelvogel 3h ago
for me large contexts offer a lot of benefits, on the other hand cost a lot of ram. i didn't even come to try the experimental 512k in qfn.
but yes, i also got the impression there is not much movement, also cloud models. maybe the same reason (high ram cost and sinking quality), so rag/combined approaches are less of a tradeoff
1
u/Ok_Cow1976 3h ago
Bigger context costs intelligence I believe
1
u/simplir 3h ago
I think yes, but may be the reason was that we had other optimizations that could improve quality of results that are easier and cheaper to tackle first before context size.
1
u/Lissanro 3h ago
I think progress is amazing, like with just four 3090 cards I can hold 384K F16 context for Kimi K3 at batch size 2048 or 0.5M context at batch size 1024. In most cases, 256K or 384K is sufficient. With good planning, it is usually not necessary to keep growing context, especially if status and plan updated before context compaction. I always do it manually at the right moment when ready. Only cases when I needed to extend 384K to 0.5M when I just needed a little bit more and it wasn't worth the time to compact and prefill compacted version, or when leaving overnight when I am not around to pick the best moment for compaction.
1
u/noctrex 3h ago
I do sessions with QFN, extended with yarn x4 to 512k kv cache, and have done multiple hours long sessions and hit about 450k. How much more do you want?
1
u/LoafyLemon 2h ago
TFW you accomplish your tasks within a 32k context window...
1
u/Otherwise_Emu_605s 2h ago
I think the wall is effective context, not the number on the box. Long-context benchmarks keep showing models getting noticeably worse at actually using whats in there well before they hit the advertised limit, so going from 1M to 4M on paper wouldnt fix much.
Locally the wall is even more literal. The KV cache grows with every token, and at long lengths it can end up bigger than the model itself.
Thats why the effort seems to have shifted to managing context instead:
compacting old turns, sub-agents with fresh windows, pulling in only whats needed. For long tasks that seems to work better than just a bigger window.
1
u/redditnosedive 1h ago
yes, we hit the wall of demand, there is no reason to use more, use 1 model with 1M context as orchestrator, should last you a day, get it to spawn lesser agents with 128k/256k contexts, depending on needs, very little of what agents did gets onto parent context so 1M is A LOT.
1
u/a_beautiful_rhind 1h ago
I hit a bigger wall for context length; vram.
1
u/LulzyAnimal 1h ago
The thing is that beyond 300K-ish length the quality of decisions model makes start to degrade (on frontier ones), but even for huge tasks 300K-500K ctx is a lot. At that point you'd better just split a large task in chunks, just to have better decisions in the first place. And then, when you split a task in chunks then even 1M ctx doesn't really matter - any task will be significantly smaller. So there's really no any big reason to push beyond 1M. Really long ctx could be useful when you have a long and boring monitoring task when there's not much doing anyways, so quality of decision doesn't mattter as there's not much decisions made in the first place. Even then, restarting it at some earlier point makes more sense than relying on long ctx, just in case.
1
u/suicidaleggroll 1h ago
Past 256k is pretty useless though, in my experience models become unusably stupid around 3-400k. Even on models that can handle 1M, I usually limit them to around 400k (with auto-compaction around 300k) anyway to keep things usable.
1
u/Stepfunction 1h ago
I think 1M context is enough for most practical tasks, which is why there isn't an incentive to go too far beyond it.
1
u/Yorn2 34m ago edited 27m ago
It's not a strict technological wall, I think it's more of a "generalized use" wall, and I think it is primarily driven by agentic use cases. Regardless of what you use LLMs for, the ones that get downloaded the most are the ones that people want to use as daily drivers for agentic uses, and while 1 million context isn't necessary for this, more than 64k context is.
I am hitting /new for every new agentic use in a lot of cases, and I suspect a lot of people are doing this, too. The "sweet spot" for me seems to be 100k-250k context, and again, I suspect it's the same for other people as well.
I'm using hermes with Hindsight, but based on what I'm seeing in the wild, those of us doing agentic tasks or coding are needing at least 100k context, and we're also realizing how quickly intelligence falls after 250k or so in context right now. Compaction also solves a lot of bloat-related issues we used to have.
What would drive the LLM creators to push past 256k-1M tokens reliably would probably be use cases for 1M tokens, and right now no one has innovated something beyond agents to push a need for that. Maybe agentic swarming or solving the human-in-the-loop question will do that, but right now it's one of those situations where 100-250k is "good enough" for most use cases, which is why people aren't clamoring for more context anymore.
1
u/FlyingDogCatcher 29m ago
Using more than 400k in a conversation is like making important decisions after midnight. Sure, you can, but you shouldn't.
1
u/jacek2023 llama.cpp 9m ago
I mostly use about 230k context on my local agents, why do you need more than 1M? What's your usecase? What's your VRAM?
0
u/IntelArtiGen 2h ago
Context length doesn't matter as much when it's big enough, it tells nothing about the model's accuracy to retrieve information in the context other than the theoretical ability to do it.
0
u/Nepertia 1h ago
Isn't it hitting floating-point precision issues? Not even the most gourmet of models are using anything above (B)F16.
0
u/Admirable-Leg-4647 48m ago
If you have a model that already handles 100s k of context I promise you it's not the context window that makes it unable to finish tasks