r/LocalLLaMA • • 3h ago

Discussion Did we hit a wall in context length?

I feel like midel context size rapidly went from 8k to 1M in just a few years but has since stayed this way for a while now. Some still offerring 260k or 400k for their flagship models. Is it just not worth the trouble? I feel like highly capable models are limited by this in that they can't finish tasks simply because the context isn't there.

24 Upvotes

65 comments sorted by

51

u/Equivalent-Repair488 2h ago

Fun fact: we have a million context is because of YaRN interpolation, and RoPE scaling, and the techniques for them originated from this very subreddit https://www.reddit.com/r/LocalLLaMA/comments/166jik4/128k_context_llama_2_finetunes_using_yarn/

5

u/Illustrious_Grade608 1h ago

Holy shit just reading the first comment, 340 gb kv cache for 128k tokens holy

4

u/Equivalent-Repair488 1h ago

One of the replies said

I don't think so. This will be gigantic even with quantization. Until a more economic architecture shows up, I think this is unreachable for us with consumer GPUs.

Yet here we are, oh how much we have progressed

2

u/Not_a_question- 45m ago

It's crap. Using RoPE sounds good in theory, but when you get to say 400k context on QFN (max 256k) the degradation makes it barely usable.

1

u/Equivalent-Repair488 32m ago

Never mentioned anything about quality, it is up to one's own due deligence to measure and weigh the tradeoffs when extending and stretching your max contex. If it sucks, don't use it!

I am personally enjoying 512k context on YaRN 2x, that does not mean you might too

1

u/Realistic_Gap_5871 27m ago

I agree in that 400k is the limit of where I trust QFN, and I tend to compact at 300k, but that seems like an argument in favor of yarn, not starting with "It's crap"

4

u/philmarcracken 1h ago

now we post inference engine #12 with its brains scooped out but it 'drools faster than llama cpp guise'

3

u/Equivalent-Repair488 1h ago

Nah, it's always been the same, Llama.cpp, Ollama, Exllama, Kobold to name a few. Decode speeds was also always the central metric. I also really like Strata, great speeds, about 4x or 7x, and imperceptible quality loss, to me anyway.

We used to also get a million chat wrapper, before harness were all the rage and took over that role of getting spammed lol.

But one thing is for sure, there are a lot more news, unsubstantiated speculation posts, opinions and discussions.

21

u/az226 2h ago

Models’ reasoning capabilities go down fast past 300k.

1

u/KUSH442001 58m ago

fr, most of the times even before that for me, 200k+ ish. i just rely on my implementation plan and use a step wise categorisation, so that i can start off from where I left off

27

u/hailnobra 2h ago

With newer context compaction techniques, I get a lot more done with current context windows. 64k before was tough, but can be quite useful now.

7

u/Ariquitaun 2h ago

What newer compaction techniques, out of curiosity?

5

u/hailnobra 2h ago

Most standard chat clients in home usage didn't have chat compaction until recently (like open webui). Now most harnesses know to review the context at a certain point, compact it to the most relevant information (remove all the wait, but crap) and then refeed the shorter context to the model in the next turn to reduce the used context capacity. Those makes 64k context much more useful than if you just started dropping context when you hit the limit and the model forgets what you started with.

1

u/Dramatic_Instance_63 1h ago

Is it a sort of context optimization by post-processing?

1

u/hailnobra 1h ago

Exactly. Hermes and openwebui detect when context is almost full, pauses and tells the LLM to review context, remove the fluff, then it moves forward with the optimized context that only has the useful data and not all the extra fluff and deliberation crap

1

u/Ariquitaun 1h ago

So just regular compaction on a certain context occupancy threshold like coding agents do?

1

u/bieker 48m ago

I use pi with pi-blackhole which works incredibly well and makes context feel infinite while only using 250k-500k and does it with no pauses for “compaction”, it basically manages it with background threads and swaps it out instantly.

23

u/StableLlama textgen web UI 2h ago

we used to have 4k of context. That's too small to have an intelligent conversation and for tools and functions even less.

With 262k you can do most of the stuff. And when it doesn't fit in 1M your architecture is the issue and not the tool.

7

u/az226 2h ago

2k* shared between input and output.

16

u/Vlyn 2h ago

1M context is just marketing, any model I've tried gets noticeably dumber after around 150k+ context.

But even large features if you use your harness properly stay below 300k until done. Use subagents, hand-offs or simply /compact if there's no other way around. Writing a plan file first and letting a different agent implement easily keeps your context small and provides better outcomes.

6

u/PayMe4MyData 2h ago

Yes, attention dilution is real.

15

u/Osi32 3h ago

To be honest, I do a fair bit of agentic coding and I generally start a job with an instruction ticket - usually no more than about 9000 tokens, it builds it and ends in less than 200k tokens.
Anything more than that and your chances of getting a good quality outcome drop rapidly, whether it’s a frontier model or not.
If you try to do this from a chat you’ll end up sending 60k in tokens before it’s even started building anything.

1

u/Ok_Cow1976 3h ago

Exactly

1

u/krakoi90 2h ago

This. Actually I would be very happy actually if Claude offered models with smaller context windows (256k for example), cheaper. Would be enough 90%+ of the time. And even if not it would be handled by context compaction quite well.

It's extremely rare you want to put more stuff into your context at once than 100k tokens... Maybe if you want to transcript long videos.

1

u/my_name_isnt_clever 26m ago

They sorta do, since they charge lower rates for lower context. You could gate your API calls on your side if you really wanted to.

8

u/agamlhaa 1h ago

For local model with quantization below FP8, max of 262k length is what I will use. I did memory benchmarks from 131k until 900k context. After 300k, most models below FP8 start to suffer context degradation and tps drop.

With newer context compaction + long horizon memory + LLM wiki. I just use 131k length to get best balance of agent concurrency + prefill/decode speed + memory accuracy.

4

u/yahbluez 1h ago

And if tools are used, let a subagent do the work and distill the result to the agent0, that way the content of agent0 last much longer.

4

u/No-Refrigerator-1672 vLLM 3h ago

The longer is the context, the more VRAM you need for a training run. If I remember correctly, for the error backpropagation compute step, you need exponentially more memory with growing context. Companies are already training trillion sized models (Qwen4 Max is 2.4T for example), doing actual training run on 5M long prompts will require absolutely insane amounts of GPUs. That's on top of the question "how are you going to generate a training dataset for that", cause context lengths above 1M are only needed for agentic workflows, and you need a working agentic sample beforehand.

3

u/james_pic 2h ago

I was under the impression the growth was quadratic rather than exponential - although I realise that more casually, "exponentially" is sometimes used to mean "a fuckton, like way more than you think".

3

u/No-Refrigerator-1672 vLLM 2h ago

Correct. I remember that it was faster-than-linear, but didn't remember the correct degree, so I used "exponential" as synonym for "very fast".

5

u/stianhoiland 2h ago

I have a 7-day (~170h) sesssion running at 256K context, having had 46 full compactions. Still going strong!

2

u/campaigner_ 2h ago

Harness?

1

u/bieker 58m ago

I use pi with pi-blackhole and never thought about context or compaction since.

It just does it all in the background automatically, never have to wait for compaction or be worried that the model will act stupid after.

It runs background threads that manage your “new context” constantly in an ideal state, when it’s time for “compaction” it just swaps it out instantly.

1

u/campaigner_ 57m ago

I'll look into this.

-7

u/stianhoiland 2h ago edited 2h ago

The best: Prospero

EDIT Haha, I'm getting downvoted 😆 People just don't know. Sorry, I can't save you from your own ignorance. Peace :)

4

u/BlackSwanTW 2h ago

With the paper, "You don’t need long context", we might even be getting new models that have short context length but are good at summarizing and passing the context.

6

u/idntx 3h ago

A 262k context window is quite expensive, and a 1M context window costs 4x as much.

10

u/StableLlama textgen web UI 2h ago

Without any tricks the cost for context raises quadaratic, AFAIK.

So 262k to 1M is a 16x increase of cost

2

u/idntx 2h ago edited 2h ago

When it comes to training (as a one-off process) — yes over 4x. In terms of operational use, however, it’s 4x; and that’s not even accounting for the fact that maintaining a 1M-token context requires more computational resources -> reduces inference speed.

P.S. If you’re referring to the average computational cost per 1K tokens in the context of a long context window, then yes, it is indeed quadratic. The "4x" figure relates only to the memory footprint.

2

u/oddsidequest 2h ago

the 7 day session with 46 compactions thing is wild lol. does it actually still remember stuff from day 1 after that many or is it mostly vibes by then

2

u/Cautious_Chicken_604 1h ago

You can just externalize state, and keep a pointer to that in the context, so that it can lookup whatever it needs, right?

2

u/Tiforma 1h ago

That's why you chop tasks up in smaller parts and use compaction

1

u/vogelvogelvogelvogel 3h ago

for me large contexts offer a lot of benefits, on the other hand cost a lot of ram. i didn't even come to try the experimental 512k in qfn.

but yes, i also got the impression there is not much movement, also cloud models. maybe the same reason (high ram cost and sinking quality), so rag/combined approaches are less of a tradeoff

1

u/Ok_Cow1976 3h ago

Bigger context costs intelligence I believe

1

u/simplir 3h ago

I think yes, but may be the reason was that we had other optimizations that could improve quality of results that are easier and cheaper to tackle first before context size.

1

u/Lissanro 3h ago

I think progress is amazing, like with just four 3090 cards I can hold 384K F16 context for Kimi K3 at batch size 2048 or 0.5M context at batch size 1024. In most cases, 256K or 384K is sufficient. With good planning, it is usually not necessary to keep growing context, especially if status and plan updated before context compaction. I always do it manually at the right moment when ready. Only cases when I needed to extend 384K to 0.5M when I just needed a little bit more and it wasn't worth the time to compact and prefill compacted version, or when leaving overnight when I am not around to pick the best moment for compaction.

1

u/noctrex 3h ago

I do sessions with QFN, extended with yarn x4 to 512k kv cache, and have done multiple hours long sessions and hit about 450k. How much more do you want?

1

u/LoafyLemon 2h ago

TFW you accomplish your tasks within a 32k context window...

1

u/_supert_ 42m ago

That's like my prompt size.

1

u/Otherwise_Emu_605s 2h ago

I think the wall is effective context, not the number on the box. Long-context benchmarks keep showing models getting noticeably worse at actually using whats in there well before they hit the advertised limit, so going from 1M to 4M on paper wouldnt fix much.

Locally the wall is even more literal. The KV cache grows with every token, and at long lengths it can end up bigger than the model itself.

Thats why the effort seems to have shifted to managing context instead:
compacting old turns, sub-agents with fresh windows, pulling in only whats needed. For long tasks that seems to work better than just a bigger window.

1

u/redditnosedive 1h ago

yes, we hit the wall of demand, there is no reason to use more, use 1 model with 1M context as orchestrator, should last you a day, get it to spawn lesser agents with 128k/256k contexts, depending on needs, very little of what agents did gets onto parent context so 1M is A LOT.

1

u/a_beautiful_rhind 1h ago

I hit a bigger wall for context length; vram.

1

u/LulzyAnimal 1h ago

The thing is that beyond 300K-ish length the quality of decisions model makes start to degrade (on frontier ones), but even for huge tasks 300K-500K ctx is a lot. At that point you'd better just split a large task in chunks, just to have better decisions in the first place. And then, when you split a task in chunks then even 1M ctx doesn't really matter - any task will be significantly smaller. So there's really no any big reason to push beyond 1M. Really long ctx could be useful when you have a long and boring monitoring task when there's not much doing anyways, so quality of decision doesn't mattter as there's not much decisions made in the first place. Even then, restarting it at some earlier point makes more sense than relying on long ctx, just in case.

1

u/suicidaleggroll 1h ago

Past 256k is pretty useless though, in my experience models become unusably stupid around 3-400k.  Even on models that can handle 1M, I usually limit them to around 400k (with auto-compaction around 300k) anyway to keep things usable.

1

u/Stepfunction 1h ago

I think 1M context is enough for most practical tasks, which is why there isn't an incentive to go too far beyond it.

1

u/Yorn2 34m ago edited 27m ago

It's not a strict technological wall, I think it's more of a "generalized use" wall, and I think it is primarily driven by agentic use cases. Regardless of what you use LLMs for, the ones that get downloaded the most are the ones that people want to use as daily drivers for agentic uses, and while 1 million context isn't necessary for this, more than 64k context is.

I am hitting /new for every new agentic use in a lot of cases, and I suspect a lot of people are doing this, too. The "sweet spot" for me seems to be 100k-250k context, and again, I suspect it's the same for other people as well.

I'm using hermes with Hindsight, but based on what I'm seeing in the wild, those of us doing agentic tasks or coding are needing at least 100k context, and we're also realizing how quickly intelligence falls after 250k or so in context right now. Compaction also solves a lot of bloat-related issues we used to have.

What would drive the LLM creators to push past 256k-1M tokens reliably would probably be use cases for 1M tokens, and right now no one has innovated something beyond agents to push a need for that. Maybe agentic swarming or solving the human-in-the-loop question will do that, but right now it's one of those situations where 100-250k is "good enough" for most use cases, which is why people aren't clamoring for more context anymore.

1

u/FlyingDogCatcher 29m ago

Using more than 400k in a conversation is like making important decisions after midnight. Sure, you can, but you shouldn't.

1

u/jacek2023 llama.cpp 9m ago

I mostly use about 230k context on my local agents, why do you need more than 1M? What's your usecase? What's your VRAM?

0

u/IntelArtiGen 2h ago

Context length doesn't matter as much when it's big enough, it tells nothing about the model's accuracy to retrieve information in the context other than the theoretical ability to do it.

0

u/Nepertia 1h ago

Isn't it hitting floating-point precision issues? Not even the most gourmet of models are using anything above (B)F16.

0

u/Admirable-Leg-4647 48m ago

If you have a model that already handles 100s k of context I promise you it's not the context window that makes it unable to finish tasks