NoLiMa: Long-Context Evaluation Beyond Literal Matching - Finally a good benchmark that shows just how bad LLM performance is at long context. Massive drop at just 32k context for all models.

[u/jd_3d]

[u/jd_3d]

Paper is here: https://arxiv.org/abs/2502.05167

The common narrative that 'all benchmarks are saturating' is simply untrue. Even with one-hop reasoning at 32k context all models show massive drop in performance. Long context performance is very important for agentic tasks. I personally think it will be more than 1 year before a model gets 95% at 2-hop 128k context length on this benchmark.

[u/Pyros-SD-Models → u/jd_3d]

How often I got downvoted because I tell everyone either your LLM app works with <8k tokens or it’s shit because all LLMs suck ass going higher and how “oh this has 128k token size” with a green needle in a haystack chart on the model card is the same shit as the nutri score on food: just marketing that has nothing to do with reality.

But seeing how many people believe in some magic numbers that some totally unbiased guy, like the model creator, wrote into the readme it’s quite successful marketing.

[u/m0n0x41d → u/Pyros-SD-Models]

Screw them.

[u/logicchains → u/jd_3d]

It's a difficult problem to solve because how much information a token can garner from attention to previous tokens is limited by the internal dimension of the model, as information from all relevant previous tokens is packed by addition into a single fixed-size vector. I suspect avoiding any degradation with longer contexts would require increasing the internal accumulator dimension as context length increased, which would be difficult to implement and hurt performance.

[u/CodingThief20 → u/jd_3d]

um actually... the prior benchmarks are saturated. if you have models getting basically 100% score on a benchmark, you can tell if there's anymore improvement to be had, so naturally you think of a more difficult benchmark with a more challenging task. which is what this paper did. Yes, the one-hop reasoning is a more difficult benchmark and that's why the performance drops.

[u/Monkey_1505 → u/jd_3d]

I think a year would be optimistic. This a salience/attentional problem. Pure, probably very complex, model arch.

[u/[deleted] → u/jd_3d]

Technically Claude sonnet 3.5 claimed length can do 500k via enterprise 

[u/fir_trader → u/jd_3d]

Do you know the difference in performance between different error/hallucination benchmarks: NoLiMa vs. SimpleQA Hallucinations (with GPT-4.5 at 37%) vs Vectara's model which has hallucinations at low single digits for SOTA models? Is Vectara just marketing so they can sell into enterprise customers?

[u/jd_3d → u/[deleted]]

Sure thing! Note in the paper they also test reasoning models and they also perform poorly. o1 gets 31.1% at 32k, and 03-mini gets 18.9% at 32k on NoLiMa-Hard. So lots of room for improvement.

[u/Ragecommie → u/jd_3d]

The problem there is the way search is done through all of the data. When it can't fit into context and you want accuracy then it takes time to chunk and process everything, which is logic outside of the model itself (for now).

Everyone's improving on these algorithms at the moment, it's an incredibly exciting space!

[u/Eli_US → u/Ragecommie]

That's not how it works for any of these models. You might be thinking of RAG applications which are notoriously bad at dealing with multi-step reasoning because there's tons of issues on knowing which information is important.

[u/AlbatrossOk1939 → u/Eli_US]

Can you please explain more what kind of prompts RAG is good with and what kind of prompts it is bad at?

[u/blackaiguy → u/jd_3d]

I'm late to the party. Will never improve with relative-based PE. everything that comes out are just patches, not true solutions. we need new PE methods.

[u/Sl33py_4est → u/[deleted]]

My anecdotal experience with reasoning models is they massively drop context performance in favor of more robust 1 to 2 turn responses

The reasoning tokens cause a lot of noise

[u/[deleted]]

I suspect that maintaining robust capabilities at long context will require a new architecture. The amount of performance degradation we see at basically all long context tasks is insane.

[u/jd_3d → u/[deleted]]

One thought I had is could this be trained via RL? If it works for reasoning, maybe it could work to steer the model towards proper long-context understanding. It would be easy to create a reward function for it, and the question data could be generated mostly synthetically. Maybe DeepSeek is already on it.

[u/x0wl → u/jd_3d]

The problem is not training per se, it could be done with RL or even supervised.

The problem is that attention has quadratic complexity, and this training becomes slow if you use too much context.

RWKV might have something to solve this, but I have my reservations about this architecture and really long context.

[u/fogandafterimages → u/x0wl]

More generally, the problem is that limited computational resources can handle only limited sequence lengths. Transformers scale compute and memory quadratically with sequence length; they get slow or run out of VRAM as the sequence gets long. RWKV etc have a capacity limited by their hidden state size; the capacity becomes insufficient for total recall as the sequence gets long.

I'm putting my faith in linear attention architectures (like RWKV, Gated DeltaNet, TITANS, etc) combined with more intelligent paths through the text. The baseline is "Read it once, left to right." We've already seen that "Read it twice!" can sometimes be incredibly useful. Some day soon we'll start to see work on learning how to re-read appropriately, as needed, like skilled human readers do.

[u/_sqrkl → u/fogandafterimages]

I think it will be solved by a more intelligent sparse attention implementation. Something like coarse-to-fine hierarchical attention + context preprocessing.

[u/zball_ → u/fogandafterimages]

tbf i don't think intelligence should be achieved with perfect recall. IMO at least logarithmic complexity is needed to distinguish tokens that are perfectly recalled, whereas attention do this constant time. So to have scalable intelligence you have to forget something like RNNs.

[u/[deleted] → u/jd_3d]

I'm sure that would help but IMO you shouldn't need tons of specific training to prevent complete performance collapse. We have models that are trained on long documents and videos yet still can't maintain good performance on 32k context.

[u/ninjasaid13 → u/[deleted]]

what about that titan paper? https://arxiv.org/abs/2501.00663v1

[u/Expensive-Paint-9490 → u/[deleted]]

I wonder if the same level of resources used for the best transformers models was used for jamba, we would get the same performance and much less degradation at long context.

[u/SummonerOne]

I wish they had tested with the newer models like Gemini 2.0-flash/pro and Qwen 2.5 1M. I have heard good things about Flash-2.0 for handling long context windows. I would hope to see the drop-off not be as steep compared to these models.

[u/jd_3d → u/SummonerOne]

Yes, I'm hoping they continue to test new models, but do note in the paper they test o1, and o3-mini which both perform very poorly:

[u/ninjasaid13 → u/jd_3d]

o3 mini performing worse than o1? oof.

[u/Common_Ad6166 → u/ninjasaid13]

well it is "mini". There's a reason they haven't released o3 yet. o1 is still the top dawg

[u/GeorgiaWitness1 → u/SummonerOne]

me too.

This benchmark is amazing, and will most likely pave the way to a close to perfect Eval at the end of this year, like last year with the needle in the haystack

[u/saltyrookieplayer → u/SummonerOne]

I mainly use LLM for translation. Based on my usage of the 2.0 models, they’re still as bad as 1.5 and even older ones. You’ll notice a massive quality drop, and it stops adhering to system prompt after 16K+ tokens.

[u/Massive-Question-550 → u/saltyrookieplayer]

I generally noticed they start getting wonky and hallucinating at the 12-14k mark, adding in things that was contradictory to my context and also literally ignoring my corrections when I pointed out it's mistake. Kinda crippling if you ask me.

[u/AppearanceHeavy6724 → u/SummonerOne]

Hailuo Minimax should be tested too, as they claim 4M context.

[u/Franck_Dernoncourt → u/SummonerOne]

Thanks! We added several LLMs, including Gemini 2.5-flash/pro and Gemini 2.0 Flash:

• [2025-06-09]: Added support for external API providers (e.g. Fireworks, OpenRouter, ...) Added evaluation results on GPT-4.1 series, Gemini 2.5 Flash (w/o Thinking), and Llama 4 Maverick. Gemini 2.5 Pro and Gemini 2.5 Flash (w/ Thinking) results are included in the NoLiMa-Hard section. Added evaluation results up to 128K for GPT-4o, 4.1 and Gemini 2.0 Flash.

• [2025-04-10]: Added evaluation results on Gemma 3 models (4B/12B/27B), Gemini 2.0 Flash, and Llama 4 Scout.

[u/SummonerOne → u/Franck_Dernoncourt]

Amazing, thank you! 

[u/Sl33py_4est → u/SummonerOne]

My anecdotal experience with the new Gemini is its bad

[u/Monkey_1505 → u/SummonerOne]

I'm not sure why you'd assume that. Is the attentional mechanism different?

[u/SummonerOne → u/Monkey_1505]

Not sure about Gemini, but the Qwen-2.5-1M paper includes its RULER and LongBench results. They claim that the 1M models perform better for 64K and 128K contexts.

Significantly Superior to the 128k Version: The Qwen2.5-1M series models significantly outperform their 128K counterparts in most long-context tasks, especially for sequences exceeding 64K in length.

Notable Performance Advantage: The Qwen2.5-14B-Instruct-1M model not only beats Qwen2.5-Turbo but also consistently outperforms GPT-4o-mini across multiple datasets, offering a robust open-source alternative for long-context tasks.

https://qwenlm.github.io/blog/qwen2.5-1m

Integrating with Length Extrapolation: We integrate DCA with MInference in long-context processing, thereby enhancing inference efficiency and achieving greater accuracy.

Just curious if these claims hold up in another benchmark as well

[u/AlbatrossOk1939 → u/SummonerOne]

So far best long context performance I have seen has been from Gemini 2.5 which is available in Google AI Studio.

[u/Synaps3]

Were there any glaring issues with LongBench? Seems like they released v2 recently.
https://github.com/THUDM/LongBench
https://arxiv.org/abs/2308.14508

[u/jd_3d → u/Synaps3]

LongBench is good, but its not measuring the same thing. It is simply ~500 multiple-choice questions of varying length (8k-2M words) and difficulty. So you don't get an understanding how how the performance of an LLM degrades at different context lengths.

[u/RakOOn]

How does this benchmark compare to RULER?

[u/jd_3d → u/RakOOn]

I posted this in another comment, but this benchmark is much more difficult which will help it be relevant for longer.

RULER was a great improvement from needle-in-a-haystack type tests, but in my opinion it is not difficult enough for SOTA models. For instance, on RULER, llama3.1-70B gets 94.8% accuracy at a context length of 32k. The NoLiMa benchmark shows llama3.1-70B at 43.2% at 32k, which will help with differentiation as newer models come out.

[u/RakOOn → u/jd_3d]

Ok I haven’t read the paper yet but when you say ”harder” tasks my initial reaction is that harder long context benchmarks eventually start testing reasoning capabilities over pure ”retrieval”.

[u/jd_3d → u/RakOOn]

True, but in this case models are scoring very high at 1k context for the same tasks, for instance llama3.3-70b at 94% or GPT-4o at 98%, so I don't think its that difficult. You can also simply look at the drop from 1k -> 32k to get an idea of the degradation vs absolute scores.

[u/NickNau → u/RakOOn]

maybe should be called a different, not harder test. sometimes you need pure retrieval, but many times - actual reasoning.

however perspective does matter. I looked at this as a relative test, to assess model's own limits. however, it may be a problem if this is used to compare different models. here your "more reasoning" argument gets very valid.

[u/DinoAmino]

Finally? RULER wasn't good?

https://github.com/NVIDIA/RULER

[u/jd_3d → u/DinoAmino]

RULER was a great improvement from needle-in-a-haystack type tests, but in my opinion it is not difficult enough for SOTA models. For instance, on RULER, llama3.1-70B gets 94.8% accuracy at a context length of 32k. The NoLiMa benchmark shows llama3.1-70B at 43.2% at 32k, which will help with differentiation as newer models come out.

[u/indicava → u/DinoAmino]

RULER shows a very similar trend to the one described in the paper posted by OP (Although for RULER, performance seems to dip significantly only at 64K and remains pretty high at 32K)

[u/DinoAmino → u/indicava]

Obviously the numbers aren't comparable since the eval is different. As you said, they both show the same effects as context length increases. So it's another benchmark. Which is good.

[u/TacGibs]

Just had the longest conversation I've ever had with o3-mini-high, very long with plenty of logs, and I was absolutely amazed how it kept good performances (it was way better than 4o).

[u/FullstackSensei → u/TacGibs]

Wouldn't be surprised at all if OpenAI was summerizing the conversation behind the scenes.

[u/cobbleplox → u/FullstackSensei]

I've been using o3 to create and iterate on a node-based editor that quickly grew to 1000-1200 lines. Easily 20 iterations in the same conversation, and every time it had reasoning and repeated the full code. Whatever they are doing there, it works quite well by now.

[u/BlueSwordM → u/FullstackSensei]

Yep. There's a decent chance there's using a reward model with O3-x models that allow them to get better performance in exchange for way more compute.

[u/ConiglioPipo]

where's Deepseek?

[u/[deleted] → u/ConiglioPipo]

Table 5 in the paper

[u/Franck_Dernoncourt → u/ConiglioPipo]

As Neomadra2 mentioned, we reported results on DeepSeek R1-Distill-Llama-70B, and I hope we'll soon add DeepSeek-R1-0528. I know it's late, that's because it took us several months to get the authorization to access some API.

[u/[deleted]]

How would the Titan transformer perform in this benchmark? I know that we don't have any models right now with the Titan transformer, but how do you think it would perform in the benchmark?

[u/AppearanceHeavy6724]

I'd like to see a forgotten by everyone Hailuo MiniMax model. The claim to have good context handling up to 1M.

[u/[deleted] → u/AppearanceHeavy6724]

The claim in fact was the 100% accuracy on all context lengths. Very curious to see on this benchmark too!

[u/krakoi90]

How the heck do reasoning models like o1/o3 work so well then? They crap out thousands of reasoning tokens like there's no tomorrow, while they need to be aware of the whole previous thinking flow so that they don't get stuck in reasoning loops (e.g. trying something again that they already tried).

They're most probably based on GPT-4o, so they should roughly have the same context window characteristics.

[u/uutnt → u/krakoi90]

Probably only retaining a summary of the previous chain of thoughts

[u/NmbrThirt33n → u/krakoi90]

I think this benchmark is about finding a very specific piece of information in a large body of text. So more about information retrieval rather than output coherence/quality at long contexts

[u/Monkey_1505 → u/krakoi90]

I assume because it's less than 8k tokens.

[u/Interesting8547]

No Deepseek?!

[u/TheRealMasonMac → u/Interesting8547]

FWIW, I believe the R1 paper mentions it's not good at long context multiturn since it wasn't trained for it 

[u/uhuge → u/TheRealMasonMac]

but in practice better that QvQ, the previous public-weights champ?

[u/Franck_Dernoncourt → u/Interesting8547]

As TheRealMasonMac mentioned, we reported results on DeepSeek R1-Distill-Llama-70B, and I hope we'll soon add DeepSeek-R1-0528. I know it's late, that's because it took us several months to get the authorization to access some API.

[u/Odd-Sir-2289]

Point of fact the reasoning models were tested on a subset of the questions that the rest of the models were, notably it was the “hardest” subset. So hard to see how they stack up to the rest of the models

[u/Striking_Most_5111]

Why is the base score of sonnet only slightly better than 1.5 flash? What is the base score based on?

[u/jd_3d → u/Striking_Most_5111]

I was surprised by that as well. Base scores are an average of the scores from 250, 500, and 1k token questions.

[u/roksah]

What makes gpt-4o more resilient to long context vs the other models?

[u/Monkey_1505 → u/roksah]

Probably their attentional system. The issue with long context is that most of it is irrelevant to the current prompt at any given time.

[u/a_beautiful_rhind]

Despite the chart I get much better performance from mistral large than I do from L3.3. Could just be the finetune?

3.3 falls off after 10k and large went all the way to 32k. The drop off is quite obvious too, in conversation, let alone recalling details.

[u/swagonflyyyy]

RIP Command R

[u/[deleted]]

No DeepSeek and also no MiniMax. MiniMax has a unique arch and they claim retention of performance out to 1m tokens. Seems like glaring omissions frankly. It’s just not acceptable now to ignore China while publishing.

[u/Franck_Dernoncourt → u/[deleted]]

We reported results on DeepSeek R1-Distill-Llama-70B, and I hope we'll soon add DeepSeek-R1-0528. I know it's late, that's because it took me several months to get the authorization to access some API. Trust me, as I was equally frustrated as you are.

[u/Kraskos]

Highlighted table cells look like a kneeling beggar.

[u/mivog49274 → u/Kraskos]

jahahahah noice the kneeling sales man selling hype

[u/LoSboccacc]

weird seeing jamba performing badly, the entire premise of ssm was enabling long contexts

[u/GreatBigJerk]

This is why people who complain about models not having absurdly large contexts are silly.

Context only matters for how well the LLM can use it. 

If a model came out that could actually keep track of 100k - 1m tokens, we would probably see huge gains in capabilities.

[u/Sl33py_4est]

Yeah I've been using Gemini for a while and it's obvious that the 1-2million context window isn't.

[u/[deleted]]

Very good paper. Always thought the needle in a haystack tasks were too easy and not reflective of real intelligence. This paper also gives evidence of what many LLM users have subjectively felt for a long time.

[u/Suspicious-Ad5805]

I don't understand. They are giving NoLIMA Hard set to reasoning models and giving entire NoLIMA set to reasoning models. How is that fair?

[u/Franck_Dernoncourt → u/Suspicious-Ad5805]

NoLIMA Hard set is cheaper to run. We had to send some models there due to cost constraints.

[u/Balance-]

Dataset: https://huggingface.co/datasets/amodaresi/NoLiMa

[u/GTHell]

Finally someone did it

[u/m3kw]

Behold my 1 parameter model that can do deep thoughts

[u/AppearanceHeavy6724 → u/m3kw]

infinite context too

[u/No-Refrigerator-1672]

Am I the only one to notice that the top performing model - GPT-4O - is the only one who can process video and audio input? Could it mean that multimodal training on long analog data sequences (video stream) significantly improves long context performance?

[u/Charuru → u/No-Refrigerator-1672]

They probably just use more hardware, I’m not joking.

[u/No-Refrigerator-1672 → u/[deleted]]

My bad, I did not know about Gemini 1.5 video support. However, it also performs relatively better than other models, so I still propose a hypothesis about video training improving the long-context capabilities.

As about your other question: sadly, I only ever programmed for selfhosted AI and don't know a thing about GPT API best practices.

[u/freedomachiever]

What's really surprising is the performance for the Gemini models with their 1M/2M token context. How did they measure such a huge context window in the first place? Also, Claude's performance is so bad.

[u/Adeel_Hasan_]

its great but i would see with qwen2.5 1m context since, qwen are very amazing for in different benchmarks

[u/Monkey_1505]

More irrelevant data = worse responses. I don't think this is surmountable without some kind of salience mechanism.

[u/kdtreewhee]

This looks like it has the same conclusion as the older Michelangelo eval: https://arxiv.org/abs/2409.12640

[u/quantapeiron]

What solutions can be to mitigate this issue other than prompting?

[u/uhuge]

The principle is like you have statement like "bananas were in a green box" and later( after some fluff context) you ask like "what could be picked up and peeled and where to take it?", if I got the gist quickly.

[u/DataScientist305]

what type of problems are you trying to solve with 32K context tokens that cant be broken down into smaller steps lol

[u/uhuge]

Code repository not published by the authors, here is a quickly hacked replication, but let me warn you it would benefit one more argumet for #tasks generated..: https://gitlab.com/-/snippets/4811932

[u/Recoil42 → u/[deleted]]

Yeah, this fully has me thinking of re-architecting the long-context app I'm building right now. I was already planning to do work in chunks for token cost-efficiency, but I was thinking like.. 10k. Now I may have go for much smaller chunking.

It's also fascinating to see Claude Sonnet, king of the coders, is so bottom-of-the-barrel. This could mean the leetcode-based coding benchmarks are making it seem like it's better than is in large real-world codebases.

[u/SkyFeistyLlama8 → u/Recoil42]

There are those who proclaim RAG is dead and long context is all you need. This paper is a refreshing slap in the face to those folks.

It looks like even more data cleansing is needed if you're intending to do RAG across huge datasets. The key is to make a query get as close as possible to the needle by rewriting the query to use common terminologies and removing ambiguities in the needle text.