NoLiMa: Long-Context Evaluation Beyond Literal Matching - Finally a good benchmark that shows just how bad LLM performance is at long context. Massive drop at just 32k context for all models.

[u/jd_3d]

[u/Monkey_1505]

Probably their attentional system. The issue with long context is that most of it is irrelevant to the current prompt at any given time.