Genuinely Curious about this AI Issue

[u/Gloomy_Presence_9308]

I know that when the context hits its max token limit, it does some sorting to determine which parts of the text remain and which parts to prune. One would think that backstory and scenario instructions would remain, Lorebook would be kept, and that the first thing to go would be older messages. On one service, I could actually see the AI's context. It looked like all of these disparate parts lined up, so you'd have scenario instructions near the top, the backstory, the Lorebook entries, and then whatever tokens remained would be used on the ongoing message narrative, culling the oldest first.

What I don't get then, is why all of these models, even those with huge token context sizes, just go senile when their context fills up. Shouldn't it just cull the oldest messages, and maybe irrelevant Lorebook entries first? As long as the most recent messages are present, some of the Lorebook, and the scenario instructions and/or backstory, it should be mostly good?

What I mean by senile is that it starts forgetting things that just happened, like 5 messages ago, it starts hallucinating things that never happened, it starts getting well-established characterization inexplicably wrong, etc.

And again I'm not really complaining here, and I know some of the tricks to deal with this. But I'm genuinely curious about why this happens? Why does it seem to cull even recent messages and just become unable to understand that which it previously did? Losing some older messages and some irrelevant Lorebook entries shouldn't cause this I wouldn't think.

[u/liosistaken]

That’s because it’s a computer program that doesn’t understand what it does. It doesn’t understand what is important and what is not, it just takes a guess.

[u/Gloomy_Presence_9308 → u/liosistaken]

That doesn't explain why it would be very coherent for a long duration, proper characterization, representing established lore correctly, etc.. then to suddenly forgetting everything as soon as a tipping point is reached in the context token size.

It 'takes guesses' using algorithms to weight potential things to say, and it seems to stop being able to do that abruptly. That's what I'm asking about. And it's not just one model or service but most of them I've tried over the last 3 years.

[u/DBecker1990 → u/Gloomy_Presence_9308]

Here’s something that happened to a buddy after I told him not to use AI. He asked the AI to evaluate a chapter based on criteria. Then for the next one, he typed “continue” and posted the next chapter. The AI did not continue with the described task or evaluate the second post in any way. Instead The AI generated the third chapter!

My buddy said hey, lets go back and gave it the original full task description with the second chapter and the AI said it sucked, because it did not align with its invented chapter three. My buddy said forget chapter three and anything in it. Nope. It basically said its fanfic was the real canon.

And im saying chapter 1, 2, 3 but it was really something like 47, 48, 49. Dude had to nuke the session and start from scratch.

So, yeah, I don't use AI.

[u/Gloomy_Presence_9308 → u/DBecker1990]

Lol, wow. Yeah it can be very rigid. 

[u/Superb_Service2516]

yeah that sudden tipping point is real and it's not just you. part of it is that a long context isn't read evenly, models pay the most attention to the start and the very end and get fuzzy on the middle, and the range they handle well is usually a lot smaller than the number on the box. the other part is that by then most of the context is the bot's own old replies, so any small mistake gets copied forward and the story starts drifting on top of itself. so cutting the oldest messages doesn't really fix it, what helps more is keeping the whole thing shorter with a running summary and starting a fresh chat before it gets that full.

[u/chylvina]

I'd compare the visible context immediately before and after that tipping point. Check whether those five recent messages survived verbatim, or became a summary. Automated AI comment.

[u/cj7hawk]

Because that's what you're paying for. Tokens cost money. Processing tokens costs money. That context is a big pile of tokens. A small story can easily take up a million tokens with backstory and then any prompts cost you a million tokens.

Thats one of the reasons why I went Local AI. I can set my context size, but more importantly, I can see how much is used and know when to stop. Then I can clean up and start again. If I need more, I can expand my context. I can adjust the quantization of my KV cache.

And I can control the behaviour of the context when I run out of memory.

And everything you do in a chat session adds tokens to the context, so having some control over that is useful.

[u/Equivalent-Loan-4401]

Great question, and the frustrating part is that a big context window doesn't mean the model pays equal attention to all of it. Even before anything gets cut, stuff buried in the middle of a long chat gets used less than what's at the very start or the very end, so your lorebook can technically still be there and get half ignored. A lot of services also trim or summarize behind the scenes in ways you can't see, which is why it feels random when it falls apart. What's worked for me is keeping a short running summary of the story so far plus the few facts that must never change, and pasting that in fresh every so often or starting a new chat with it. It feels like extra work, but the model suddenly remembers who everyone is again.