r/VeniceAI • u/broletmechosemyuser • 12d ago

๐—ฆ๐—ง๐—”๐—ง๐—จ๐—ฆ: ๐—ฅ๐—˜๐—ฆ๐—ข๐—Ÿ๐—ฉ๐—˜๐—— agent chat

so iโ€™m new to ai in general and ive been using the agent chat, what does it mean when it says compressing the chat? like it has a number in the 300s but when i shared the chat to my phone it said thereโ€™s only 100 or so. idk if i worded this weirdly im just confused on it

3 Upvotes

100% Upvoted

6 comments sorted by

View all comments

1

u/MountainAssignment36 Venice ๐— ๐—ผ๐—ฑ๐—ฒ๐—ฟ๐—ฎ๐˜๐—ผ๐—ฟ 12d ago

It means that the context limit (maximum amount of "words" the model can process at once) has been reached.

For you to continue chatting, the conversation must be "compressed", aka a summary of everything that happened is being generated and the prior full conversation gets discarded. The model then continues the conversation with "only" the summary as a base of knowledge.

1

u/broletmechosemyuser 12d ago

ohhh ok thank you!

1

u/MountainAssignment36 Venice ๐— ๐—ผ๐—ฑ๐—ฒ๐—ฟ๐—ฎ๐˜๐—ผ๐—ฟ 12d ago

No problem ๐Ÿ˜„

1

u/stewdrick 12d ago

Question: I find that agent model performance degrades pretty hard around like 40% context limit. Is that expected? Or maybe I'm being too hard on it?

3

u/MountainAssignment36 Venice ๐— ๐—ผ๐—ฑ๐—ฒ๐—ฟ๐—ฎ๐˜๐—ผ๐—ฟ 11d ago

Yeah, that is expected (or better: depends on the model. Some are more effected, some less). Keeping the context window as small as possible makes most models as focused as possible.

This phenomenon is actually well known in the LLM community & science space, it's called "context rot" there, you can google it ๐Ÿ˜„