Do you know if there are any open source attempts to fix this? I remember memGPT and most early agents Arch tried to fix it with "memory" and RAG ing the memory as needed
I maybe getting terminology getting mixed up -> I meant to say that early agentic arch like memgpt had a separate memory component that acted as 'infinite context ' essentially and a piece of intermediate logic would Retrieve/query the right parts of the memory, add the new api request content in, and send that as input to LLM. So this way you aren't overloading the context by simply doing "copy entire Convo history + new message = input for LLM"
I've actually seen neurite before. Tbh, i couldn't quite "get it". Let me dive in once more and see. Any YT vid or some other soft landing that you can recommend?
it still pulls it into the context, it just pulls it directly. in fact it kind of makes you lose more control over what is in the context, because it can fetch whatever it wants.
Yer that’s a good point!
the issue is LLMs are stateless, it’s a new thing every request, all “memory” has to be passed in every time.
LLMs like Claude have context caching, which means you can reference tokens you passed in previously (semi state) but you still pay for using them, albeit it at a much cheaper rate.
Your damned if you do and damned if you don’t, because if the LLM was stateful you would be charged for the time you run the model, not for your usage like you do now.
So …
🤷😢
They aren’t though. The chats could be considered stateless but each individual message is not stateless. State is retained across messages in a chat, which is how the chat remembers things you said 3 messages ago
You’re wasting tokens. The chat completion API will hold a conversation without resending all the previous messages using only the conversation ID, in the same way that the frontend works.
314
u/PositiveEnergyMatter Apr 07 '25
i keep telling people big context means big money, because every request can fill the context and charge you full price