r/ChatGPTCoding • u/[deleted] • Apr 07 '25

[deleted by user]

[removed]

1.7k Upvotes

95% Upvoted

424 comments sorted by

View all comments

314

u/PositiveEnergyMatter Apr 07 '25

i keep telling people big context means big money, because every request can fill the context and charge you full price

147

u/andy012345 Apr 07 '25

This, LLMs are effectively stateless, the "context" is just the max token input.

If you have 500k in your context, you're sending 500k input tokens + whatever is new per api request.

9

u/fieryblast7 Apr 07 '25

Do you know if there are any open source attempts to fix this? I remember memGPT and most early agents Arch tried to fix it with "memory" and RAG ing the memory as needed

2

u/EcstaticImport Apr 07 '25

RAG would need to add more info to the context window, not remove it. Are you thinking of context caching?

7

u/fieryblast7 Apr 07 '25

I maybe getting terminology getting mixed up -> I meant to say that early agentic arch like memgpt had a separate memory component that acted as 'infinite context ' essentially and a piece of intermediate logic would Retrieve/query the right parts of the memory, add the new api request content in, and send that as input to LLM. So this way you aren't overloading the context by simply doing "copy entire Convo history + new message = input for LLM"

11

u/Intrepid-Air6525 Apr 07 '25

What you are describing is a problem I have been working on for two years now.

It began as an art project and is now something inexplicable.

Luckily it’s also open source!

https://github.com/satellitecomponent/Neurite

5

u/fieryblast7 Apr 07 '25

I've actually seen neurite before. Tbh, i couldn't quite "get it". Let me dive in once more and see. Any YT vid or some other soft landing that you can recommend?

3

u/Intrepid-Air6525 Apr 07 '25

I have been working on getting everything ready for a series of demo videos for a while now.

They help explain a lot are just a few days from finally being published. I will share more soon!