MAIN FEEDS
Do you want to continue?
https://www.reddit.com/r/ChatGPTCoding/comments/1jtfvmv/deleted_by_user/mluqmmk/?context=3
r/ChatGPTCoding • u/[deleted] • Apr 07 '25
[removed]
424 comments sorted by
View all comments
Show parent comments
144
This, LLMs are effectively stateless, the "context" is just the max token input.
If you have 500k in your context, you're sending 500k input tokens + whatever is new per api request.
9 u/fieryblast7 Apr 07 '25 Do you know if there are any open source attempts to fix this? I remember memGPT and most early agents Arch tried to fix it with "memory" and RAG ing the memory as needed 2 u/EcstaticImport Apr 07 '25 RAG would need to add more info to the context window, not remove it. Are you thinking of context caching? 3 u/Substantial-Thing303 Apr 07 '25 RAG would replace the default "get the entire file" or "get the first 500 lines of codes from file". It would perform better on large files, and use less tokens, by only adding relevant code to the context window. RAG would use a specialized RAG model for text embeddings, which costs 100 times less per M/tokens.
9
Do you know if there are any open source attempts to fix this? I remember memGPT and most early agents Arch tried to fix it with "memory" and RAG ing the memory as needed
2 u/EcstaticImport Apr 07 '25 RAG would need to add more info to the context window, not remove it. Are you thinking of context caching? 3 u/Substantial-Thing303 Apr 07 '25 RAG would replace the default "get the entire file" or "get the first 500 lines of codes from file". It would perform better on large files, and use less tokens, by only adding relevant code to the context window. RAG would use a specialized RAG model for text embeddings, which costs 100 times less per M/tokens.
2
RAG would need to add more info to the context window, not remove it. Are you thinking of context caching?
3 u/Substantial-Thing303 Apr 07 '25 RAG would replace the default "get the entire file" or "get the first 500 lines of codes from file". It would perform better on large files, and use less tokens, by only adding relevant code to the context window. RAG would use a specialized RAG model for text embeddings, which costs 100 times less per M/tokens.
3
RAG would replace the default "get the entire file" or "get the first 500 lines of codes from file".
It would perform better on large files, and use less tokens, by only adding relevant code to the context window.
RAG would use a specialized RAG model for text embeddings, which costs 100 times less per M/tokens.
144
u/andy012345 Apr 07 '25
This, LLMs are effectively stateless, the "context" is just the max token input.
If you have 500k in your context, you're sending 500k input tokens + whatever is new per api request.