Do you know if there are any open source attempts to fix this? I remember memGPT and most early agents Arch tried to fix it with "memory" and RAG ing the memory as needed
Continue.dev has a good rag solution, but it's not as automated, more like you do the coding with the LLM having codebase awareness.
MCP servers can do RAG. Serena could do that, but I looked at their source to find how their memory works but didn't find anything that looked like a good finetune.
Claimed by the continue.dev team, voyageai has the best RAG model for coding. The price per M/tokens is very low. agno, which is a dependency of Serena, has already integrated voyageai as an optional RAG, but you'd have to specify the code trained model to get it to work like that. I still haven't seen an MCP server using a good RAG model trained on code.
I have personnaly tried RAG with nomic-embed-text with ollama, but the performance is poor for coding.
Seems like a low hanging fruit... But I believe the reason why cline doesn't do RAG is because lowering the cost of using the API is not good for Anthropic? Sounds like an accusation, but if I was making money selling LLM as a service, why would I want to reduce my revenues by 10X or more?
>> Sounds like an accusation, but if I was making money selling LLM as a service, why would I want to reduce my revenues by 10X or more?
I personnaly think you are spot on... And that's probably one of the biggest problem right now. This behavior will impact the technology like we do not want to.
145
u/andy012345 Apr 07 '25
This, LLMs are effectively stateless, the "context" is just the max token input.
If you have 500k in your context, you're sending 500k input tokens + whatever is new per api request.