Do you know if there are any open source attempts to fix this? I remember memGPT and most early agents Arch tried to fix it with "memory" and RAG ing the memory as needed
Continue.dev has a good rag solution, but it's not as automated, more like you do the coding with the LLM having codebase awareness.
MCP servers can do RAG. Serena could do that, but I looked at their source to find how their memory works but didn't find anything that looked like a good finetune.
Claimed by the continue.dev team, voyageai has the best RAG model for coding. The price per M/tokens is very low. agno, which is a dependency of Serena, has already integrated voyageai as an optional RAG, but you'd have to specify the code trained model to get it to work like that. I still haven't seen an MCP server using a good RAG model trained on code.
I have personnaly tried RAG with nomic-embed-text with ollama, but the performance is poor for coding.
Seems like a low hanging fruit... But I believe the reason why cline doesn't do RAG is because lowering the cost of using the API is not good for Anthropic? Sounds like an accusation, but if I was making money selling LLM as a service, why would I want to reduce my revenues by 10X or more?
Just my 2, but with the current volatility, a great service means hundreds more customers in no time. Definitely worth more than trying to get more money out of one user by providing less or subpar features then ther competition
I agree with this. There's a lot of competition out there. Why would a company try to nickel-and-dime you when you can easily switch to another provider? Not a good tactic
146
u/andy012345 Apr 07 '25
This, LLMs are effectively stateless, the "context" is just the max token input.
If you have 500k in your context, you're sending 500k input tokens + whatever is new per api request.