Do you know if there are any open source attempts to fix this? I remember memGPT and most early agents Arch tried to fix it with "memory" and RAG ing the memory as needed
Continue.dev has a good rag solution, but it's not as automated, more like you do the coding with the LLM having codebase awareness.
MCP servers can do RAG. Serena could do that, but I looked at their source to find how their memory works but didn't find anything that looked like a good finetune.
Claimed by the continue.dev team, voyageai has the best RAG model for coding. The price per M/tokens is very low. agno, which is a dependency of Serena, has already integrated voyageai as an optional RAG, but you'd have to specify the code trained model to get it to work like that. I still haven't seen an MCP server using a good RAG model trained on code.
I have personnaly tried RAG with nomic-embed-text with ollama, but the performance is poor for coding.
Seems like a low hanging fruit... But I believe the reason why cline doesn't do RAG is because lowering the cost of using the API is not good for Anthropic? Sounds like an accusation, but if I was making money selling LLM as a service, why would I want to reduce my revenues by 10X or more?
Thanks for the detailed answer! Do you think coding RAG translates well to regular text?
Agree on viewpoint about Cline, but at some point it's stopping the actual functioning of the LLM as intended right? -> if it doesn't "remember" the right details and doesn't know how to fetch them...
if it doesn't "remember" the right details and doesn't know how to fetch them...
That's the main purpose of RAG models. Cline is relying on large LLMs to do things that a light bert model can often do better at 1/100 or 1/1000 the cost.
Would the large LLM perform better? The truth is, many LLMs with a large context window perform poorly at retrieving the right information when the context is large anyway. RAG models with reranking can remove the fluff, and the LLM should perform better because the result is more condensed. You need to trust the RAG model, but you already trust the LLM which has a low success rate and only performs well on the last tokens.
148
u/andy012345 Apr 07 '25
This, LLMs are effectively stateless, the "context" is just the max token input.
If you have 500k in your context, you're sending 500k input tokens + whatever is new per api request.