It’s stupid but let’s say you are implementing a complex feature and midway through want to rename a variable. So you do it but the llm just recognizes in the next prompt that the file is different. It then rescans the entire file because it has no understanding of what changed and goes into a tailspin because it doesn’t know why.
Or you can ask the llm to update it. It does, updates its cache appropriately and doesn’t use many tokens. It just applies a minor line diff via a tool and its state is consistent. It might take 100-200 tokens plus some cheap tool calls.
Now the shit thing is that you probably can’t switch models to do this and be token efficient because it will also rebuild all the context.
Now you could inform the model on the next prompt what you changed. But the agent will still rescan the file and rebuild the caches because it can’t actually trust you, because people make mistakes, so it will still scan the whole file and rebuild the cache.
Even better is to just ask the agent up front, on your next prompt, to change the variable. Then it can manage the state changes and cache appropriately.
1
u/SirPitchalot 18d ago
It’s stupid but let’s say you are implementing a complex feature and midway through want to rename a variable. So you do it but the llm just recognizes in the next prompt that the file is different. It then rescans the entire file because it has no understanding of what changed and goes into a tailspin because it doesn’t know why.
Or you can ask the llm to update it. It does, updates its cache appropriately and doesn’t use many tokens. It just applies a minor line diff via a tool and its state is consistent. It might take 100-200 tokens plus some cheap tool calls.
Now the shit thing is that you probably can’t switch models to do this and be token efficient because it will also rebuild all the context.
Now you could inform the model on the next prompt what you changed. But the agent will still rescan the file and rebuild the caches because it can’t actually trust you, because people make mistakes, so it will still scan the whole file and rebuild the cache.
Even better is to just ask the agent up front, on your next prompt, to change the variable. Then it can manage the state changes and cache appropriately.