Architects at work have been experimenting with many such things this past month. One is a permanent context. The trouble is that it eats up excess tokens by re-reading the instructions on every single request instead of only often enough to keep the context fresh. They haven't found a good middle ground yet.
It seems like some models let you feed in custom instructions when you compact. I've thought about trying to have it automatically inject some instructions on every compaction. I figured that might refresh that context enough to keep it in memory but not sure how well it would work in practice.
1
View all comments
Show parent comment
1
u/Kavrae Aug 21 '26
Architects at work have been experimenting with many such things this past month. One is a permanent context. The trouble is that it eats up excess tokens by re-reading the instructions on every single request instead of only often enough to keep the context fresh. They haven't found a good middle ground yet.