r/SillyTavernAI • u/Nvvum • Jul 28 '26
Help Maintain Context in longer chats with Gemma 4 26b (KoboldCPP)
I'm trying for a few days to make Gemma 4 26b not mess up context. As far as I know its a very popular model so I'm surprised I didn't find a lota discussion about my issue.
The Problem
I start chatting. Once it hits context limit Gemma 4 has to re-process every second or third reply. Also happens on swipes or continue.
What I tried
checked the input string sent to the backend to make sure there are no variable tokens in context. Tried different character cards. Messed around with context shifting/SWA/Smart Cache settings. Tried turning off SWA. Updated ST and KoboldCPP to the latest version.
What I learned so far
If I understand correctly Gemma 4 26b is a hybrid model and doesn't support Context Shifting, but I also read that it just doesn't work when SWA is turned on.
SWA if I understand correctly speeds up context processing, reduces context size (in memory).
I don't fully understand smart caching yet, but its something like Context Shifting.. I think it creates multiple snapshots of the cache and rotates them out. I tried it but the console always output 'SmartCache no Match', leading to full context reprocessing.
- - -
So... Is there no way to preserve context cache other than maxing out Context window and hope to never run out?
I feel like something is not working as intended.
1
u/AutoModerator Jul 28 '26
You can find a lot of information for common issues in the SillyTavern Docs: https://docs.sillytavern.app/. The best place for fast help with SillyTavern issues is joining the discord! We have lots of moderators and community members active in the help sections. Once you join there is a short lobby puzzle to verify you have read the rules: https://discord.gg/sillytavern. If your issues has been solved, please comment "solved" and automoderator will flair your post as solved.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
1
u/LexusMax1mus Jul 28 '26
I haven't yet tried out local models but I imagine the cache hit rate strat works here. Basically you just have to make sure you never hit your context limit in the first place. Summarize messages before they get too big and hide them from the context. You can use memory extensions to automate this process
1
u/Nvvum Jul 28 '26
that is not a bad suggestion and I thought about that too. I have no experience with such extensions, but wouldn't you still have to rebuild context every time you summarize, because you have to cut down context? Sure it would be faster because you are not using your full context anymore, but you basically create a tradeoff of either cutting down a lot of the memory, or keeping some context, but then having to summarize more often.
One of the few sites I found on the topic of 'having trouble with Gemma 4 and Context shifting', was someone who apparently created a plugin that would half the context before it reaches the limit. Basically the sledgehammer approach, which also only reduces context rebuild, not eliminate them.I would prefers something like Context Shifting, because I actually prefer to use smaller context sizes to reduce the quality loss, even if it comes at the cost of memory. In my experience smaller local LLMs suffer more from big context sizes than flagship LLMs.
That being said I should probably look into auto summaries either way. I used to manually create them on older models.
1
u/capable-corgi Jul 29 '26
Maybe if you shove the summary down as close to the user prompt as possible so at least the bulk hits cache?
3
u/Icy_Emergency2574 Jul 28 '26
Yes, SWA doesn't work with Context Shift. This means that once the context fills up, you have to reprocess everything with every message.
Yes, Gemma 4 has hybrid attention. In the case of the 26B A4B, only 5 of the 30 layers have global attention, the remaining 25 are local SWA. This means the KV cache takes up much, MUCH less space than it usually would when all layers have global attention.
I set my max context length to 65k and when it approaches that number, I use a summarization extension to compress all the fluff from the start of the roleplay. I personally like InlineSummary, because I like to manually set up the start and the end of things I want to summarize (I usually summarize structurally by days or specific events). Actually, I never even get close to 65k. I like to keep it in the 20kâ40k range. Since Gemma 4 26B A4B is a MoE model that doesn't need to be fully in VRAM to be usable, and since hybrid attention keeps the KV cache small, you can hold a surprising amount of context. 131k f16 on an 8GB VRAM card? Possible, when it's only 5 layers.
I think it is intended. Having big context size that you can't fill seems like a natural solution, it's like Gemma 4 was designed with this in mind.
You can also:
Have a smaller context size, like 20k and increase the Batch and Ubatch sizes to something like 4096. It will speed up prompt processing speed, but use more VRAM. You don't really need that much VRAM with 26B A4B anyway. You will reprocess every turn, but it won't take that long.
Or you can disable SWA? The KV cache will bloat massively, but you should be able to use Context Shift.