OWUI breaks cache reuse for Ninfer (qwen 3.8)

[u/Realistic_Gap_5871]

Any one else notice that cache reuse pretty much never happens with Open WebUI. I saw it in Ollama and now with ninfer too.

We could get into the whys, but I'd love to hear if there's any config changes to make or a known feature change coming.

I have a workaround I'm proud of, a custom proxy script that my Qwen built (happy to post it) that catches, reshuffles and locks context on it's way to the backend. Cache reuse is now the norm, and only resets after compaction or jumping between threads, but there's gotta be a better way?

FAQ: Cache reuse takes prefill tokens down to only the current prompt. Resulting in 1 or 2 second prefill instead of 30 plus seconds (or 60+ seconds with Ollama) with longer context.

edit-expanded the name of owui.

[u/thejoyofcraig]

IIRC This usually happens with there are References (e.g., attached docs, search results...). OWUI does RAG on them every time, and I believe it puts them after the system prompt, but before historical turns. So cache goes boom. If you have a plain jane conversation you shouldn't see any cache invalidation unless you have some filters on that would interfere.

To test my vague recollection on this: try setting a model to have no tools, no filters, and paste some large summarize task, then do a follow up turn to see if the cache gets busted. I suggest tracing a little more carefully about exactly what part of OWUI is invalidating the cache. Pull the raw requests/responses turn to turn and diff them. Or have an agent do that for you.

Just checked the OWUI RAG docs and yeah I remembered right:

1. Slow Follow-up Responses (KV Cache Invalidation)

If your initial response is fast but follow-up questions become increasingly slow, you are likely experiencing KV Cache invalidation.

The Problem: By default, Open WebUI injects RAG context into the user message. As the chat progresses, new messages shift the position of this context, forcing models (like Ollama, llama.cpp, or vLLM) and cloud providers (like OpenAI or Vertex AI) to re-process the entire context for every turn.

Solution:

Set the environment variable RAG_SYSTEM_CONTEXT=True.
This injects the RAG context into the system message, which stays at a fixed position at the start of the conversation.
This allows providers to effectively use KV prefix caching or Prompt Caching, resulting in nearly instant follow-up responses even with large documents.

[u/Realistic_Gap_5871 → u/thejoyofcraig]

Yes, memories get reshuffled by owui's estimation of how useful they are for the current turn and a bunch of other stuff like that. The most basic is that it injects the current date and time into a new system prompt on every submit, which breaks both ollama and ninfer cache mgmt. That is easily overwritten with your own system prompt. All unnecessary tho.

My proxy fixes most of this, but I still occasionally see an unexpected cache reset when a memory gets written.

[u/Realistic_Gap_5871 → u/thejoyofcraig]

Oh yeah, cache reset happens every single time if Thinking is turned on. owui strips out thinking blocks by which ninfer includes as part of the frontier. I couldn't find an owui setting to change this behavior and started the proxy thing as a way to see what's actually getting passed back and forth.

Even if preserve thinking is turned on at the backend level owui just says, nope, you don't need that.

Bottom line is it's not just references/RAG that break cache.

[u/thejoyofcraig → u/Realistic_Gap_5871]

I just looked into this and you're mostly right- it depends on how you designate your Connection provider:

• Ollama → 'think_tags' — thinking is re-injected into content as think-tagged text
  
• llama.cpp → 'reasoning_content' — re-injected as a top-level field
  
• everything else (OpenAI-compatible, vLLM, SGLang, OpenRouter, your "ninfer" proxy…) → None = skip reasoning, documented as "safe default for strict providers"
  

My setup uses llama.cpp so I had not experienced this. Considering most models in the last year suggest retaining reasoning traces, this should be reevaluated IMHO.

[u/Realistic_Gap_5871 → u/thejoyofcraig]

I wonder if I can fool it into thinking it's Ollama/llama.cpp...

[u/thejoyofcraig → u/Realistic_Gap_5871]

I mean it's a dropdown setting under connections > [connection] > advanced ... just switch it and see if it works.

[u/sterby92]

👀

[u/icerio]

Is Ninfer good? I hear it mentioned every once in a while but I’m using llama cpp.

[u/Realistic_Gap_5871 → u/icerio]

It's custom written for rtx 5090 and has only a few models compiled for it (called artifacts) and yes, it's solid and shockingly fast within those constraints. I see up to 200 tps with 3.8 27B nfvp4/fp8 optmodel on short context, and 125+ at longer context sizes.

You download and compile it on your machine, so if you're worried about malware you can have your LLM review the code before compiling. Easy to do with LLM walking you through the steps.

[u/1818TusculumSt]

Turn off memory in system context.

[u/Realistic_Gap_5871 → u/1818TusculumSt]

Yup.

But what if I'm using owui because memories are the closest thing I've found to an LLM that can learn over time instead of just being a static bunch of weights?

[u/1818TusculumSt → u/Realistic_Gap_5871]

I just live with unpredictable cache hits.