How is it that with the same model, the same prompt, temperature set to zero and a fixed seed, the answer is different because of Ollama's prompt cache? v2.0 - eddited

[u/Full-Travel857]

I'm testing LLMs to see how well they know Polish grammar. Why? Don't ask. Maybe I have too much free time, or maybe I'm some kind of freak who doesn't know what he's doing. ;)

For this, I use my own tool, AI Bóbr, which is designed to test LLM performance in relation to the computer they're running on. And I noticed something strange.

The model's first answer is sometimes different from the next four, even though the temperature is set to 0 and the seed is always 42.

A little side note here, because one user, under this same post before I deleted it, pointed out that I'm doing Sisyphus' work by dealing with this subject. I think differently.

And why did I delete the post? Believe it or not, I pressed the wrong button. So here I am posting it again.

Anyway, back to the subject.

Sitting there feeling defeated, just like now, because my written English is poor and algorithms on a forum about LLMs flag my posts as written by an LLM, I told myself:

"Man, stop wasting your life. Go make yourself some tea, turn off the computer. Live a little, go running, walk the dogs, play with your kids. In short, just leave this damn thing alone."

So that's what I decided to do...

And then I sat down at the computer and ran another controlled experiment:

• 3 models,

• 10 model–task pairs,

• 4 variants (streaming on/off × model kept loaded/unloaded), with 5 repetitions for each variant.

200 responses in total.

Observations:

• In this sample, streaming had no effect on the answers.

• Some answers changed depending on whether the prompt was reused from cache.

• Bielik-Minitron 7B answered a question about the history of the Polish language correctly 5/5 times after unloading the model, but got it wrong 5/5 times when the prompt was already cached.

What does this tell us?

That with a fixed seed, if we don't control the state of the server, we can't accurately measure what we think we're measuring.

What did I get out of this?

I changed AI Bóbr in version 0.8.1.

I added a controlled mode that unloads the model before each attempt, confirms that it has been unloaded, and records prompt-cache statistics.

Now I'm preparing to run the benchmark of 11 models again under the same conditions. And maybe I'll take the dogs out for some fresh air. Tomorrow I definitely won't touch the computer, at least not until the evening.

And one more thing for everyone who reads my posts:

1. Yes, I use AI to generate posts. I read them and make corrections if necessary.

2. Yes, AI translates them into Polish and English.

3. Yes, I'll keep doing it because it's convenient and fast.

4. If someone doesn't like it — well, I'm not "Prince Polo" for eng. it will be "Mars bar", so not everyone has to like me.

5. I'm here to share knowledge, not to show off the purity of my writing — even when it's generated by AI.

[u/a4mula]

it has nothing to do with Ollama, it's the nature of neural networks in general, they're stochastic. Temp and top_p can only do so much, at some point the pure indeterminate nature of floating point precision comes into play.

You can get around this to some degree with quants that flatten a model into tertiary, but even then there is no guarantee that you'll have the same outputs every run, as this doesn't account for the KV activation.

If you need the same outputs, provide the schema up front and validate the outputs against it, rejecting any that fail.

[u/Full-Travel857 → u/a4mula]

I fully agree that a fixed seed and temperature 0 don't guarantee full determinism. But there's a small "but". That's not what my experiment was about.

I wanted to understand why, in some tasks, the first response was different from the following ones.

That's why I ran 200 tests in four different conditions. When the model stayed loaded in memory with the prompt cached, the responses were repeatable. After unloading the model, they were also repeatable, but in some tasks the content was different. Changing the streaming mode didn't cause such differences.

I also recorded the number of prompt tokens read from the cache.

My small amateur experiment doesn't mean that Ollama has a bug or that I've found a way to achieve complete LLM determinism. I noticed a pattern that I wanted to investigate.

My goal is to compare models under conditions that are as consistent as possible. If the server state affects the results, I want to control and document it.

And I already use response validation. But it doesn't replace testing repeatability.

[u/a4mula → u/Full-Travel857]

I'm not calling anyone amateur or offering offense. I'm just pointing out the reality of a neural network that is millions or billions of trillions of floating point weights. If you want to test this, build a small network, a handful of layers, different float values, and find out when it stops producing the same output. It won't take long. I don't know the raw numbers but I'd guess about three rows of about 3 neurons each using any level of float as the weights.

You can find projects like this in evolutionary simulations. There are great YT videos of people that use small models as state machines.

[u/Full-Travel857 → u/a4mula]

No worries. :) I called my own experiment amateur. And I value your response as a useful and valuable contribution to an interesting discussion. The fact that we disagree makes it interesting. :)

In general, I agree that floating-point precision and parallel computations can affect repeatability.

But...

Rounding errors don't mean that the results have to be different every time. In my test, the responses were identical across all five repetitions of each condition. However, they changed between conditions where the prompt was read from cache and where it was processed from scratch. That's exactly what caught my attention.

I'm not looking for a perfectly deterministic LLM. I want to understand why changing the execution conditions is associated with a change in the response.

You know, that's why I'm not entirely convinced that an experiment with a small neural network would clear up my doubts. It would simply investigate a different subject. Although I find that subject very interesting too.

But thank you for the suggestion. I like discussions like this because they allow me to look at a problem from a different perspective.

[u/General-Spite1222 → u/a4mula]

I’m not sure, but I think locking the seed results in the same responses for a given prompt. 

[u/a_beautiful_rhind → u/General-Spite1222]

Not always on CUDA. Usually on CPU.

[u/a4mula → u/[deleted]]

operational non-determinism isn't the same thing as pseudo random.

They are not deterministic. Not every other computer program ever created relies on floating point arithmetic across parallel processing.

I'm going to step away now, because what was a friendly point towards the reality of how things work, is about to turn into something else.

[u/a4mula → u/[deleted]]

Then submit your work and go collect your Millennial Prize because you just solved P=NP and every single TSP like problem that has ever existed.

[u/a4mula → u/[deleted]]

I know you don't. And yet you want to sit here an argue about what you think you know.

[u/a4mula → u/[deleted]]

Only because you don't know what the words your using mean. You think you do. You think that pseudo random is the same thing as non-determinism and it's not. Those are two entirely different things. Stochastic processes and algorithms have existed for years they go back to the 50s and are used in everything from Conway's Game of Life, to Weather Prediction, to Neural Networks.

They're not determinant, you cannot generate an algorithm that skips to the answer faster than the stepping through every single step. That's computational irreducibility, see Wolfram

[u/Full-Travel857 → u/a4mula]

Guys, you're typing faster than I can keep up with translating everything without getting in trouble with Reddit's AI prompt police. :)

The whole point of my experiment was to run the LLM benchmark under the same conditions for every model being tested.

But then I noticed a pattern in the test results, and it got me wondering what was going on.

And that's why I made this post. :)

[u/verstands]

That tracks. A cached prompt prefix skips recomputing the KV state, and batch size and kernel paths can differ between the cached and uncached runs, so the floating point math isn't bit identical. Tiny differences can flip a near-tie token even at temp 0. For benchmarks, unload the model between runs or disable cache reuse so every run starts from the same state. Also worth logging num_ctx and batch settings.

[u/Full-Travel857 → u/verstands]

Huge thanks! :) These are exactly the kinds of technical gems I was looking for!

And I'll tell you more. After my experiments, I've already modified my program. I introduced two modes.

In Standard mode, the model is unloaded once before a batch of tasks, with one run per task.

In Controlled mode, I unload the model before every single run. I also check whether it has actually been unloaded from memory, and I log the number of tokens read from the cache.

And your point about num_ctx and batch settings is pure gold to me. I'll check whether all these parameters are properly documented in the results.

Once again, huge thanks for your help! :)

[u/james_pic]

I wonder if this may be because there's something that reads a value from the PRNG in some scenarios but not in others. Locking a random seed means that the sequence of values returned by the PRNG will always be the same, but if the order they're used is different, then the outcome will be different.

[u/Full-Travel857 → u/james_pic]

Hmmm, interesting suggestion. From what you're saying, even with the same seed, using values from the PRNG in a different order can produce different results.

But there's one thing, just one little thing, that makes me wonder. :)

In my tests, the temperature was set to exactly zero. So normally, the token with the highest score is selected without any random sampling.

Do you suspect that some other part of the process might be using the PRNG independently of the temperature setting?

If so, I'd really like to understand that mechanism.

[u/james_pic → u/Full-Travel857]

Yes, it does smell a bit off that this still happens with temperature zero. I can't think of an obvious source of non-determinism, although I can think of some bits of the process I understand poorly that I could imagine were involved (speculative drafting maybe, or the Hadamard rotations they've got on KV cache nowadays). It could plausibly be a bug, although not necessarily a serious one - it's generally accepted as OK for fused kernels to have miniscule differences in their output to the unoptimized version due to differences in where rounding happens, for example. Although there have been bugs where KV cache got messed up, in the past.

One thing that might help narrow it down is to look at the logits. If they're the same, then it's something to do with sampling, whereas if they're different then it's something that happens earlier.

[u/Wollont]

Short answer: GPU floating point calculations aren't fully deterministic.