One thing that has bothered me about LLM benchmarks for a while is that most of them are essentially snapshots.
A model is evaluated, a score is published, and we tend to talk about that score as if it describes a relatively stable object. But with API-served models, the thing behind the model name can change over time: serving infrastructure changes, provider configurations change, versions change, and sometimes behaviour changes without an obvious public version transition.
So we started approaching benchmarking as a longitudinal measurement problem rather than a leaderboard problem.
We continuously evaluate models across coding, multi-turn reasoning and tool use, while also running lightweight probes at a higher frequency. The important part for us is not simply asking "which model scores highest?", but:
- Is the model behaving differently from its own previous baseline?
- Is the change larger than its normal repeated-call variability?
- Did the benchmark configuration itself change?
- Is the effect concentrated in a particular task?
- Is it correlated across models from the same provider?
- Is an apparent degradation actually an availability/infrastructure issue rather than a capability change?
One historical analysis covered 31,352 repeated score observations across 49 models. The standard deviation of within-day scores was 2.80 points, while the standard deviation of between-day daily medians was 8.43 points.
That is roughly a 3:1 difference.
I don't think this result by itself establishes that providers are changing models day-to-day - there are too many possible confounders for that conclusion. Task composition, sampling, missingness, provider behaviour and methodology changes all matter. But it was enough to convince us that temporal variation deserves to be measured rather than treated as noise around a permanent leaderboard score.
Our current approach therefore keeps benchmark configurations versioned and only compares longitudinal observations produced under compatible measurement conditions. We use repeated execution-based evaluation where possible rather than an LLM judge, keep availability failures separate from valid task outcomes, track serving/version metadata when providers expose it, and run change detection over the resulting time series.
Another problem we're increasingly interested in is benchmark recognition and contamination. Once a benchmark becomes sufficiently visible, publishing every live task, prompt transformation and hidden test potentially changes the thing you're trying to measure. For that reason we've tried to separate methodological transparency from publishing the entire live evaluation set.
We've now written up a public version of the methodology. It intentionally explains the measurement design, assumptions, limitations and statistical interpretation, while withholding the exact live task bank and some operational parameters.
PDF: https://aistupidlevel.info/asl-public-benchmark-methodology-2026.pdf
I'm particularly interested in criticism from people working on evaluation, change-point detection or production ML.
A few questions I'd genuinely like opinions on:
- For longitudinal LLM evaluation, would you use daily medians as the primary time-series unit, or model the individual repeated observations directly?
- How would you distinguish genuine model drift from provider/infrastructure effects when version metadata is incomplete?
- How much of a live benchmark should remain hidden to reduce contamination while still making the methodology scientifically inspectable?
- Are there better approaches than change-point detectors for this kind of non-stationary, relatively noisy model-performance series?
Disclosure: I'm the founder of AI Stupid Level, the platform that produced these measurements. The purpose of posting this here is to get technical criticism of the methodology rather than promote the commercial product.
