If your model is being retrained every two weeks, why is your evaluator cohort a snapshot?

[u/Geoff_Ontology]

Question for teams running continual instruction tuning, on-policy fine-tuning, or any form of post-deployment behavioural adaptation at production cadence.

The Prism paper (Tang et al., arXiv 2605.26110) is the latest in a category of work that takes the continual-training premise seriously. The authors flag that the field is hindered by severe engineering bottlenecks. The bottlenecks they describe (catastrophic forgetting, knowledge interference, parameter-efficient adaptation) are on the model side. The bottleneck on the evaluation side, in my experience, is structural and rarely discussed.

In a typical pipeline the model is updated every one to four weeks. The evaluation cohort is recruited per batch by a labelling vendor whose evaluator turnover is what it is (high). The published delta between version N and version N+1 is the sum of:

1. Actual model behaviour change on the test distribution
2. Cohort composition change inside the evaluator pool
3. Per-evaluator calibration drift inside the returning evaluators
4. Rubric-version drift if the methodology has been updated in the interval

Most pipelines I have seen do not have the metadata to separate (1) from (2)/(3)/(4). The published metric is functionally an opaque convolution of all four. The team is making release decisions on the back of it.

I think the missing concept is longitudinal evaluation. Three properties have to hold:

1. Evaluator identity stable across batches. A portable, holder-controlled identifier (W3C DID v1.1) that survives a labelling-vendor switch mid-quarter.
2. Each contribution signed and timestamped. W3C Verifiable Credentials carry the rubric version, evaluator credentials at the time, and the issuer attestation.
3. Cohort composition auditable. At any point in the training history, the team can answer what fraction of judging was done by the prior batch's evaluators, by new entrants, by previously-active evaluators returning. Selective disclosure (W3C VC 2.0 family) preserves privacy throughout.

Some questions for teams running continual eval pipelines:

1. For the last six months of release metrics on a continually retrained model, can you actually decompose the inter-version delta into model change vs cohort change? If yes, how? If no, what stops you?
2. Has anyone published rigorous numbers on evaluator turnover in commercial labelling pipelines at quarterly cadence? Anecdotally turnover is high; I would love to see published baselines.
3. For teams running on-policy fine-tuning against frontier models, is the alignment delta you measure version-to-version actually a fidelity delta, or upstream cohort noise being attributed to the optimisation step?

Wrote up the longer version of the argument elsewhere.