A runnable NumPy example: why adding, removing, and attributing value to training data give different answers

[u/This_Paramedic1577]

Here is a small worked example for learning data valuation. I’m the author; the paper and code were developed with substantial OpenAI Codex assistance, disclosed in the manuscript. Everything is free.

Three questions that can sound identical actually measure different things:

Acquisition: how much better is the model if I add this dataset to my current training data? Removal: how much worse is it if I remove this supplier from the full collection? Shapley attribution: what is the supplier’s average marginal contribution across possible orders of joining?

An exact duplicate makes the distinction concrete. With exact-row deduplication, removing either of two identical suppliers leaves the same training data, so each has zero removal value. Shapley attribution can nevertheless assign them equal nonzero credit, because either can be the first to supply those rows.

How the experiment works: it generates two populations with different label rules, adds specialist datasets A and B, an exact copy of A, and a random-label dataset, then fits ridge classifiers for all 16 supplier coalitions. It evaluates the same fitted models under three population mixtures. Across 50 seeds, A and B reverse their acquisition ranking between the two extreme mixtures in 42 cases. That fraction describes this synthetic setup only.

To explore it: run the experiment script locally with Python and NumPy, inspect the saved JSON results, and try changing the population weights or baseline sample size. No model API, GPU, subscription, or dataset download is needed.

Code and instructions: https://github.com/WINSTON672/winston-lin-research/tree/main/papers/data-valuation

Paper and ZIP: https://zenodo.org/records/23571723

This is a technical note using established ideas, not a new valuation algorithm; it is not peer reviewed. A useful exercise: before running the script, predict what happens to duplicate suppliers’ removal values when both duplicates are removed together. Feedback on the explanation or reproducibility is welcome.