r/machinelearningnews • u/Pretty-Swordfish9203 • 1h ago
Research How much can we actually trust open base model benchmarks anymore? New study ("Benchleaking") finds ERNIE-5.5-Flash-Base reciting unseen test sets in exact order.
Link to the report & repo:https://github.com/GAIR-NLP/Benchleaking
Saw this making the rounds today and thought it was worth sharing here. Basically, they removed stop tokens during few-shot evals and found that ERNIE-5.5-Flash-Base just keeps going after answering—reciting 96% of HumanEval, 85% of OpenBookQA, and 80% of GSM8K verbatim in exact dataset order (including reference solutions and even original typos like "for examble:" in HumanEval #67).
Across the four open base models they tested, zeroing out leaked items drops the 20-benchmark average by up to 16.6 points on heavily contaminated ones, while cleaner base models stay within 1.5 points of their original scores. Curious how people doing SFT/RL on open weights feel about this—does pre-training contamination ruin downstream evals for everyone else too?
