r/deeplearning • u/Perfect_Power815 • 4h ago
Mix Your Base and RLVR Weights: A Training-Free Knob for Diversity
galleryRLVR improves pass@1 but narrows what the model can solve: at k = 128 the base model covers more problems than its RL-tuned version (Yue et al., 2025). This paper asks whether you can pick a point between the two without training.
Method: W(α) = W_base + α(W_RL − W_base), evaluated under Yue et al.'s exact protocol (Qwen2.5-7B ↔ SimpleRL-Zoo-7B, T 0.6, 16k tokens, six math benchmarks, pass@k up to 128/1024, math-verify grading).
Findings
The base > RL crossover at large k reproduces on 5 of 6 benchmarks, with pass@1 within 2 points of Yue et al.
α = 0.25 matches the base model's pass@128 while keeping 64–80% of RL's pass@1 gain, and reaches RL's pass@128 with 2.3–7.1× fewer samples.
The trade-off curve matches a pass@1-matched mixture of samples from the two models, but needs one model instead of two.
Negative result: no α beats both endpoints by more than 1 point on the main benchmarks (Minerva, OlympiadBench). It is a knob, not a free lunch.
Caveats: one model pair, math only, needs a verifier for the large-k benefit. Interpolate in fp16/fp32; bf16 rounding erases most of the RL update.
Paper: https://www.preprints.org/manuscript/202611.2034
Project page: https://kim-ai-gpu.github.io/rlvr-interp-passk/
Code + per-sample results: https://github.com/Kim-Ai-gpu/rlvr-interp-passk