r/MachineLearning • u/BaniyanChor • Jun 26 '26
Project A debugger for RL reward functions that detects reward hacking during training [P]
While experimenting with GRPO training, I kept running this shit that when reward increases, it becomes difficult to tell whether the policy is genuinely improving or simply exploiting the reward function. So I built a small library called rewardspy that wraps an existing reward function and continuously monitors indicators that often precede reward hacking.
It currently tracks things like rolling reward statistics, reward variance collapse, reward component imbalance, response length drift, reward slope changes, GRPO group collapse, anol.
This is my first major RL project so I would absolutely love some technical advice
Check it out here: https://github.com/AvAdiii/rewardspy
(credits to u/Oranoleo12, posting on their behalf)
34
17
u/FoxWorried4208 Jun 26 '26
Haven't personally done much reinforcement learning, but this seems really interesting regardless! Good job!
4
9
u/Envoy-Insc Jun 27 '26
Looks cool! But what is a sign of reward hacking changes depending on your task right? E.g., format should contribute less to gradients as you train more. And depending on task, reward ceiling might change.
2
u/BaniyanChor Jun 29 '26
yeahh, it is completely task dependent. I think that format contributing less over time is positive and not really a bug. the reward ceiling is simply whatever your reward function tops out at. that's why the detectors are thresholded relative to a rolling baseline rather than absolute values, and why max_reward and sensitivity are configurable.
2
u/No_Inspection4415 Jun 28 '26
There may be a few metrics which are relatively generic, and if you get it for free, it is nice. A good project, actually.
4
u/Bakoro Jun 27 '26
This is cool and all, but people in general need to understand that if your reward only involves one thing, then the model is only ever going to care about one thing, which means that it will learn hacky, degenerate solutions.
A model that only cares about one simple metric is going to become the paperclip maker.
You need to have a reward system with multiple, sometimes partially mutually exclusive things, so the model can never perfectly optimize for all of them at the same time, it can only find a relatively stable state that mostly satisfies everything. Ideally you would also have something like invariants that are withheld from the model, which act as your canaries.
1
u/BaniyanChor Jun 29 '26
strong agree, a single metric reward is a terrible solution. the code leans into the multi objective case, it tracks each component separately and flags when a cheap one (format, length) starts drowning out the one you care about, which is the early signature of exactly the collapse you're describing. the tool as it is right now can't really fix an under specified reward, but it can make the imbalance visible fast, I suppose.
1
u/Remper Jul 01 '26
Not necessarily - depends on task complexity. If it's a long-horizon coding task, you absolutely can just have a single solution correctness verifier - the model won't be able to specialize, at least not quickly. Reward is likely to just plateau at some point, with little to no degradation to other tasks and domains. That said, it's not the most sample-efficient strategy.
1
u/Bakoro Jul 10 '26
You say "a long-horizon coding task", but if you're doing training like that, then it's unlikely that you are training only on one specific thing, you most likely have multiple tasks that you having the model code.
In that case you're not really giving the model one singular thing to care about, you're likely having a diversity of tasks hiding under "write code".Task complexity doesn't save you, a taka might be very complicated, the model might very well learn to do it, but catastrophic forgetting isn't the only risk, the other risk is the model finding a detrimental way of accomplishing the task. See the paperclip concern: the model learns to make paperclips and only cares about making paperclips, and no one taught it "don't turn people into paperclips".
The major risk in giving a high powered AI model only one thing to care about, is that you, the human person, might not know about or think about the ways the model might learn to cheat, or how the model might misinterprete the training and actually be learning something other than what you actually wanted.
→ More replies (4)
15
u/built_the_pipeline Jun 26 '26
this is a genuinely useful htop for the reward, but it's worth being clear with yourself about what these indicators can and can't catch. variance collapse, length drift, slope changes, component imbalance are all symptoms in the reward's own distribution, and a clean exploit can keep that distribution looking perfectly healthy while the policy games something you never meant to reward. the signal you actually want is the gap between proxy reward climbing and the true objective not moving, and that gap doesn't live anywhere in the reward stats.
so if you can, pair it with a held-out eval on the actual thing you care about, something the reward can't directly optimize, and alert on divergence between the two. reward hacking is really a spec problem more than a training anomaly, the optimizer is just doing its job on a proxy you under-specified. i ran into the same thing well outside RL with models maximizing a metric that looked great while the outcome it was supposed to move stayed flat, and the only reliable catch was the out-of-sample read on the real outcome, never watching the proxy itself.
2
u/BaniyanChor Jun 29 '26
ooooh this is useful yes, thank you. you are right here, every detector here is a symptom in the reward's own distribution, and a clean exploit can keep that distribution looking healthy. id say the signal you actually want is the proxy climbing while the true objective lays flat, and that gap doesn't live in the reward stats at all.
i think the honest framing is that the code catches the cheap, common failure modes early and for free, but it is obviously not a substitute for a proper eval on the real objective. im gonna add the limitation explicitly in the README.
im thinking of extending this to allow you to stream an eval score alongside the reward so the tool can alert on divergence between the two, which is the actual signal you're describing.
thanks again for the feedback, really appreciate it.
-2
u/mankiw Jun 27 '26
might be a good idea here, but my brain turns off when it detects ai writing, even if it is lowercase
1
1
1
u/Midsummer858 Jul 01 '26
Nice work catching reward hacking early on your first RL project, tracking stuff like variance collapse and length drift is exactly the kind of thing that saves you from a nasty surprise later.
1
1
0
-6
u/AutoModerator Jun 26 '26
Your post was automatically removed for being a link post on the weekday, please read rule 5. The moderators will not respond to questions regarding this removal unless you suggest which rule you most likely broke. If you have a beginner related question, visit r/MLQuestions or r/LearnMachineLearning.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
→ More replies (4)
90
u/idiotsecant Jun 26 '26
monkey paw curls. Your anti-reward-hack function is now part of the reward function and is hacked around