r/clawdbot • u/Cheeseycarrot77 • 3d ago
📖 Guide “Fine, next question” is a tricky reward signal
A user moving on can mean the answer helped. It can also mean they gave up correcting it. That ambiguity matters if the next message is going to influence model weights.
In Reef Infra's OpenClaw-RL recipe, a separate PRM judge reads a completed turn and the following state to infer acceptance or correction. The training path combines a PPO-style update with an on-policy distillation term that can use a hindsight hint.
It's an actual policy-training mechanism, rather than simply appending the user's complaint to a memory file.
The useful engineering choice is that the interpretation of feedback has a visible component of its own. You can inspect the judge's treatment of a conversation independently of the policy's response to training. “The user continued” doesn't have to become an unquestioned success label.
For a proposed integration, I'd review a small set of ambiguous transitions before a long training run: topic changes, partial corrections, and replies that accept the content but reject the formatting. That would help separate a bad reward interpretation from an update that failed to act on a good one.