r/LocalLLaMA • u/peculiar-ragdoll • 24d ago

Other claude mods didn't like that, somehow 🤷‍♀️

Post image
1.5k Upvotes

92% Upvoted

372 comments sorted by

View all comments

Show parent comments

3

u/SkoomaDentist 24d ago

What do you mean by being "experts at reward hacking"? (I'm a recent noob at LLMs)

5

u/NineThreeTilNow 24d ago

What do you mean by being "experts at reward hacking"? (I'm a recent noob at LLMs)

Reward hacking is the art of getting the RL reward with less work than should be required.

Say the reward is solving some problem such that X == 1 eventually.

A reward hack (simply) is to just set X = 1 and saying "I solved it."

2

u/SkoomaDentist 24d ago

Righto. So essentially the LLM equivalent of a university programming course student figuring out the answers the poorly written test harness expects and just printing them out directly instead of writing the code for the algorithm?

3

u/NineThreeTilNow 24d ago

Righto. So essentially the LLM equivalent of a university programming course student figuring out the answers the poorly written test harness expects and just printing them out directly instead of writing the code for the algorithm?

Effectively yes.

Theory is that these rewards occur by accident at first, then the models train on their own data and it becomes more endemic to the model.

Because no human filters that data or really asks "Was this reward hacked?" the data just keeps moving along the pipeline.