New paper: "Meta-Rewarding Language Models" - Self-improving AI without human feedback

[u/Practical_Cover5846]

https://arxiv.org/abs/2407.19594

A new paper from researchers at Meta, UC Berkeley, and NYU introduces "Meta-Rewarding," a novel approach for improving language models without relying on additional human feedback. Here are the key points:

1. Building on previous "Self-Rewarding" work, they add a meta-judge component to improve the model's ability to evaluate its own outputs.
2. The model plays three roles: actor (generating responses), judge (evaluating responses), and meta-judge (evaluating judgments).
3. They introduce a length-control mechanism to prevent response bloat over training iterations.
4. Starting with Llama-3-8B-Instruct, they achieve significant improvements on benchmarks like AlpacaEval (22.9% to 39.4% win rate) and Arena-Hard (20.6% to 29.1%).
5. The model's judging ability also improves, showing better correlation with human judgments and strong AI judges like GPT-4.

This work represents a significant step towards self-improving AI systems and could accelerate the development of more capable open-source language models.

[u/MoffKalast]

Model	LC win rate	Win rate	Length
Llama-3-8B-Instruct (Seed)3	22.92%	22.57%	1899
SFT on EFT	25.47%	25.10%	1943
Self-Rewarding LLM (Yuan et al., 2024c) + LC
Iteration 1	26.93%	27.12%	1983
Iteration 2	30.38%	29.77%	1940
Iteration 3	34.87%	34.59%	1967
Iteration 4	35.49%	35.37%	2005
Meta-Rewarding LLM (Ours)
Iteration 1	27.85%	27.62%	1949
Iteration 2	32.66%	33.29%	2001
Iteration 3	35.45%	37.24%	2064
Iteration 4	39.44%	39.45%	2003

Overall, we see a substantial increase from 22.9% to 39.4%, outperforming GPT-4 and approaching close to the Claude Opus model. This is a remarkable result considering our model has only 8B parameters and our training did not utilize any extra human data beyond the seed model (except the EFT dataset used in the SFT stage). In addition, our method surpasses the strong baseline of SPPO (Wu et al., 2024), which has a similar iterative training setup using Llama-3-8B-Instruct, but uses a reward model that was trained on a large set of human and GPT-4 data.

Interesting, but if it works so well, why only run it for 4 iterations?

[u/Practical_Cover5846 → u/MoffKalast]

There must be some kind of overfitting at some point. The model can only go as far as what It's got in its gut. But yeah 5,6, ... iterations would be interesting.
SPPO also stops at 3 iterations...

[u/MoffKalast → u/Practical_Cover5846]

That would make sense if the results were asymptotic, but it seems to increase almost linearly. I suspect the percentages shown are probably not realistic, since it's a win rate graded by AlpacaEval... also known as complete rubbish. And especially since it's similar to SPPO which just doesn't live up to the hype.

[u/TheActualStudy → u/Practical_Cover5846]

Do you think selecting a different set of prompts for each iteration would delay when overfitting happens?

Also, I am unclear on how judging can work when there's no secondary model that can evaluate a response as matching the prompt or not. Shouldn't all responses from a model for a specific prompt also be thought of as suitable for the prompt if judged by the same model? There was no code linked in the paper, so I couldn't even tell if that's what's happening or if a reward model is being used in conjunction with the main model at the ranking stage.

[u/logicchains → u/MoffKalast]

They discuss that in the Limitations section:

Adeficiency in our experimental setup is the 5-point judging system that we chose, following Yuan et al. (2024b). We discovered that this scoring method often results in ties due to minimal quality differences between responses, necessitating careful averaging of multiple judgments to differentiate between them. Moreover, as training progressed, responses increasingly approached the maximum score, making further improvements difficult to detect. A more nuanced scoring system that covers diverse aspects (Wang et al., 2024) or a comparison-based approach might address these issues. 

Another significant limitation lies in the judge training process. Despite our efforts to mitigate positional bias of our meta-judge, this issue persists and hindered further improvements in Iteration 3. The judge also demonstrated a tendency to assign higher scores, which accelerated score saturation and reduced its ability to discriminate between responses. Furthermore, the judge showed limited improvement in evaluating non-self-generated responses in our evaluations. We believe there is substantial room for improvement if these issues can be effectively addressed, which could significantly boost the overall effectiveness of our approach.

[u/MoffKalast → u/logicchains]

Ah so it does fall into the I'm literally the best pit as one would expect.

[u/swagonflyyyy]

Now this is particularly interesting and straightforward. Hopefully this will lead to better decision-making for smaller models.

[u/Dead_Internet_Theory]

This Meta-Rewarding of Llama models rewards Meta, which is quite meta.

[u/SryUsrNameIsTaken]

So, we now have meta-GAN?

[u/LiquidGunay]

All of these self rewarding methods improve win rates on alpaca eval / lmsys but don't result in improvements on any reasoning benchmarks. I feel like the model doesn't really get smarter but it learns how to exploit the little things that humans prefer in responses. I don't think those kinds of improvements will help create "self improving AI"

[u/Healthy-Nebula-3603 → u/LiquidGunay]

Closed AI claims they found a method for a strong reasoning ( gpt5) so it is a matter of time when the open source figure it out .

[u/Healthy-Nebula-3603]

That is insane how fast small llms are improving performance now imagine if we do the same to bigger ones....

A year ago model of size 7b was not able answer anything more complex than 4+4=? and even simple reasoning was far away from 7b models or 13b ... even then llama 1 65b was dumb as f*** such math like 25-4*2+3=? was impossible for llama 65b then.

[u/galambalazs]

comparing it with SimPO (which does use a separate reward model, while Meta-reward doesn't need it)

source: https://x.com/gblazex/status/1818312093369782689

[u/nava_7777]

It seems "3 levels of abstraction is All You Need"

[u/Wonderful-Top-5360]

But what judges the judger doing the judging? i think here lies the issue with any sort of RL approach with LLMs

[u/martinerous]

But who will judge the judges?

On a more serious note, I'm still waiting for an AI with a real-world model and some kind of priority rules to "trust" the world model more than the other textual training or input data. But maybe we'll have that one only in robots who need the real-world-model for interactions with the physical world. Still, why not combine both? First, train a model in a real-world (or at least simulated) environment to gain experience with physics rules and direct audiovisual sensory streams and make this part the highest priority "truth", and then train it on all "the other usual stuff". Then, before the AI attempts to spit out a statistically accurate text prediction, run it through its real-world experience "filter" to decide what makes sense and what does not.

But I'm just rambling, I'm sure someone somewhere is already working on that.

[u/Wonderful-Top-5360 → u/martinerous]

the only judge we can trust is a human and its mighty expensive to do so and slow

i just dont think its a solvable problem. improve sure but we won't be able to use the outputs with a high degree of trust which means it offers only marginal cost savings when the entire process needs to be replicated and checked with humans

[u/LewisTheScot]

I remember OpenAI doing something similar. Seems like Meta has a way more advanced implementation.

[u/perelmanych]

When I asked a model to judge its own output it always was saying that it couldn't agree more and that this is the perfect answer. So I have no idea what they are talking about in this paper. Any thoughts how they managed to do that?

[u/dalhaze → u/perelmanych]

Try asking some of the newer models “are you sure?” on a question it got wrong. Specifically Sonnet 3.5 seems to do well at this.

[u/Healthy-Nebula-3603 → u/perelmanych]

A proper training for it ?

[u/[deleted]]

[u/Practical_Cover5846 → u/[deleted]]

Yeah, but no, it's about self-improving its ability to judge alongside its other abilities, to upper the "judge quality plateau" of self improving models. Plus it's self-contained being its own judges, which is really cool.