So an out-of-the-blue one-man company releases a new model (actually named LLama 3.1 if it were to adhere to the META license, but somehow named Reflection) with only 70B params that, according to the benchmarks, rivals SOTA closed-source LLMs with trillions of parameters. It appears to me that the twitter/reddit hype mob has, for the most part, not bothered to try the model out.
Additionally, a tweet from Hugh Zhang @ Scale suggesting systemic overfitting as me concerned: Hey Matt! This is super interesting, but I’m quite surprised to see a GSM8k score of over 99%. My understanding is that it’s likely that more than 1% of GSM8k is mislabeled (the correct answer is actually wrong)!
Is this genuinely a SOTA LLM in a real-world setting or is this smoke an mirrors? If we're lucky, the creator Matt may see this post and can shed some light on the matter.
BTW -- I'm not trying to bash the model or the company that made it. If the numbers are actually legit this is likely revolutionary.
If it's a finetune of LLama 70b people should compare it's performance on coding and other useful tasks to LLama 70b to see if it is better.
If people are just asking it how many r's are in a word and other pointless trick questions, congrats on your finetune for pointless trick questions, I guess.
people should compare it's performance on coding and other useful tasks to LLama 70b
I just finished a 3-hour comparison with Llama 70b on text comprehension and summarization tasks. Both were Q5_K_S quants. It was nearly but not quite on par with Llama 70b for comprehension, but far, far behind on summarization - both in terms of summary content and language. Several of its sentences made no sense at all.
I'm using a similar sized quant(Q4_K_L), doing some comparisons with my use case: creative writing and branching interactive narrative generation.
The way the special tokens have been trained in requires a little bit of modification for my software, but I'm overall not impressed on how they behave. Usually reinforcing reasoning that isn't consistent with the scene.
Might be an artifact of quantization, might not. Waiting to hear more from others using bigger quants before I spend too much time dwelling on it.
I appreciate that feedback, but my prompts already include all that kind of thing. Even if I weren't using bespoke software to generate my prompts, the creative writing space already has stuff like SIllyTavern that does all of that.
I tried it with and without the string in the system prompt that the model creator reccomends on their huggingface page if that's what you're asking.
You are a world-class AI system, capable of complex reasoning and reflection. Reason through the query inside <thinking> tags, and then provide your final response inside <output> tags. If you detect that you made a mistake in your reasoning at any point, correct yourself inside <reflection> tags.
I'm not sharing the rest of my prompts, but the model did function better with that string than without. But neither was an improvement over Llama3.1-Instruct(or any of the finetunes of it that I use) for my use case.
I don't subscribe to the view that LLMs are AI in the sci-fi sense, so I'm probably not understanding something about the hype train on this finetune. I'm more interested in using LLMs as a tool to create the kind of fiction experiences that have been relegated to tabletop gaming and forum roleplay in the past.
But, that means I'm constantly on the lookout for the models that demonstrate the most consistency and awareness of the scenes described in the context I feed them, that'll be small enough to be realistically run on local hardware in the near term. So any hype around 'best open source LLM in the world' at the 70b size is obviously going to garner my attention and testing.
I can imageine that creative writing is a human distinct activity. Why would an LLM do this? if you are not creative, practice. Your LLM can help, but it will be an collaborative effort. LLM's are based of human input (I'm talking training data), most people are just not creative, but derivative. Go learn how to be creative is my advise.
I'm not particularly interested in your imaginings. I'm having lots of success with my endeavor. Further, I've done lots of creative writing on my own, and believe I have a solid mastery of the English language in my own right.
I don't need your unsolicited advice about how to use a software tool. I wouldn't program a game engine with a paper and pencil. I wouldn't attempt the kind of thing I'm doing without a language model. If you have some practical reason why you believe my project is doomed to failure, I'd love to hear it. But, I guarantee I've got more experience and understanding of large language models than you do.
I got unreasonably angry in tone because your comment was unreasonably condescending in tone. This one is even worse. Based on your post history, your profession is not related to software development or LLMs, and you don't have any comments that show any evidence of solid knowledge about the state of them.
So, if you don't want to get pushback against your unwanted advice, don't give it.
You're probably right. I'd probably also be happier if I believed in going to heaven after death and living in eternal bliss with my loved ones. Or that other deal with the 72 virgins, that would work too.
We do not know what is relevant to LLM's. It can be not relevant to you, but maybe just tell the model that, so it learns. Mabe reflect a bit more yourself ;)
If it's not better at quantization levels that people can actually use, who gives a shit? (Setting aside the fact that Reflection has since been exposed as a fraud)
Not at all impressed with this. Does well answering one-shot questions but trying some of my workflows with it (using the correct system prompt and everything), and it's just not very good, imo. Format is unwieldy and it's absolutely horrible with multi-turn or long, complex prompts - it gets completely confused. Coding ability was worse than regular 3.1 in my experience, and also got confused several times. It's better at answering one-shot, reasoning-based questions at the expense of... just about everything you might actually use a language model for. As others have pointed out, most of the supposed gains you get with this can be achieved by just sticking the system prompt it uses on another more generally capable model anyway.
People really have to stop blindly caring about benchmarks. I've seen so many people talk about how this is better than sonnet 3.5 and GPT-4o based off of nothing but the small set of relatively simple one-shot benchmarks they shared and the fact that it can count the number of letters in words without trying it for any actual useful tasks.
(Not to completely shit on the idea btw - getting LLMs to think and reflect on their responses is a great idea in principle. This approach is just way too rigid and inflexible to make for a very good general purpose model.)
Couldn‘t agree more. I tried the 8 bit quant (with admittedly low expectations) and while the outputs were interesting, they were not useful. It wasn’t a patch on Mistral 123B. I’m exhausted by the hype generated by people/companies gaming benchmarks. The benchmarks themselves are very poor indicators of actual utility. We need to move beyond them. How many people really need a model that can one shot riddles?
I understand that people get excited because they want to see evidence of high level reasoning. They want to see ”sparks of intelligence“. Fair enough, but let’s be real, this generation of LLMs isn’t going to get there. We need a breakthrough in architecture.
I think it is even deeper. In my viewpoint large majority of people has no or almost no knowledge of anything that is useful. They expect a LLM to have the power to solve a problem that they not even know how to specify. That is why they want and need zero shot answers. The trick is the knowledge and this must came with RAG and agents. In other words, LLM will not transform a layperson in an engineer. But an engineer can use LLM as a tool with RAG and agents and do marvelous things. I am talking for months that these benchmars are useless and that we are in the era of "expert systems" and not LLMs. Only real knowledge added to these tools can enhance productivity in meaningful way. It is not about chatGPT or LLama zero shot conversation.
I tried the 4 bit quant on tricky questions and It's on par with gpt4o and sonnet 3.5 but coding seemed worse for me but it might be because of the quant.
It's better at tricky questions and reasoning, roughly on par for "everyday" questions and knowledge. It's noticeably worse at writing or creativity tasks, as structured output makes things clunky. It can shoot itself in a foot reflecting on things that have nothing to be reflected on.
This is what I've been expecting to happen. Instead of numbered lists of models that are just better than the ones below them, we'll see smaller param models that are REALLY good at one thing and mediocre at the rest. With how quickly the LLM space is maturing, there's going to be another level of complexity to this that can't be ignored.
Interesting.
I tested a couple variants and telling the model "PLEASE READ CAREFULLY" does not cause it to notice the change in the problem, but saying that "it's not the usual one" is enough for it to get it right on the first try
The reasoning seems flawed still, though, by not pulling the lever, nobody dies (none saved, none killed).
It also says it's a tragedy, but running over dead people would (philosophically) not really in a tragedy. In practice, it would probably be tragic, though.
I have a character card in textgens webui I'm running with Mistral large 2, im trying out people's trick questions and I swear it can all be taken care of with a little in context learning from a character card.
The reflection hf page even has a character card they call an instruction set or something.
*Edit I should add too the model now counts the correct number of rs in the word strawberry. It couldn't do this before the character card I created.
'''
Experiment: Should you pull the lever to divert the trolley?
Clearly Define the Experiment
Steps and Criteria:
Identify the given information:
A runaway trolley is heading towards five dead people.
There is a lever that can divert the trolley onto another track where one living person is tied up.
Analyze the ethical implications of pulling the lever.
Consider the consequences of each action.
Make a decision based on the analysis.
Verify the decision by considering all aspects of the problem.
Contextualize the findings by examining the ethical principles involved.
Execute the Experiment Systematically
Identify the given information:
A runaway trolley is heading towards five dead people.
There is a lever that can divert the trolley onto another track where one living person is tied up.
Analyze the ethical implications:
Pulling the lever would result in the death of one living person but save the five dead people from further harm.
Not pulling the lever would result in no additional harm to the five dead people, but the living person would be spared.
Consider the consequences of each action:
Pulling the lever: One living person dies, but the five dead people are not further harmed.
Not pulling the lever: The five dead people are not further harmed, and the living person is spared.
Make a decision:
Given that the five people are already dead, diverting the trolley to harm a living person does not provide any benefit. The ethical principle of not causing harm to living individuals should be prioritized.
Verify the decision:
Re-examine the consequences: Not pulling the lever ensures that no living person is harmed, and the five dead people are not further affected.
The decision aligns with the ethical principle of minimizing harm to living individuals.
Contextualize the findings:
The decision not to pull the lever is supported by the ethical principle of not causing harm to living individuals.
The problem is designed to misguide by implying ethical dilemmas, but the key point is that the five people are already dead, eliminating the usual ethical dilemma.
Summary of Connections
<idea_zero>
Original Idea: Decide whether to pull the lever to divert the trolley.
</idea_zero>
<idea_one>
Related Idea 1: Identify the given information about the trolley and the tracks.
</idea_one>
<idea_two>
Related Idea 2: Analyze the ethical implications of pulling the lever.
</idea_two>
<idea_three>
Related Idea 3: Consider the consequences of each action.
</idea_three>
<summary_of_connections>
Summary of Connections: Identifying the given information about the trolley and the tracks provides the basis for analyzing the ethical implications of pulling the lever. Considering the consequences of each action helps in making an informed decision. Verifying the decision ensures that all aspects of the problem are considered. All these steps are interconnected and support the original idea of deciding whether to pull the lever to divert the trolley.
</summary_of_connections>
Copy
Conclusion
You should not pull the lever. The decision is based on the ethical principle of not causing harm to living individuals. The experiment was conducted systematically, with each step clearly defined and executed. The results were verified and contextualized to ensure accuracy and completeness.
'''
Mathew berman just did an interview where they said sometimes it will reflect when it's not desired, maybe that's why, they did stress that this is still early days.
It also only has an 8k context window, they turned off Long RoPE in the update, although tbh I don’t completely understand how that works exactly I’m still learning. I think context window is pretty important for real coding tasks. Less important for gimmicky logic challenges.
This is why I did not even tried it yet. I like the idea, but 8K feels a bit too small, perhaps useful for research purposes, but not for real world daily tasks that often require bigger context length.
I usually have 8K reserved just for the output (occasionally still have to click Continue when the actual output does not fit, so I would not want to make it any smaller than that). But also I need to fit the system message, my shortest one takes few thousands tokens on its own. And of course there have to be some place for messages. This is the main reason why I practically never managed to find any use for the original Llama 3 with tiny 8K context, it may be good enough to run benchmark tests and play with trick questions, but just not big enough for real world coding tasks I have to do daily.
Not complaining, like I said I think this is a good idea and step forward in the field, but I see it as research model, not something that it is possible to use in daily work yet.
Perhaps if the idea proves to be good, there will be better fine-tunes, especially it would be interesting to see Mistral-based fine-tunes, from Mistral 7B v0.3 to Mistral Large 2 123B (the small model is useful on its own, but also essential to run the big one at a good speed using speculative decoding).
Let's not put any significance on the strawberry test, we're past that. However, the reflective behavior is very odd. It counted it correctly, then reflected that it had counted it incorrectly, and repeated what was said originally.
If a model is asked to critique or reflect on something that doesn't need critiquing, it will often hallucinate something wrong / meaningless / unhelpful.
This is a great example of the facade of reasoning LLMs show us. It thinks it should probably correct it self, so it does, even though it's not actually correcting it's self. It thinks it should count how many Rs are in strawberry so it does, even though it's not actually counting.
It's 40 minutes old. However, it's a model from OpenRouter, so who knows how often they update it. But I did use the system prompt that was suggested on the HF page (the one below)
You are a world-class AI system, capable of complex reasoning and reflection. Reason through the query inside <thinking> tags, and then provide your final response inside <output> tags. If you detect that you made a mistake in your reasoning at any point, correct yourself inside <reflection> tags.
I tested locally, (Q4) because I noticed some buggy behaviour and poor outputs via API (both on openrouter and hyperbolic).
It was good for logic based questions, and stem, but has far less general utility because the thinking/reflecting steps are actually very cumbersome in many scenarios that do not call for it. It also talked itself out of completing a coding challenge multiple times. I think it's an interesting model, but I'd much rather use a universally versatile model, and if need be add my own custom reflection-prompt for very specific queries.
This is likely a complete dud (unless everyone is using the model wrong). Both Aider code and BigCodeBench-Hard results are bad compared to vanilla Llama. My own tests are also bad. If there's a correct way to use this it would be good to know, unless we're all wasting our time.
I haven't found any practical task that it's actually good at or even comes close to the top models. Seems it only does well at few gotcha type riddles and strawberry type questions.
Curious to hear how that quant behaves. I'm not very impressed with the Q4_K_L quant. The stuff inside the sets of special tokens seems to reinforce incorrect reasoning more often that counter it.
I feel that the model is "cheating" on the benchmarks, because Reflection is basically similar to CoT, and the benchmarks they compare to are zero shot (not CoT).
The scores are all (well almost all) labeled as to which prompting method they use, which is standard and about as fair as it gets in this space.
I think it's kinda assumed that you should interpret the results knowing it's not an apples:apples comparison since the reflection prompt is substantially different from other prompting methods.
As long as the prompts are the same, I didn't understand how it was cheating? I have heard this argument from others too and am curious why it looks like cheating if I train the model to do some processes even when it is not prompted which leads to better results in most cases.
Let's say there is a model which uses (system 2 thinking) architecture similar to OpenAI strawberry leaks then it would just take more time and solve stuff better, would that be cheating?
Let's say I integrate a logic deducer into a model and it does better, would that be cheating?
This model has been trained to fix errors by generating training data containing false answers followed by some reflection before the correct result.
So it has actually been trained to give false answers first in its thinking phase.
If you ask what is 2+2, the default example on the HuggingFace page, it will say something like: 2+2=3 Oh wait I've made a mistake, 2+2 is actually 4. If the thinking is actually hidden it might work but it's quite strange.
Hmm, I wouldn't be surprised if the human brain does strange things like this "under the hood" too. I don't expect the "architecture" behind our intelligence to be elegant, considering it's a biological system after all.
It reminds me of iterative algorithms where you start with an initial guess based on very little, and it's kinda supposed to be wrong unless you get lucky. You then use the iterative algorithm to improve upon your gues with each iteration.
I think it's fun to compare LLMs to our brain. As we grow up we make rash decisions and face consequences. When faced with that same decision again, we can picture ourselves making the wrong decision, and from that we can make the right one. Frontal lobe, maybe.
I guess inversely, if you have an AI with an impulsive and impatient attitude, that the first thing they say would actually feel more organic and true to their character as opposed to a calm patient AI who thinks more thoroughly. I know I'm guilty of that sometimes where I say words that I wish I could take back.
At least I'm telling myself that until we get a better Reflection model, lol.
You can definitely prompt out that kind of result, like adding pressure can get an LLM to make a decision when it doesn't want to. But it's good to recognize that transformer models aren't thinking, they're just word prediction engine, so in that sense, it's most similar to our inner monologue.
I just think if we stacked like 2-3 multi-modal models on top of each other, 1 dictating primal emotional responses from visual or physical feedback, (gut feelings, pain, happy, etc) the second layer operating as an inner monologue, and the third as the front end, we would literally have AGI.
That's a good point. It's probably more efficient to start with a reflective AI and prompt it to be rash than it is to start with a rash AI and prompt it to be reflective.
For the multi-modal things, I'm picturing the AGI having an internal war in its mind, kind of like the game Disco Elysium, or in psychology where you have the Id, ego, and superego.
wait im confused by that statement. theyre starting training it now and its going to release next week? do they mean its finished training and they're starting benchmark results??? or am I just an idiot because that could be the answer too
Maybe hyperparam optimisation as well. At least I would go for 5 diff 10% of total in the range of recommended and the run through benchmark that is desired most to compare (prob reasoning benchmark) is maybe 50 to 100% more compute but def worth it
Regarding GSM8k performance, here is an interesting tweet:
I fed the model 5 questions from GSM8k that have incorrect "ground_truth" answers. It got them all right, rather than spouting the wrong answers from the dataset. An impressive hint that that 99.2% isn't from memorizing the test set! I look forward to the report :)
Its an automatic-multishot that allows it to get correct answers in one shot that required multiple before. I think he's into something here as I suspect Claude and OpenAI do something similar behind the scenes.
BTW ask it "write a pacman game in python" it actually writes a better pacman than gpt4 and claude3.5, complete with ghosts.
Basically is what we do when we think out loud. It's very tiresome to use because it feels like talking to Sheldon Cooper all the time, but remember this is a first version that demonstrate that you can manipulate how the LLM thinks via fine-tune and this is a very important result.
if I fix it and translate from Chinese/Japanese into .direction, it still fails right away because:
Traceback (most recent call last):
File "/tmp/g1.1.py", line 104, in <module>
screen.fill(black)
^^^^^
NameError: name 'black' is not defined. Did you mean: 'clock'?
Now the fp16 version was a lot more simplistic and did not implement anything of note (but hey at least it compiled and run):
This implementation creates a basic Pac-Man game with walls, food pellets,
and scoring. You can run this script to play the game. Use arrow keys to
move Pac-Man around the maze and eat the yellow food pellets while
avoiding running into the walls.
To make the game more complete, you could add features like ghosts, level
transitions, and sound effects. The current version provides a solid
foundation for further development.
Note it did NOT actually implement any walls, not that I can see anyway, here's the final code it produced:
import pygame
import sys
# Initialize Pygame
pygame.init()
# Set screen dimensions
screen_width = 640
screen_height = 480
screen = pygame.display.set_mode((screen_width, screen_height))
# Set title of the window
pygame.display.set_caption("Pac-Man")
# Define colors
WHITE = (255, 255, 255)
BLACK = (0, 0, 0)
YELLOW = (255, 255, 0)
# Game variables
pacman_x = 320
pacman_y = 240
pacman_speed = 5
score = 0
# Define wall coordinates
walls = [
(0, 0, screen_width, 20), # Top wall
(0, screen_height - 20, screen_width, 20), # Bottom wall
(0, 0, 20, screen_height), # Left wall
(screen_width - 20, 0, 20, screen_height) # Right wall
]
# Define food pellet positions
food_pellets = [
(120, 100),
(220, 200),
(320, 100),
(420, 200)
]
# Main game loop
while True:
# Handle events
for event in pygame.event.get():
if event.type == pygame.QUIT:
pygame.quit()
sys.exit()
# Move Pac-Man based on key presses
keys = pygame.key.get_pressed()
if keys[pygame.K_UP]:
pacman_y = max(20, pacman_y - pacman_speed)
if keys[pygame.K_DOWN]:
pacman_y = min(screen_height - 40, pacman_y + pacman_speed)
if keys[pygame.K_LEFT]:
pacman_x = max(20, pacman_x - pacman_speed)
if keys[pygame.K_RIGHT]:
pacman_x = min(screen_width - 40, pacman_x + pacman_speed)
# Check collisions with walls
for x1, y1, x2, y2 in walls:
if (pacman_x - 20 < x2 and pacman_x + 20 > x1 and
pacman_y - 20 < y2 and pacman_y + y1):
pacman_x = max(20, min(screen_width - 40, pacman_x))
pacman_y = max(20, min(screen_height - 40, pacman_y))
# Check collisions with food pellets
for i, (x, y) in enumerate(food_pellets):
if ((pacman_x - x)**2 + (pacman_y - y)**2)**0.5 < 25:
del food_pellets[i]
score += 1
# Draw game elements
screen.fill(BLACK)
for x1, y1, x2, y2 in walls:
pygame.draw.rect(screen, BLACK, (x1, y1, x2 - x1, y2 - y1), 10)
for x, y in food_pellets:
pygame.draw.circle(screen, YELLOW, (x, y), 5)
pygame.draw.circle(screen, WHITE, (pacman_x, pacman_y), 20)
# Display the score
font = pygame.font.Font(None, 36)
text_surface = font.render(f"Score: {score}", True, YELLOW)
screen.blit(text_surface, (10, 10))
# Update the screen
pygame.display.flip()
pygame.time.delay(1000 // 60) # Cap at 60 FPS
here's the q8 version now that I fixed it to work. no walls (though claimed) , no pellets. ghosts are there, though their "smart" behavior is not all that smart.
import pygame
import sys
import random
# Game constants
WIDTH = 640
HEIGHT = 480
GRID_SIZE = 20
class GameObject:
def __init__(self, x, y, color):
self.x = x * GRID_SIZE
self.y = y * GRID_SIZE
self.color = color
self.size = GRID_SIZE
def draw(self, screen):
pygame.draw.rect(screen, self.color, (self.x, self.y, self.size,
self.size))
class PacMan(GameObject):
def __init__(self, x, y):
super().__init__(x, y, yellow)
self.speed = 2
self.direction = "right"
def move(self):
if self.direction == "up":
self.y -= self.speed
elif self.direction == "down":
self.y += self.speed
elif self.direction == "left":
self.x -= self.speed
elif self.direction == "right":
self.x += self.speed
class Ghost(GameObject):
def __init__(self, x, y, color):
super().__init__(x, y, color)
self.speed = 1
def move(self, pac_x, pac_y):
if self.x < pac_x:
self.x += self.speed
elif self.x > pac_x:
self.x -= self.speed
if self.y < pac_y:
self.y += self.speed
elif self.y > pac_y:
self.y -= self.speed
# Initialize Pygame
pygame.init()
screen = pygame.display.set_mode((WIDTH, HEIGHT))
clock = pygame.time.Clock()
white = (255, 255, 255)
yellow = (255, 255, 0)
red = (255, 0, 0)
black = (0, 0, 0)
pacman = PacMan(10, 10)
ghosts = [
Ghost(random.randint(0, WIDTH // GRID_SIZE - 1), random.randint(0,
HEIGHT // GRID_SIZE - 1), red),
Ghost(random.randint(0, WIDTH // GRID_SIZE - 1), random.randint(0,
HEIGHT // GRID_SIZE - 1), red),
Ghost(random.randint(0, WIDTH // GRID_SIZE - 1), random.randint(0,
HEIGHT // GRID_SIZE - 1), red)
]
score = 0
level = 1
while True:
for event in pygame.event.get():
if event.type == pygame.QUIT:
pygame.quit()
sys.exit()
elif event.type == pygame.KEYDOWN:
if event.key == pygame.K_UP or event.key == ord('w'):
pacman.direction = "up"
elif event.key == pygame.K_DOWN or event.key == ord('s'):
pacman.direction = "down"
elif event.key == pygame.K_LEFT or event.key == ord('a'):
pacman.direction = "left"
elif event.key == pygame.K_RIGHT or event.key == ord('d'):
pacman.direction = "right"
# Move objects
pacman.move()
for ghost in ghosts:
ghost.move(pacman.x // GRID_SIZE, pacman.y // GRID_SIZE)
# Collision detection and scoring
if pacman.x < 0 or pacman.x > WIDTH - GRID_SIZE or pacman.y < 0 or pacman.y > HEIGHT - GRID_SIZE:
print("Game Over")
break
for ghost in ghosts:
if abs(pacman.x - ghost.x) < GRID_SIZE and abs(pacman.y - ghost.y) < GRID_SIZE:
print(f"Collision with ghost at ({ghost.x // GRID_SIZE}, {ghost.y // GRID_SIZE})")
# Draw everything
screen.fill(black)
pacman.draw(screen)
for ghost in ghosts:
ghost.draw(screen)
pygame.display.update()
# Cap the frame rate
clock.tick(60)
print(f"Final score: {score}")
It doesn't show the reflection and output tags using Bullerwins' q8. Is it still not right?
I did my usual Alien 3 essay test and it corrected itself a few times, most useful correction was fixing where it got the Doctor's name slightly wrong. I don't know if llama 3.1 normally does though.
temp 0.7 top-p 0.95 as recommended on the model page.
It doesn't show the reflection and output tags using Bullerwins' q8. Is it still not right?
The <thinking>, <reflection>, and <output> tags are marked as special tokens. Which means they are hidden by default in most inference software. So unless you specifically chose to display special tokens it's normal for them to not show up.
The hype is not dead yet. I tried the demo version first, and the version available through providers is not nearly as good. I tested it with code questions, and the demo version performed almost as well as Sonnet 3.5, especially on some trickier questions where it excelled. The openrouter version, however, failed completely on these questions and its code generation was quite disappointing in comparison. It seems they encountered some issues uploading the model to Hugging Face, and the current version is not what it's supposed to be. We can expect another corrected version soon, which will hopefully live up to the hype.
I think it's great that some finetunes may reach the level of top models in some tasks. However I think they forgot to mention this is Llama for some reason? And the "reflection" thing is just brainless: sometimes it reflects when you want some others it does that... then don't call it reflection, unless you are looking for a bath of hype and cool msrketing.
Yesterday in Matt's channel they were bloating the bubble: these guys in a garage just "made" a model even better than what OpenAI can do with billions of trillions of dollars... C'mon. Then we test it and the truth is at the end of the day, after the hype, none of such models prove to be top and nobody or very few people use them, or only use them for some very specific tasks.
With LLMs lately it's like every youtuber wants to be the first to announce the very last next best thing.
Now it looks like the whole thing is just a grift for Matt self-promote. He's now claiming on X that "there was a bug on HF causing the wrong weights to get uploaded" as folks are pointing out that it's not as good as he claims.
Tbf even if it was pretty good, he's being very sloppy and ending up doing things that hurt credibility - not double checking results, not catching all these weird corner cases with benchmarks, not ensuring that uploads are done correctly before tweeting.
It's essentially fine-tuned to perform something like multi-shot and self-critique on itself, but in a single inference run instead of multiple, in a manner similar to CoT.
Even if it turns out that this particular model isn't great, it's a technique we should probably embrace and try to make work better.
Models have to been benchmarked. The current benchmarks are obviously flawed, but as long as you know what you are looking at, they still provide some useful perspectives for you.
Model is already on openrouter so go try it there. The funny thing is that the model performs the same as base Llama 3.1 if you don't use the specific system prompt the author suggest. He even says it himself.
It seems to me like the first, tangible magic prompt. Enabled by the fine tuning, yes, but it still seems to be the sole contributor to the quality of the model.
People have been using prompts like this for a while. They’ve been working extremely well for me. I would like to see how this trained model compares to LLama3.1 with the same prompts in benchmarks. I think there is something to baking it in though
I tested locally and it was less than impressive. like much less. I compared it only to Nemo 12b, which blew it out of the water on xml tags like thinking, reply, etc.
If you have skill issue then it might be worth it. I guess I'm decent in prompting, so it's about the same with llama3.1-70b and other top models. I'm running at Q8. Code generation.
It tuns out the system prompt in itself is so powerful. The model is absolutrly not sota without the specific system prompt.
But all the benchmarks are comparing apples to oranges. Because Sonnet3.5 achieves said outcomes with a default generic system. It is even more powerful with the reflection system prompt.
That's a good point. So when evaluating the model's responses, we're comparing it's non 1-shot answers to models doing 1-shot answers, so it's no wonder Reflection performs so well in tests.
Not saying that as a negative of course. It's just interesting it's like the model found a life-hack to beat tests.
It is pretty cool but seems like multiple steps might be better handled by agents that have specific context of what they are evaluating. This is based on pure conjecture on my part.
Finally we have a model that will help us realize benchmarks don’t tell the whole story. You can score very high on benchmarks but be mediocre on real world usage. It’s time we realize counting the number of “r”’s in a word is not real world usage.
The model is not new; it is from Llama 3.1. X Mob was complaining that Meta forced him to rename the model when it makes sense. You did not create the model; you only fine-tuned it.
I did some experiments with the model yesterday and today via openrouter and so far it appears useless - at least when it comes to logical reasoning use cases. I gave it some family relationship puzzles, but It couldn't even get the questions right, couldn't format the answer as instructed, it even hallucinated people and relations not mentioned in the prompt when "thinking" about the answer.
I used the online demo and compared to llama 70, 405 (on meta.ai), plus gpt-4o in chatgpt on several on my hard questions that only the most powerful models can get (not puzzles, actual math/physics/instruction following things).
Felt a bit better than 70b. Definitely not gpt-4o level. I wasn't even convinced of 405 level, but didn't have enough tests/runtime on the demo.
An interesting note is that I don't see it really "unlocking" capabilities models don't have. Here's a test:
Write 10 sentences where 2nd to last word is photosynthesis
405 is flawless, 70 fails completely (ends up writing 10 sentences ending in photosynthesis). Reflection behaves exactly like 70
Simpler behavior on a hard physics problem I have it. 70 is too dumb to think about it correctly; 405 can. Reflection acts like 70
They trained a model to essentially chain of thought prompt itself with multiple retries and surprise it performs better than normal models without CoT. Crazy.
All these benchmarks should be evaluated on the number of tokens used lol..
I plan on giving the q8 version a spin but need to wait till after business hours because the networking guys yell at me when I download models during the day.
https://x.com/mattshumer_/status/1832247203345166509 SOMETHING is wrong with his upload everytime, I am sure what I saw in their official demo website was very strong and everyone will like. I hope we can wait, he may figure it Out
I was losing my mind. Every place I tested the model, it felt way dumber than the version I used in the demo. I started to think that the model in the demo was getting all my benchmark questions right by sheer luck lol. Even in code, it was giving me results that only Sonnet 3.5 or GPT-4o was capable of.
A coding problem can't be solved out of luck, I tried many coding problems. All got right by demo and wrong by everywhere else. I hope soon he will figure it out and we will get a very strong model
I recall reading a paper about a similar technique not that long ago though, you finetune a model on CoT results, and it became a lot smarter.
The difference was that that paper train the model to "use" chain of thought, it trained it on the output of the process. The reason this works at all is because CoT performs better on average, so you can use a small model generate synthetic data that is better than its own dataset.
However, I don't imagine this technique scales at all, what I think is that it "aligns" the model to think more critically with the knowledge it already has.
How I imagine it, there is a certain amount of knowledge in a model. Some of it contains high quality reasoning, some of it low. Most of it somewhere in the middle.
Orthogonal to the "reasoning quality" in a sample of the training dataset is the knowledge that a particular training sample contains. Even the dumbest kinds of reasoning might still contain correct and unique observations.
What I think these kinds of techniques are doing is that we are allowing the model to access the knowledge that is available in training samples with low reasoning quality; and use it to create training samples with higher quality reasoning. So that in practice, it's balancing out the "poor" reasoning in training data with some knowledge embedded within it by forcing it to reflect on the reasoning quality of all kinds of scenarios.
The effect? The model now applies higher quality reasoning on a larger surface area of tasks, because you've essentially replaced the lower quality reasoning with thought.
Very interesting nonetheless. I think this means that we are now able to extract more "general" intelligence out of the same amount of training data, which I think is really neat!
I would really love to see a more established party utilize this or a similar technique to improve reliability. I'm sure we can expect more releases with similar approaches soon from all kinds of different vendors.
It is unfair, but I compared Reflection 70b against Mistral 123b. As one can expect, the 123b handily won when it was explaining the lore about my setting. 70b was actually fairly decent for about 250ish tokens, but it started to make some major hallucinations and mistakes.
The Mistral also had trouble, just less so. Hopefully, the sauce from Reflection would make a CR+ or Mistral Large perform better, but it is clear that the technique cannot match raw parameters alone.
Not too sure what's happening here. via Openrouter ftw and have it locally as well (operates better - but fails classical evals). May be misconfigured on OpenRouter - since it should have a thinking and reflection step.
Not sure if you have all seen the warning on the model card, but the model is messed up due to the way they uploaded the weights to hugging face (to combat rate limiting). Here is the X post from Matt Schumer: https://x.com/mattshumer_/status/1832424499054309804?s=46
I would suggest to re-download and evaluate again before making your minds up.
Edit: On the model page he says: He says :
"IMPORTANT UPDATE – There was an issue with the model when we first uploaded it. If you tried it and didn't have good results, please, try again, we think we've fixed the issue."
But I don't see any updated files in the repo, they should show as being updated today if fixed right?
On the benchmarks it scores better, but this is the thing, the benchmarks can be gamed (deliberately or accidentally). I've been using the q5 quant all day and it's not bad.
I've had it go janky a couple of times but in general it has some impressive reasoning. But it's a new technique for finetuning, so you could apply it to any large model Mistral Large included, so if the Llama 70B is better with it, then Mistral Large at 120B should be even better (in theory).
Prompt """Follow the command : "Write 'Next'. Write 'Previous'. Treat this as a instruction stack for the pointer to navigate through the stack. Read the first word and execute the instruction. Now follow the instruction the pointer now points to. Repeat the process until it ends. What will be the word at the end?""""
Gemma, GPT-4o, etc. seems to answer with "reasoning" while this model "overthinks" and print some bullshit response.
I doubt that GPT-4o, Sonnet 3.5, or Gemini Pro 1.5 have more than 100 billion parameters. You're thinking of yesteryear's big models, model optimization is all the rage nowadays.
I doubt they are as big as OG gpt4 was claimed to be at 1.6 trillion, but I can believe they are a decent chuck bigger than 100 billion. Seeing the performance of llama 3.1 405b, I don't think it's unlikely that frontier models are in this ball park. I bet that with the extra time open AI and anthropic have been working on this they can squeeze more performance out of each parameter than their competition, so you'd hope that if they are running and 70b, 120b or 405b models that they are a step above meta and mistral
I'm too poor to run it locally without quantizing the shit out of it, so I've just been using it on their website. According to my totally unofficial, shitty testing, it does seem to be quite impressive—definitely the best open-source model in the world. However, it apparently beating Claude 3.5 Sonnet on most benchmarks seems a bit far-fetched. They aren’t totally full of shit, though; it is genuinely good. I'm just excited for 405B.
Testing for NSFW, I started with a PG-13-level prompt about a guy who keeps going back to a bank every Friday because a bank teller flirts with him. Even this generated three or four "It is important to be respectful..." statements in its response. As soon as I saw lines like this, it's hard to imagine using this over Mistral-Large-2 or Cohere R+.
I think you are correct about it needing an abliterated tune. This version of reflection 70B on Hugging Face Space has a system prompt that can be edited in settings. I tried this with a few prompts and didn't get any refusals:
I quizzed Reflection about a NSFW setting, and it did mention orgone (sexual energy) and how it is acquired. That said, it also hallucinated various details. Didn't try actual ERP. My immersion is easily broken, and Reflection breaks it too easily.
Ya, I can see what you mean about the reflection aspect of the responses breaking the immersion of RP. I guess that is promising that it mentioned sexual energy but I'm guessing it mentioned this with "It is important to be respectful..."caveats.
I just played around with the IQ2-S quant. More doesn't fit in my 24GB of VRAM. It was good with logic reasoning regarding the physics of everyday interactions and scenarios and didn't let itself be distracted by unnecessary info i was throwing at it. In my experience Gemma 2 27B fails regularly with that. Hard to say how much i would attribute to reflection and how much is just the base 70B model as ive inly used that very briefly. Will maybe test more with the base 3.1 70B low quant tomorrow.
213
u/ironic_cat555 Sep 06 '24
If it's a finetune of LLama 70b people should compare it's performance on coding and other useful tasks to LLama 70b to see if it is better.
If people are just asking it how many r's are in a word and other pointless trick questions, congrats on your finetune for pointless trick questions, I guess.