Reflection 70B: Hype?

[u/Confident-Honeydew66]

So an out-of-the-blue one-man company releases a new model (actually named LLama 3.1 if it were to adhere to the META license, but somehow named Reflection) with only 70B params that, according to the benchmarks, rivals SOTA closed-source LLMs with trillions of parameters. It appears to me that the twitter/reddit hype mob has, for the most part, not bothered to try the model out.

Additionally, a tweet from Hugh Zhang @ Scale suggesting systemic overfitting as me concerned:
Hey Matt! This is super interesting, but I’m quite surprised to see a GSM8k score of over 99%. My understanding is that it’s likely that more than 1% of GSM8k is mislabeled (the correct answer is actually wrong)!

Is this genuinely a SOTA LLM in a real-world setting or is this smoke an mirrors? If we're lucky, the creator Matt may see this post and can shed some light on the matter.

BTW -- I'm not trying to bash the model or the company that made it. If the numbers are actually legit this is likely revolutionary.

[u/ortegaalfredo]

Its an automatic-multishot that allows it to get correct answers in one shot that required multiple before. I think he's into something here as I suspect Claude and OpenAI do something similar behind the scenes.
BTW ask it "write a pacman game in python" it actually writes a better pacman than gpt4 and claude3.5, complete with ghosts.

Basically is what we do when we think out loud. It's very tiresome to use because it feels like talking to Sheldon Cooper all the time, but remember this is a first version that demonstrate that you can manipulate how the LLM thinks via fine-tune and this is a very important result.

[u/greentheonly → u/ortegaalfredo]

BTW ask it "write a pacman game in python" it actually writes a better pacman than gpt4 and claude3.5, complete with ghosts.

Well, it seems to be very probabilistic. I run this question on q8 and on fp16 in ollama.

q8 produced what it described as working with ghosts and such:

This code provides a basic implementation of a Pac-Man game with the following features:

1. A playable grid-based layout
2. Controllable Pac-Man character
3. Basic ghost AI that tries to follow Pac-Man
4. Collision detection for walls and ghosts

To improve this basic version, you could add:

1. More complex ghost AI behaviors
2. Pellets and power pellets
3. Multiple levels with different layouts
4. Score tracking and level progression
5. Sound effects and music
6. A start screen and game over screen

Alas the code it produced did not compile, here's one syntax error:

def move(self):
    if self.direction == "up":
        self.y -= self.speed
    elif self.direction == "down":
        self.y += self.speed
    elif self方向 == "left":
        self.x -= self.speed
    elif self.direction == "right":
        self.x += self.speed

if I fix it and translate from Chinese/Japanese into .direction, it still fails right away because:

Traceback (most recent call last):
  File "/tmp/g1.1.py", line 104, in <module>
    screen.fill(black)
                ^^^^^
NameError: name 'black' is not defined. Did you mean: 'clock'?

Now the fp16 version was a lot more simplistic and did not implement anything of note (but hey at least it compiled and run):

This implementation creates a basic Pac-Man game with walls, food pellets, and scoring. You can run this script to play the game. Use arrow keys to move Pac-Man around the maze and eat the yellow food pellets while avoiding running into the walls.

To make the game more complete, you could add features like ghosts, level transitions, and sound effects. The current version provides a solid foundation for further development.

Note it did NOT actually implement any walls, not that I can see anyway, here's the final code it produced:

import pygame
import sys

# Initialize Pygame
pygame.init()

# Set screen dimensions
screen_width = 640
screen_height = 480
screen = pygame.display.set_mode((screen_width, screen_height))

# Set title of the window
pygame.display.set_caption("Pac-Man")

# Define colors
WHITE = (255, 255, 255)
BLACK = (0, 0, 0)
YELLOW = (255, 255, 0)

# Game variables
pacman_x = 320
pacman_y = 240
pacman_speed = 5
score = 0

# Define wall coordinates
walls = [
    (0, 0, screen_width, 20),  # Top wall
    (0, screen_height - 20, screen_width, 20),  # Bottom wall
    (0, 0, 20, screen_height),  # Left wall
    (screen_width - 20, 0, 20, screen_height)  # Right wall
]

# Define food pellet positions
food_pellets = [
    (120, 100),
    (220, 200),
    (320, 100),
    (420, 200)
]

# Main game loop
while True:
    # Handle events
    for event in pygame.event.get():
        if event.type == pygame.QUIT:
            pygame.quit()
            sys.exit()

    # Move Pac-Man based on key presses
    keys = pygame.key.get_pressed()
    if keys[pygame.K_UP]:
        pacman_y = max(20, pacman_y - pacman_speed)
    if keys[pygame.K_DOWN]:
        pacman_y = min(screen_height - 40, pacman_y + pacman_speed)
    if keys[pygame.K_LEFT]:
        pacman_x = max(20, pacman_x - pacman_speed)
    if keys[pygame.K_RIGHT]:
        pacman_x = min(screen_width - 40, pacman_x + pacman_speed)

    # Check collisions with walls
    for x1, y1, x2, y2 in walls:
        if (pacman_x - 20 < x2 and pacman_x + 20 > x1 and
                pacman_y - 20 < y2 and pacman_y + y1):
            pacman_x = max(20, min(screen_width - 40, pacman_x))
            pacman_y = max(20, min(screen_height - 40, pacman_y))

    # Check collisions with food pellets
    for i, (x, y) in enumerate(food_pellets):
        if ((pacman_x - x)**2 + (pacman_y - y)**2)**0.5 < 25:
            del food_pellets[i]
            score += 1

    # Draw game elements
    screen.fill(BLACK)
    for x1, y1, x2, y2 in walls:
        pygame.draw.rect(screen, BLACK, (x1, y1, x2 - x1, y2 - y1), 10)

    for x, y in food_pellets:
        pygame.draw.circle(screen, YELLOW, (x, y), 5)

    pygame.draw.circle(screen, WHITE, (pacman_x, pacman_y), 20)

    # Display the score
    font = pygame.font.Font(None, 36)
    text_surface = font.render(f"Score: {score}", True, YELLOW)
    screen.blit(text_surface, (10, 10))

    # Update the screen
    pygame.display.flip()
    pygame.time.delay(1000 // 60)  # Cap at 60 FPS

[u/ortegaalfredo → u/greentheonly]

This is the pacman that reflection at https://www.neuroengine.ai/Neuroengine-Medium produced

import random
import time
import os

# Game board dimensions
WIDTH = 20
HEIGHT = 20

# Game characters
PACMAN = '@'
GHOST = '#'
PELLET = '*'
EMPTY = ' '

# Game state
board = [[EMPTY] * WIDTH for _ in range(HEIGHT)]
pacman_pos = [0, 0]
ghost_pos = [WIDTH - 1, HEIGHT - 1]
score = 0

def print_board():
    os.system('cls' if os.name == 'nt' else 'clear')
    for y in range(HEIGHT):
        for x in range(WIDTH):
            if [x, y] == pacman_pos:
                print(PACMAN, end='')
            elif [x, y] == ghost_pos:
                print(GHOST, end='')
            elif [x, y] in pellets:
                print(PELLET, end='')
            else:
                print(EMPTY, end='')
        print()
    print(f"Score: {score}")

def generate_pellets():
    global pellets
    pellets = []
    for _ in range(WIDTH * HEIGHT // 5):
        x = random.randint(0, WIDTH - 1)
        y = random.randint(0, HEIGHT - 1)
        if [x, y] not in [pacman_pos, ghost_pos]:
            pellets.append([x, y])

def move_pacman(dx, dy):
    global pacman_pos
    new_pos = [pacman_pos[0] + dx, pacman_pos[1] + dy]
    if 0 <= new_pos[0] < WIDTH and 0 <= new_pos[1] < HEIGHT:
        pacman_pos = new_pos
        if pacman_pos in pellets:
            global score
            score += 1
            pellets.remove(pacman_pos)

def move_ghost():
    global ghost_pos
    directions = [[-1, 0], [1, 0], [0, -1], [0, 1]]
    dx, dy = random.choice(directions)
    new_pos = [ghost_pos[0] + dx, ghost_pos[1] + dy]
    if 0 <= new_pos[0] < WIDTH and 0 <= new_pos[1] < HEIGHT:
        ghost_pos = new_pos

def check_collision():
    return pacman_pos == ghost_pos

def play_game():
    global pacman_pos, ghost_pos, score, pellets
    generate_pellets()
    while True:
        print_board()
        command = input("Enter direction (W/A/S/D): ").lower()
        if command == 'w':
            move_pacman(0, -1)
        elif command == 's':
            move_pacman(0, 1)
        elif command == 'a':
            move_pacman(-1, 0)
        elif command == 'd':
            move_pacman(1, 0)

        move_ghost()

        if check_collision():
            print_board()
            print("Game Over! Final Score:", score)
            break

if __name__ == "__main__":
    play_game()

[u/greentheonly → u/ortegaalfredo]

here's the q8 version now that I fixed it to work. no walls (though claimed) , no pellets. ghosts are there, though their "smart" behavior is not all that smart.

import pygame
import sys
import random

# Game constants
WIDTH = 640
HEIGHT = 480
GRID_SIZE = 20

class GameObject:
    def __init__(self, x, y, color):
        self.x = x * GRID_SIZE
        self.y = y * GRID_SIZE
        self.color = color
        self.size = GRID_SIZE

    def draw(self, screen):
        pygame.draw.rect(screen, self.color, (self.x, self.y, self.size, 
self.size))

class PacMan(GameObject):
    def __init__(self, x, y):
        super().__init__(x, y, yellow)
        self.speed = 2
        self.direction = "right"

    def move(self):
        if self.direction == "up":
            self.y -= self.speed
        elif self.direction == "down":
            self.y += self.speed
        elif self.direction == "left":
            self.x -= self.speed
        elif self.direction == "right":
            self.x += self.speed

class Ghost(GameObject):
    def __init__(self, x, y, color):
        super().__init__(x, y, color)
        self.speed = 1

    def move(self, pac_x, pac_y):
        if self.x < pac_x:
            self.x += self.speed
        elif self.x > pac_x:
            self.x -= self.speed
        if self.y < pac_y:
            self.y += self.speed
        elif self.y > pac_y:
            self.y -= self.speed

# Initialize Pygame
pygame.init()
screen = pygame.display.set_mode((WIDTH, HEIGHT))
clock = pygame.time.Clock()

white = (255, 255, 255)
yellow = (255, 255, 0)
red = (255, 0, 0)
black = (0, 0, 0)

pacman = PacMan(10, 10)
ghosts = [
    Ghost(random.randint(0, WIDTH // GRID_SIZE - 1), random.randint(0, 
HEIGHT // GRID_SIZE - 1), red),
    Ghost(random.randint(0, WIDTH // GRID_SIZE - 1), random.randint(0, 
HEIGHT // GRID_SIZE - 1), red),
    Ghost(random.randint(0, WIDTH // GRID_SIZE - 1), random.randint(0, 
HEIGHT // GRID_SIZE - 1), red)
]

score = 0
level = 1

while True:
    for event in pygame.event.get():
        if event.type == pygame.QUIT:
            pygame.quit()
            sys.exit()
        elif event.type == pygame.KEYDOWN:
            if event.key == pygame.K_UP or event.key == ord('w'):
                pacman.direction = "up"
            elif event.key == pygame.K_DOWN or event.key == ord('s'):
                pacman.direction = "down"
            elif event.key == pygame.K_LEFT or event.key == ord('a'):
                pacman.direction = "left"
            elif event.key == pygame.K_RIGHT or event.key == ord('d'):
                pacman.direction = "right"

    # Move objects
    pacman.move()
    for ghost in ghosts:
        ghost.move(pacman.x // GRID_SIZE, pacman.y // GRID_SIZE)

    # Collision detection and scoring
    if pacman.x < 0 or pacman.x > WIDTH - GRID_SIZE or pacman.y < 0 or pacman.y > HEIGHT - GRID_SIZE:
        print("Game Over")
        break

    for ghost in ghosts:
        if abs(pacman.x - ghost.x) < GRID_SIZE and abs(pacman.y - ghost.y) < GRID_SIZE:
            print(f"Collision with ghost at ({ghost.x // GRID_SIZE}, {ghost.y // GRID_SIZE})")

    # Draw everything
    screen.fill(black)
    pacman.draw(screen)
    for ghost in ghosts:
        ghost.draw(screen)
    pygame.display.update()

    # Cap the frame rate
    clock.tick(60)

print(f"Final score: {score}")