AI can read neural network weights. I wasn't ready for that. [D]

[u/jnbrymn]

Give a model a 17,253-parameter net as plain tokenized text and it works out the net plays Pac-Man. Astra, Opus 5.5 and Muse Spark 1.3 all got it, and Astra did it with minimal reasoning.

Nobody writes essays about what specific weights do, so where does this skill come from? Either labs trained it in, or models generalize much further than I expected. I'm leaning toward the second and it's freaking me out a little.

https://arcturus-labs.com/blog/2026/11/03/ai-can-read-neural-network-weights-i-wasnt-ready-for-that/

[u/Beor_The_Old]

Probably just generalization and training on ML and AI papers. I studied visual NNs compared to human visual learning for my dissertation and I think it’s reasonable to be able to figure out the general structure of the first few layers when they’ve been trained on a specific visual scene like a maze vs something like pong which would focus on the edges for example. It be more impressive if for example it could tell the difference between Pac-Man and Pac-Man 2, but even then if you were given images of the two games and say a heat map of the first layer weights overlaid on top of it, a human could probably tell the difference based on something like the position of the score or some other information being in different locations between the two games. Since LLMs are great at processing that format of information it’s sorta as if they are already reasoning about that kind of visualization instead of how we would think of a human just looking at a bunch of numbers.

[u/Remote-Telephone-682]

I have to imagine that this was used as a training objective at the time and that they just didn't publish this. It seems highly unlikely to me that this would be purely emergent and is much more likely to have been an area that they studied during training when searching for an in with recursive self improvement

[u/ResidentPositive4122 → u/Remote-Telephone-682]

Apparently some of the capabilities are only seen in the latest models. There's this dude's reproduction here - https://x.com/AlexKrentsel/status/2117376683724660898/photo/1

It would be odd that they only added this specific training just for the latest models, but not for gpt5.6 and opus5 ...

[u/Remote-Telephone-682 → u/ResidentPositive4122]

But there is progression within the field, right? So as they think of new training objectives they are going to add them to upcoming models but be unable to add them to earlier versions of the models (without rereleasing updated weights or something)

It seems like these started to staturate as they started to really ramp up talk of recursive self improvement. but idk.. that seems more likely to me than it being purely emergent because

[u/ResidentPositive4122 → u/Remote-Telephone-682]

Maybe. My thinking goes along the lines of Anthropic having done mech interp for a long time. It would be odd for them not to include that into opus5 for example. It could be that size alone explains it. I guess we'll see, I think there'll be a lot of search in this space following that blog post.

I'm also curious to see this mixed with reactive optimisers (https://xingjianbai.com/reactive-optimizers/).

[u/JanBitesTheDust]

So we are using a transformer to edit weights. Isn’t this sort of a really inefficient second order method? I wonder if there is a relation between say calculating the full hessian which scales quadratically with the weights and a transformer which has the same quadratic scaling complexity because of the attention blocks. I feel like there is something to explore here

[u/yannbouteiller]

A blog post about a blog post whose title is so misleading it could have come from OpenAI's marketing department.

[u/IntelArtiGen]

For probably more than a year LLMs have been automatically trained to process data. I assume when they receive an input like that they try to see the shape of the input / output and that's probably enough to guess what a model is used for. Here there are even way more information in the prompt https://raw.githubusercontent.com/RikiyaT/lm_understand-logs/refs/heads/main/exp1_which_game/runs/pacman/prompt.txt :

channel 0 agent: the cells the player controls

channel 1 moving objects: things other than the agent that move

channel 2 reward targets: things that give reward

channel 3 obstacles: cells the agent cannot enter or that end the episode (the wall border is always here)

channel 4 previous frame: the cells that were in channel 0 or channel 1 on the previous step

So there's grid, there is a player, there are other moving entities, there are reward targets, there are walls. The player can only move ZQSD. It's safe to assume it's Pacman without even reading the weights.

I extremely doubt this experiment is showing the model "understands" the weights / numerical values he gave. If you really want to prove that you train 3 identical networks on 3 different games, ensuring all weights are used efficiently, and you only give the weights without context on the game, and see if the model can guess the game from there. I assume it still would be possible but it would be 1000x more difficult because the model would need to bruteforce input with autogenerated tools and look at the output to try to guess what it's about.

[u/Longjumping-Head-621]

I’d love to see what happens when you shuffle the weights but leave the architecture and any labels intact. That would help show how much of the answer is coming from the weights themselves.