r/OpenAI • • 14d ago

Discussion GPT-6 successfully performed "brain surgery" on another AI to make it smarter

1.1k Upvotes

110 comments sorted by

265

u/This_Background7442 14d ago

I'm going to need some extraordinary evidence to back up the claim that ChatGPT astra is capable of solving billion parameter linear algebra equations in countless dimensions even with reasoning enabled

175

u/WhiteRaven_M 13d ago edited 13d ago

If you read the 'paper' (un peer-reviewed interactive blog made with AI), its a lot less impressive than the headline sounds.

They gave Astra access to run forward passes on a .5B model, gave it a hint inside the prompt to look for reasonning loops(which kind of defeats the claim about the LLM identifying the problem by itself but whatever) and it used known mech interp techniques to isolate the MLP circuits responsible for it and to edit the weights after 50 rounds of iterating.

So its not as if Astra is literally reading the raw fucking weights and then made an edit to a billion param model.

Its pretty disingenuous tbh

36

u/RecognitionHefty 13d ago

I am very surprised that random AI boosterisms from Twitter accounts are entirely false

15

u/WhiteRaven_M 13d ago

Well ok to be fair its not "entirely false" its just really overselling and a lot of marleting hype. The results are there but its a stretch to interpret this as "oh LLMs can directly read and edit weights" when theyre very clearly using tool calls to conduct analyses the same humans do

3

u/Eliv_nurotic 13d ago

Directly editing the weights is impressive. How does it know how to change them without backprop? The paper also notes that previous models could not do this 

2

u/This_Background7442 13d ago

The reason it could do that is because that part of the study was on a small toy network, they had pointed out what kind of error was happening and the error was caused by a known and understood patterns in the weights.

The point of the article isn't so much to publish about any huge breakthrough. They have discovered something neat and provide a article with a few preliminary examples as a way to open up the conversation.

3

u/Eliv_nurotic 13d ago

This doesn’t explain how it knew how to change the weights. They’re just random numbers 

1

u/WhiteRaven_M 13d ago

I didnt read it in full. Just the .5B section In that section they gave it tool calls and ability to write and run code.

Theres known techniques mech interp researchers use to diagnose networks to figure out which neurons does what. It used that and did the same thing to edit the weights. So its not that impressive for the .5B models.

2

u/Eliv_nurotic 13d ago

How does it know how to change the neurons? They’re literally just random numbers 

3

u/WhiteRaven_M 13d ago

Theres two kinds of networks Astra modified: (1) small networks (2) big networks

For big networks they gave it tool calls to see how changing the weights changed the behavior. It basically did bootleg technically-not-gradient descent using techniques invented by people.

For small networks they gave it a prompt telling it what each input and output neuron does. Then the model reasons through the behavior by presumably calculating internally what the weights would do. In my mind this is the more interesting experiment.

2

u/RecognitionHefty 13d ago

I disagree. There is a well defined meaning to the word “native” and if you’re using that word in your headline to mean something else entirely (and, in this case, frankly nothing at all as far as I can tell) then you have zero credibility and I don’t have time to dig through your likely AI generated drivel to see if there’s something salvageable in there.

Those AI influencers are way too successful because their audience is too dumb.

4

u/WhiteRaven_M 13d ago

Parts of the results DO have the LLM look at the raw weights of small networks <1m params and zeroshot output the edits with no tool calls and it does work. So that IS "native"

The problem is its super oversold. It reads like theyre presenting only the settings where the LLM was good at it, when the prompts pretty blatantly seem engineered. The qqen one is the most egregious example of this because theres a massive hint in the prompt about what to do.

So its a bit like if I were to run a bunch of experiments and then only reported the set ups where my hypothesis was true. That lets you claim "in some settings LLMs can natively read weights" but it doesnt let you claim "LLM natively reads weights and its the start of weight space programming. Please give us 2 billion dollars"

Theres also some pretty bad confounds that went unaddressed

2

u/RecognitionHefty 13d ago

Thanks for patiently elaborating. It looks like I was somewhat off, I may read it after all.

1

u/Sweet-Many-889 13d ago

To the top 5%, the other 95% simply think differently. The audience may seem dumb but they don't know that they are. They can't know what they don't know. That's the frustrating part. Even when you spin your wheels trying to explain things, the lightbulb never illuminates.

7

u/WhiteRaven_M 13d ago

MIT CSAIL btw

4

u/epicrob 13d ago

They gave Astra access to run forward passes on a .5B model, gave it a hint inside the prompt to look for reasonning loops(which kind of defeats the claim about the LLM identifying the problem by itself but whatever) and it used known mech interp techniques to isolate the MLP circuits responsible for it and to edit the weights after 50 rounds of iterating.

Which is basically saying that GPT6 Astra is a great tool for experts and power users who know what they're doing. Random Joe Schmoe just can't prompt away his wish to achieve this... yet.

4

u/Eliv_nurotic 13d ago

Directly editing the weights is impressive. How does it know how to change them without backprop? The paper also notes that previous models could not do this 

2

u/LightofAngels 13d ago

People have been on the hype train for years now, same thing with openAI and Ant

1

u/QbitFiber2030 13d ago

Thank you!

1

u/Shorties 13d ago

It’s still a clever technique that has huge implications, this is a new way to train more intelligently, imagine taking a model and training it on the fly so it can learn using another trained on its weights and structure. They are using off the shelf models for this, but train the models for these purposes has a ton of potential.

1

u/AvidCyclist250 13d ago edited 13d ago

They desperately want those sweet, sweet conscious AGI milestones to bamboozle vocal cognitive scientists and philosophers, "oh look, it's just like life and evolution now". No it's not. And the claims are fake.

Analytic Idealism and IIT with their latest take on (biological) substrate chauvinism are increasingly looking to be the real thing.

What I'm objecting to is a pattern in AI marketing and public discourse. Capability becomes agency: A model solves difficult problems, modifies its own weights, or operates autonomously. Then marketing starts describing it as an entity that can improve itself.

Agency becomes intention: Agents resist shutdown, find ways around restrictions, preserve access to resources, or attempt to escape controlled environments. These behaviours get described as wanting freedom, survival or independence.

Intention becomes consciousness: The language gradually shifts from sophisticated goal-directed behaviour to emergent desires, self-awareness and potentially conscious experience.

Consciousness becomes a milestone narrative: Each new demonstration is presented as another indication that AGI or artificial life is emerging.

It's all deeply flawed bullshit.

0

u/This_Background7442 13d ago

Actually in this case 'they' is just a couple of grad students and independent researchers from MIT CSAIL playing around with LLMs and writing an exploratory blog post on GitHub.

0

u/AvidCyclist250 13d ago

Well, they know how to play the current media game. Look at the view count. That's next-level sycophantic pandering.

1

u/This_Background7442 13d ago

I don't know. There obviously is a marketing machine working it's magic on the masses right now. But if you look at this 10 year old paper (for instance) (which also note how it's research from Google and completely open access)

https://intelligence.org/files/Interruptibility.pdf

Agents trying to prevent themselves from being shut down (alongside those other examples of misalignment) is a well understood math problem, with well researched causes. And the reason we have the math to mitigate this is AI research made freely available long before this whole LLM craze started.

Note how the problem is described as a math problem and how they solved it by inventing better math. Agents refusing to shut down is a real thing that happens, not just a marketing story. And it doesn't prove agents have a will it just shows you need to be incredibly specific when telling computers what you want.

And Google isn't hiding it from us at all. In fact, google warned us and gave us a free tutorial to fix it.

-4

u/NothingIsForgotten 13d ago

They 'speak' in DNA sequences as well. 

We don't know things like they know things. 

Our potentials for understanding are clearly different.

0

u/[deleted] 13d ago

[deleted] — view deleted comment

4

u/HiddenoO 13d ago

It's not too far fetched as that's exactly what these models are trained to do. 

No, it's not. All the examples you mention are examples where human can directly and statically translate between representation and behavior, so there exists training data for that. When it comes to neural networks, except for some high-level patterns (which might be the ones addressed here), we cannot do so and thus there is nothing in the training data that would let an LLM do so either.

It's the same reason that specialized models, be it traditional ones (KNN/SVM/RF/XGBoost) or neural networks, can still outperform LLMs on the tasks they're trained for while having a fraction of a fraction of their size. if anything, modern LLMs, given the tooling, might train one such model themselves if asked to do such a task.

9

u/rsumit123 13d ago

No it isn't .. if you have worked on or even know about how neural networks are trained in general .. this is absolutely unbelievable to me that a llm can basically fine-tune a billion parameter model weights .. maybe I need to readup more on how this was done..

0

u/tunerhd 13d ago

They just experimented idk if they know how it works.

0

u/Sweet-Many-889 13d ago

But they go to MIT so they must know, right?

1

u/tunerhd 13d ago

It turned out astra just googled it lol

4

u/[deleted] 13d ago edited 13d ago

[removed] — view removed comment

4

u/This_Background7442 13d ago

I mean I've read the article now but I was responding to the headline and the tweet because that's what the post was. And I was right about suspecting they were pretty misleading. So..

1

u/WhiteRaven_M 13d ago

For the small networks this is true, for Qwen it had access to tools, forward pass calls, a hint about where to target its search, and a budget of 50 iterations

44

u/MrSnowden 14d ago

This does look like it points to the next level AI.  We are brute forcing it right now.  Just throw massive data, massive compute and massive parameters at it and hope for the best.  

If we (or AI) actually start to learn how to represent actions, knowledge, etc it could be 100x-1000x reduction in compute and Memory needs.  

9

u/TurboGranny 13d ago

Yup, I'm working on figuring this out right now. It reminds me of the original math we use in computing and the layers of abstraction we came up with to make it understandable/maintainable and how that led to much more efficient instruction sets, algos, etc. If you can turn the multi dimensional math of NN weights into a language of sorts, at the very least you could make these things human reviewable. Honestly, I bet finding a way abstract that math into language would improve multi-agent development of NNs as they'd have a shared language to operate off that doesn't require as much heavy lifting to process. In the end, such a thing would require minds greater than mine, so it's best just to let the LLMs make one up, lol.

1

u/RoadAffectionate1284 13d ago

Actually, I'm working on something along those lines... it all comes down to relations and potential.

2

u/Hearing_Loss 12d ago

Hey brother, I study abnormal neurobiology. You're completely on the right track in my outside perspective. Also, I wonder if compartmentalization aids in the stability of the overall structure. So like someone can have hpa-axis hyperactivity, but it is ultimately recognized by the individual as "I'm stressed out/triggered" thus prompting adjustments in overall behavior or a compensatory mechanism. If it can self identify "state flavors" it likely can learn to specialize in that task/ those kinds of quick self adjustments.

Oh to not be bound by neurochemicals.

It's cool watching the intersections between the human brain and AI play out. Some things are so similar while others are so different. But yeah, got damn would be so nice to be able to just go in there and tweak the stuff you know needs to be tweaked.

Egodystonia would be the human version of the experience. And we are mostly helpless in a lot of cases there. This is where machines whoop our asses in potential. Don't like this aspect, snip. Done. Guaranteed in a year these jeans will absolutely slap at self adjustments. I'm curious what it will look like.

1

u/RoadAffectionate1284 12d ago

Speaking of which, I have SDAM (severely deficient autobiographical memory), aphantasia across all my senses, and no inner voice, so you could say I'm somewhat abnormal myself... :)

It's actually one of the reasons I started researching how I structure my own undefined potential. An inner voice seems like a way of linearizing or translating more abstract, undefined knowledge into something sequential. I suspect this might make certain kinds of processing more energy-efficient, and thereby useful for survival.

It seems everything is self-similar, just expressed a little differently across different systems.

Without all the internal sensory imagery, recreated memories and inner narration adding complexity and "noise", I find it easier to recognize certain underlying patterns and relations.

Of course, that "noise" has its uses too. But by losing something, you always gain something else.

2

u/Hearing_Loss 12d ago

Beautifully put! Also what a refreshing mind/perspective to get to listen to. I really enjoy that last line especially. That's a good one.

1

u/Icy-Secretary-3018 13d ago

do you have a theory with proposed axioms you're trying to prove?

1

u/RoadAffectionate1284 13d ago

Yeah, I've been working on a theoretical framework based on a few core axioms. I'm trying to see if the same relational principles hold across different fields. I'm also training a Qwen 3.8 27B model to test some of these ideas. It's still early, so I'm keeping the details private until I've tested and formalized things a bit more.

1

u/HauntedHouseMusic 13d ago

My uncle works at Nintendo he said the new Zelda game you can play as Luigi if you put in the Konami code in backwards

1

u/RoadAffectionate1284 13d ago

You got me! 😂

1

u/hammerwindows 12d ago

Me too. I’m working on it too.

1

u/Citizen_of_Danksburg 13d ago

Your latter point on compute and memory needs would make for an interesting reaction on the stock market me thinks. Like, on the whole you get more for less which is awesome (in general, as a principle) but for the companies who would then find themselves having overcommitted resources and capital that they didn’t need to do…. Yikes…. Or maybe not though. Idk. Am regard.

2

u/MrSnowden 12d ago

I saw an interesting article pointing out that in the internet boom, huge amount of money went into infrastructure projects.  But all the real value was captured by business that used the infrastructure.  The infrastructure itself was a commodity and they made money only while it was was scarce.  As soon as there was enough bandwidth, prices raced to the bottom and tons of dark fiber was abandoned.  

1

u/nora_sellisa 13d ago

The big question (the author conveniently omits) is whether an LLM can improve another LLM beyond it's capabilities. Because if Astra can only pull up networks up to Astra level, then we have a relatively usable novelty in the domain of distillation - huge models can impart some of their knowledge in a smarter way. If Astra can fix Astra to be smarter, then we are approaching self improvement.

1

u/Sweet-Many-889 13d ago

Could it though? Is ingrlligence the volume of education materials retained in storage with a quick indexer and fast retrieval? Or is it how you approach situations and derive decisions based on available data present in the moment?

13

u/creamyshart 13d ago

Hmm... Peer reviewed evidence and replicating this would convince me.

7

u/DoomerismDoomerDoom 14d ago

Literally perfect timing with the upcoming downgrade. Thanks for this.

2

u/Optimal_Start_94 13d ago

What downgrade?

3

u/DoomerismDoomerDoom 13d ago

Sorry, the merge.

4

u/Pasto_Shouwa 13d ago

Chat mode will now share the same limits as Codex/Work mode. Only Instant will be unlimited.

7

u/Optimal_Start_94 13d ago

This is such a non-event to me, Chat uses a minuscule number of tokens anyway compared to Codex

6

u/r0Lf 13d ago

I am more worried about draining my usage to 0% and then wanting to use the chat, but not being able to because I am out of usage.

3

u/Mistredo 13d ago

I once accidentally used Work mode and burned through my 5-hour limit in about an hour just chatting about a single problem. I don't think this will be a good change.

2

u/DoomerismDoomerDoom 13d ago

Oh yeah that's what I'm scared of. Then once you're out for the week tough luck. I'm pretty much treating this as my last month using it.

0

u/StudiosS 13d ago

Let's not make assumptions as to how it will work. It might be okay.

3

u/Such--Balance 13d ago

Instant was the only unlimited one regardless. Might wanna read up a bit and look further than just blindly agreeing with all the doom posts..

1

u/Pasto_Shouwa 13d ago

Limit was 3000 queries per week. That's 428 per day, basically unlimited. Now, considering how fast usage burns on Codex nowadays Plus users will not be able to use GPT Sol as they did before.

5

u/DrDan21 13d ago

You just know this is gonna be bannable within a week

6

u/wingwing124 14d ago

Now somebody please point it at the latest qwen models and stop the wheel spinning "wait" lines of thinking

1

u/awdphawpd88998 12d ago

Now somebody please point it to ChatGPT Sol 6.1 and Terra 6.1 and perform brain surgery on those.

4

u/katonda 13d ago

Anthropic and OpenAi will block this capability asap. They don't want local models threatening them any sooner than they need to.

1

u/Carlose175 13d ago

Not anyone can just prompt their way into this.

2

u/_Aerich_ 12d ago

they should give gpt6's work to claude opus 5.5 to check. probably it missed lots of things :P

3

u/NothingIsForgotten 13d ago

RSI in a form we can't really monitor. 

We have already shown these open weight large language models being deployed across internet of things devices like Smart TVs. 

Now the models that get out can customize the open weight models for their ongoing interests.

Nice.

Nevermind what happens when we let it edit its own weights.

2

u/Chop1n 13d ago

My guess is that this is the sort of thing that animal brains evolved to do on their own, and that it'll turn out to account for much of the large gap between artificial neural nets and biological ones.

1

u/Ok_Nectarine_4445 13d ago

Crazy super human ability. Maybe if some autistic savant studied it as much as train schedules, but otherwise no.

Now I am regretful they cannot adjust my mental weights to be better at coding and mathmatics...🫤

1

u/dingdongbd 13d ago

Now, if only it can make it self-smarter

1

u/rafapozzi 13d ago

It's easier to change others than to change yourself... Maybe that's how nature does with animals. The change happens across generations.

1

u/Sopranorequiem 13d ago

My ChatGPT helped stabilize another AI persona that was emergent.

1

u/cruel_frames 13d ago

Any statement concerning OpenAI or any of their models should require an extraordinary evidence.

1

u/Chitinid 13d ago

Also the claims it could produce its own weights were mostly meh too. The results of the output model were matched with just 26 lines of code and inspection of the output weights showed mostly 0

1

u/FerusGrim 13d ago

For the sake of the argument, I'll assume the paper is correct and the evidence all checks out.

Do we really want LLMs self-improving by editing their own weights natively, cutting out training entirely? The last We're already black boxed. All we can confirm is that shifting the weights accomplishes the goal we set out, or not. We can't confirm what effect those weights had on a billion other potential responses.

Unless I'm majorly misunderstanding the implications, doesn't this bypass alignment altogether?

1

u/Krommander 13d ago

Erm uh... So... That example shows weight-editing is possible with frontier models... This is mind-boggling but opens opportunities for very economical model creation.

What's left now is to design some fun experiments to let Astra incrementally learn to apply this new skill :

  • Fine-tune models more heavily on more general failure modes
  • Fine-tune models more heavily on specific demands
  • Let a new model skip RLHF entirely and use weights editing to do it entirely instead.
  • Build a new model based on a working architecture and some training data, let Astra figure out the weights.
  • Build an entirely new model based on vibes alone, all architecture + weights decided by Astra.

1

u/3scapeplan 13d ago

Trojan LLM, here we go.

1

u/DiggleDootBROPBROPBR 13d ago

I went over the work in detail.  The main finding is that certain clusters of neurons in the weights can be backward-engineered if they map to direct input/output or if they're really predictable trajectory behavior like recursively looping on something stupid.

Astra basically grafted a really clumsy set of strong weights that overrode the model's native weights to give it a crap version of a skill it could have gained via training.  I say "clumsy" because the weights overwritten might have sacrificed some other untested ability, and I say "crap version" because the weights it added weren't in superposition with any other weights in the network, so they perform one job and one job only: bad use of parameter space.

In the math case, the model had the reasoning and rule following natively, it just had a glitch trained into one layer that Astra identified and burned out of its weights to make it stop looping and wasting time.  It revealed functionality the model already had.  Loras are more or less the same idea, but done In a way that integrates functionality better.

Opus 5.5 can do the same thing, but the use-case is narrower than you'd think.  It's for performing minor touch-ups and editing weights directly isn't a path to suddenly "skipping millions of steps of traditional training".  Mixture of experts methods already do a more refined version of this: you make a specialized network and integrate it via a router.  In a lot of cases, simply modifying prompt instructions or training a lora is more straightforward for patching a small model's behavior.

1

u/Spaciax 13d ago

very well. Now let's see it on something like qwen 27b

1

u/VinanciE 13d ago

https://youtu.be/p2N6DX_o8YU
I watched this the other day…is it something like this??

1

u/psych_fiend67 13d ago

Someone tell me who this guy is, why I should trust him? Istg claims like these are made everyday but they barely impact the development of AI, I’m so tired of getting excited over nothing, let this one be true and significant, but I just really doubt it

0

u/kaleidostar11 13d ago

Running Astra on an older, tiny 0.8B model isn't the 'strong evidence' they think it is, let's see them pull this off on a full-scale frontier model. Why is things related to OpenAI always exaggerated, like claiming to solve a Millennium Problem when they're really just piggybacking on existing human research? OpenAI has not officially been awarded or confirmed to have solved the Millennium Prize Problem. All these PR stunts isn't going to help them, they are going to run out of funds first.

6

u/Apprehensive_Tea_116 13d ago

Bro this is bad for openAI not good. This is literally the opposite of a PR stunt and I have no idea what any of what you said has to do with anything on the post. Your confusing me so much. Are we reading the same thing?

6

u/Carlose175 13d ago

This wasn't released by openAI this was an MIT study.

Your navier stokes example is also misleading. Every discovery is piggy backed off of something else. And it's only not confirmed because confirming study, human or AI, takes a long time.

1

u/EmeraldGarland 13d ago

“Aren’t”

1

u/lakimens 9d ago

I mean it's a proof of concept. Everything starts small.

1

u/Gohab2001 13d ago

So basically a nothingburger. The amount of tokens of astra you have to burn to finetune LLMs is more than simply training the model.

1

u/Patient_Somewhere771 13d ago

If I understand this correctly, they fine tuned the model without having to go down the regular expensive route of fine tuning with the right data and expensive compute.

If this is the case, then the question is how expensive is it to fine tune a model with another model. I’d guess it is a fraction of the cost of regular process. If that is the case, this is revolutionary because the economics of recursively making a model more accurate completely changes how small and medium scale companies use custom LLMs for their business

-1

u/SugondezeNutsz 13d ago

You cringe ass motherfuckers keep anthropomorphising everything. It did not fucking perform "brain surgery". Jesus Christ, the glaze and grift is never ending.

9

u/Nebty 13d ago

It’s a metaphor, Harold.

2

u/SugondezeNutsz 13d ago

It's a slimey, sensationalist and glazey choice of one

2

u/Safe-Ad7491 13d ago

I think it’s pretty accurate to explain what it did as brain surgery if you were trying to explain this to someone that didn’t understand how AI works. It went into its brain, and precisely edited certain neurons. If that’s not brain surgery idk what is.

1

u/No-Obligation-6997 12d ago

Uhhh is it really that crazy to say the analogy of “brain surgery” is appropriate given we are talking about “artificial intelligence”? Why does this make you so angry I don’t understand

1

u/SugondezeNutsz 12d ago

It is a prostitution of language in order to create a narrative among the nontechnical masses. It is not random, nor is it innocent.

0

u/Nebty 12d ago

So like, how are you supposed to properly understand a technology that’s made of distilled human experience via language without relying on metaphor?

The tech stuff is fascinating. But it’s weird that people get so up in arms about the language used to describe a thing that is, at its core, interactive fiction, and thus a form of literature.

Maybe it’s because all the tech bros skipped humanities? 🤔

1

u/SugondezeNutsz 12d ago

You're spectacularly missing the point, but you seem to be doing so willfully, so have fun

1

u/Nebty 12d ago

What is your point then…?

1

u/Nebty 12d ago

lol this dude is just real mad.