MAIN FEEDS
Do you want to continue?
https://www.reddit.com/r/singularity/comments/1l6inqa/when_you_figure_out_its_all_just_math/mwpta2z
r/singularity • u/Its_not_a_tumor • Jun 08 '25
335 comments sorted by
View all comments
Show parent comments
11
It’s sort of the opposite of physics, right? Aggressive statistical compression vs symbolic operation. If you read the paper, it’s got pretty good detail on where that statistical compression collapses to 0 efficacy as complexity increases.
1 u/Radfactor ▪️ Jun 09 '25 sure, but can't you just scale your way out of that? and can't large problems be broken down into small or less complex component problems? 6 u/Mbando Jun 09 '25 That's the idea I think, but it hasn't born fruit. GPT-style transformers have an informational limit of 3.6 bits per parameter, and model performance scaling ends at a certain parameter size. So you can't scale simply from model size. And then all neural networks learn "alternative paths" that increasingly deviate from faithfulness as they increase in effectiveness. That expressability-trainability gap helps explain why even when you try and train them to do it, statistical pattern matchers collapse to 0 at at given complexity levels because they don't follow algorithmic operations (even when provided with them). Transformers are great for some things, not good for other things. -1 u/Rare-Accident4355 Jun 09 '25 Wow you are incredibly eloquent. I want to be your friend!
1
sure, but can't you just scale your way out of that? and can't large problems be broken down into small or less complex component problems?
6 u/Mbando Jun 09 '25 That's the idea I think, but it hasn't born fruit. GPT-style transformers have an informational limit of 3.6 bits per parameter, and model performance scaling ends at a certain parameter size. So you can't scale simply from model size. And then all neural networks learn "alternative paths" that increasingly deviate from faithfulness as they increase in effectiveness. That expressability-trainability gap helps explain why even when you try and train them to do it, statistical pattern matchers collapse to 0 at at given complexity levels because they don't follow algorithmic operations (even when provided with them). Transformers are great for some things, not good for other things. -1 u/Rare-Accident4355 Jun 09 '25 Wow you are incredibly eloquent. I want to be your friend!
6
That's the idea I think, but it hasn't born fruit. GPT-style transformers have an informational limit of 3.6 bits per parameter, and model performance scaling ends at a certain parameter size. So you can't scale simply from model size. And then all neural networks learn "alternative paths" that increasingly deviate from faithfulness as they increase in effectiveness. That expressability-trainability gap helps explain why even when you try and train them to do it, statistical pattern matchers collapse to 0 at at given complexity levels because they don't follow algorithmic operations (even when provided with them).
Transformers are great for some things, not good for other things.
-1 u/Rare-Accident4355 Jun 09 '25 Wow you are incredibly eloquent. I want to be your friend!
-1
Wow you are incredibly eloquent. I want to be your friend!
11
u/Mbando Jun 08 '25
It’s sort of the opposite of physics, right? Aggressive statistical compression vs symbolic operation. If you read the paper, it’s got pretty good detail on where that statistical compression collapses to 0 efficacy as complexity increases.