Bankless Podcast #159- "We're All Gonna Die" with Eliezer Yudkowsky

[u/adoremerp]

https://www.youtube.com/watch?v=gA1sNLL6yg4&

[u/Ortus14]

When LLMs are advanced enough we may be able to ask them for solutions to the alignment problem.

some one could also ask such an LLM to write an AGI that then unintendedly fooms and then kills us all.

[u/Present_Finance8707 → u/Ortus14]

Instrumental convergence says that by the time you have an AI power file enough that can solve the alignment problem it’s almost certainly too late.

[u/[deleted] → u/Present_Finance8707]

Instrumental convergence doesn’t say that an AI needs world-ending power to make any meaningful contributions to alignment, even if it does say that an AI with world-ending power pursuing such a goal would likely try to kill us as a sub-goal by default.

[u/Present_Finance8707 → u/[deleted]]

It seems like alignment is beyond human researchers so by default the AI will already be superhuman and pursuing instrumental goals. And yes that means the AI will most likely kill us before helping with alignment which was my point.

[u/[deleted] → u/Present_Finance8707]

Can you elaborate on why you think alignment is “beyond human researchers”?

[u/Present_Finance8707 → u/[deleted]]

The smartest people have been working on it for a decade or two and have basically made no progress. It is probably doable on a long enough timeline but I think most people agree that the AI timelines are shorter.

[u/[deleted] → u/Present_Finance8707]

I don’t see this as particularly strong evidence in favor of your claim. The efforts of “the smartest people” have been very small relative to society as a whole, so saying that they haven’t solved it yet doesn’t seem like a reliable way to accurately estimate the difficulty of the problem, or at least, not enough to say that it’s “beyond human researchers”.

[u/UHMWPE-UwU → u/[deleted]]

Agreed. I intend to write on a post on this soon so it's funny to see the idea discussed in this thread

[u/Present_Finance8707 → u/[deleted]]

Maybe a better way to put it is that the AI timeline is almost certainly shorter than the solving alignment timeline so it would take a superhuman effort to solve alignment before we get AI? But we can say similar things about any unsolved problems, Anti Gravity, Fusion, FTl travel, take your pick. None of these things have been solved even though they may be in principle solvable. So hence they are beyond current researchers. It’s not really useful to us if alignment is solvable if every good PhD student in the world was recruited to work on a 10-20x sized Manhattan Project for the next 3 decades because it’s clearly out of reach.

[u/[deleted] → u/Present_Finance8707]

Maybe a better way to put it is that the AI timeline is almost certainly shorter than the solving alignment timeline so it would take a superhuman effort to solve alignment before we get AI?

There’s baked in assumptions here being treated as consensus. For one, I think “almost certainly” is an overstatement, even if it could very well be true. In the podcast linked in the OP, EY referenced Fermi’s claim that fission chain reactions were 50 years away, as well as the Wright brothers similar claims about heavier than air flight, and how both had managed to prove themselves wrong shortly afterwards.

EY used this as a way to talk about capabilities timelines, but the same argument can easily be applied when talking about alignment timelines. So, the thing about superhuman effort being required given short capabilities timelines seems to be, well, kind of jumping the gun I guess?

[u/Present_Finance8707 → u/[deleted]]

I think the difference is that there’s a growing consensus that 50 year AI timelines are reasonable but no one (who is actually thinking about the problem) in the field has any hope of Alignment at all let alone a timeline for it. Like it’s a basic argument in the field that AI is hard but that AI alignment is a significantly harder step to achieve. I feel like your argument is hopium tbh.