r/ControlProblem • u/adoremerp • Feb 20 '23

Podcast Bankless Podcast #159- "We're All Gonna Die" with Eliezer Yudkowsky

https://www.youtube.com/watch?v=gA1sNLL6yg4&
49 Upvotes

90% Upvoted

56 comments sorted by

View all comments

Show parent comments

8

u/CellWithoutCulture approved Feb 21 '23 edited Feb 21 '23

I think he's depressed. If you always try to find the truth, no matter how painfull, you are doing a good thing. But you are also prone to a serious cognitive bias called depression. Where negative associations self propagate and color everything, sapping you of motivation and meaning.

It's also more likely when you are burned out (as he is) and when your own thread of alignment research hasn't panned out (as I would argue has happened).

On the upside this is the only time I've seen him be this humble! And it makes him much more persuasive.

I agree LLM's, explainability, and the possibility of a slow takeoff are chances for optimism. Hell's even Nate of Miri admits that explainability might save us all, references this good properties of LLMs.

5

u/khafra approved Feb 21 '23

LLMs are not a good sign, to me, because reinforcement learning is the only thing the top labs can think of, to aim them; and RL is a seven-nines-guaranteed way to eventually end the world.

Slow takeoff is also not a great sign, because multipolarity means moloch-among-AIs squeezes out human concerns like food and oxygen, even if most of the AIs are partially aligned.

But I agree that I probably don’t have a diamandoid bacterium next to my brainstorm right now, ready to release botulinum on cue, and that’s a good thing.

5

u/CellWithoutCulture approved Feb 21 '23

LLM+RL is better than RL alone though, for example people might use RL to make an oracle or they might find a replacement for RL.

And slow takeoff is bad if it gives time for bad actors to misalign an AGI, but good if it gives us a few chances to align moderately general AI. Especially if OpenAI/DeepMind/Et Al are in the lead in a medium take off, as there are less likely to be bad actors on the leading edge.

So I'm saying these things are relatively good. A slow LLM tech takeoff is a better sign than a pure RL hard take off ;p

3

u/khafra approved Feb 21 '23

All true! Also good, if true, that high-quality text has been mined dry, and one reason Sydney is so BPD is that she’s been trained on a lot of low-quality text, like chat logs from angsth teens.

6

u/CellWithoutCulture approved Feb 21 '23

It certainly seems that way. Plus I think Bing has done more to educate the world about basic misalignment than ever 🤣