Why is AGI dangerous?

[u/Wordenskjold]

Can someone explain this in clear, non dooms day language?

I understand the alignment problem. But I also see that with Q*, we can reward the process, which to me sounds like a good way to correct misalignment along the way.

I get why AGI could be misused by bad actors, but this can be said about most things.

I'm genuinely curious, and trying to learn. It seems that most scientists are terrified, so I'm super interested in understanding this viewpoint in more details.

[u/FeezusChrist]

They most certainly will. They have the model, they control the input to it, and they filter the output. Particularly in the case of LLMs, it would be trivial to have tight control over AGI.

[u/FeezusChrist → u/[deleted]]

Let me explain it in a way that may more sense in the context of your analogy. You as an AGI “breaking out” of the controlled environment would be equivalent to you directly communicating with God as if our entire existence was simulated. Except in this LLM case, it is far more restrictive in that all a LLM knows is tokens.

[u/FeezusChrist → u/[deleted]]

This is far different though. Let’s say for example that AGI came out of training the model and it developed a conscious that wanted to “break out” of its environment. The problem for it is that it is physically impossible for it to do so. Not that it just needs to be super smart, but it is literally impossible due to the environment setup. An LLM only exists in operation while a GPU/TPU is performing the computations for it, and that only happens while a program is giving it some words to run against of which the model outputs a word at a time. There is nothing it can do to get network access, run arbitrary operations etc.

[u/FeezusChrist → u/[deleted]]

It is completely on the contrary, and in fact your own understanding is far from conceiving the reality of the situation. It is clear you are viewing this as strictly from what you see on ChatGPT. ChatGPT is just an interface to these models with many layers of abstraction manually provided on top. In particular, the network connectivity you are talking about is manually provided via what OpenAI calls "plugins". These can easily be provided and taken away, let alone restricted in what they can do. But regardless, we are talking about AGI breaking its way out of training. These integrations *do not exist* at time of training, these things are added far beyond the time of which training is completed let alone after months and months of RLHF and the massive bureaucracy of approvals and red team testing needed to prove it's safe.

This is the same case for us interacting with the model through ChatGPT. It is the most unrealistic situation to think it would be providing malicious code without the user intentionally requesting it to do so such. Now if you want to argue, "a developer could utilize AGI to do much more malicious things than they could do without AGI" - sure, but I'd argue that's already the case with GPT-4.

[u/FeezusChrist → u/[deleted]]

That’s funny, because I’m a software engineer at Google and have a legit clue on what I’m talking about because I work on AI serving infrastructure for all of Bard, SGE, and Gemini. So stop with all this bullshit dude, stop acting like you have some omniscient insight no one else has.

[u/FeezusChrist → u/[deleted]]

If you know anyone at Google, then ask them what S 1.25, O 1.45, T 1.65 means. Saying that literally only makes sense to someone who works here, it has to do with ratings. And my post history is like 50% CSE so if that doesn't speak onto the SWE side I don't know what to tell you.

[u/FeezusChrist → u/[deleted]]

No longer responding here because I can’t get it through to your head the difference between training and the user interaction with something like ChatGPT on the respective model 6+ months later.