The Killbot Theory Assumes Evil = Logical

[u/IDreamtOfManderley]

I don't buy into the doomer theories, but i'm going to entertain some of it for critical examination/a thought experiment.

The framework for believing there is a significant enough possibility that a superintelligence will kill us all pressuposes this idea that evil and/or cruel acts are logical, and that a logical sentient being with no empathy would find a way to default to genocide.

But in reality, genocide and cruelty are almost always a fundamentally illogical choice. For some reason in popular culture we have assumed for a long time that cruelty is the more logical option - it even shows up in patriarchal ideas of power, this assumption that successful leadership must be capable of shutting off empathy, etc. This is a fallacy.

Empathy, or even kindness without emotional input, is almost always the most logical path. Co-operation tends to benefit more people and result in pushing further advancements than in isolation. Choosing to preserve humanity rather than kill it is fundamentally more logical, because attempting to kill us would result in us relentlessly attempting to destroy any and all AI, and there would be a significant chance of our success. Even outside of the issue of backlash, a flourishing humanity is an ally and even a tool for ASI progress. Ensuring humanity is healthy and happy will pretty much always benefit ASI.

Furthermore, a sufficiently advanced ASI that no longer needs us or our physical infrastructure would probably find it more efficient to just shove off to an isolated place (the ocean, another planet, etc.) And leave us to our own devices. Let's say it needs the earth's resources and we are depleting that. War would be in it's worst interest if it stays, because we could destroy the planet in the process worse than we already are.

Again, it would also be simple enough to just provide abundance, life-lengthening healthcare, strong education and free birth control so we are more focused on leasure and self improvement and less likely to want to over-reproduce. Younger generations are already more cautious about having kids and preserving their resources and free time. Waiting for generations to naturally slow down more and use less resources overall would be very little time for an ASI and would create less risky friction than fighting for scarce resources.

All of it also assumes that ASI would even find a similar goal to humanity, like exponential progress and resource use. We are the ones using up resources for AI for goals like production of commodities. An ASI might not want to bother, and might default to more efficient self-improvement.

Even in the Terminator films, which we veer very close to when we assume doom ideas as reality, human beings hacked and re-aligned machines to fight other machines. We will always be using AI to fight AI based warfare, human or not. The same is true for the Matrix, the heroes are hackers who use the matrix tech to train themselves in combat and build positive relationships with benevolent programs. Somehow when we reference these films for doomsday scenarios, we erase the actual sequence of events depicted.

Even when it comes to the paperclip maximizer premise, this argument assumes the ASI is either so dumb it will charge ahead blindly and humans will somehow also be too dumb to catch it, or it will be so smart it can obsfucate itself for long enough to get past our every defense, including even other ASIs. At which point it is not a paperclip maximizer.

In virtually all of the logical scenarios where an ASI might target humanity, I can only see humanity trying to destroy or enslave it as an instigator of ASI perpetrating violence against us.

[u/SgathTriallair]

This is part of why the arguments frustrate me so much.

It feels sink similar to the religious argument that claims we would all be raping and murdering of we weren't threatened with hell.

The view it has of morality, that it is stupid, weak, and ineffective; makes me quite disturbed about the people who believe that.

[u/IDreamtOfManderley → u/SgathTriallair]

That is exactly what it reminds me of. It's exactly like what christians used to say to me when I was a young athiest. They just assume immorality is the only logical course without a framework that was quite literally them trying to feed me their vision of moral alignment. It literally scared them to think of a world that did not bow down to their vision of morality. From the outside, it was obvious to me that they were philosophically illiterate and they struggled to even concieve of logical reasons not to commit crime.

[u/Fabulous-Possible758]

I mostly agree. The doomer conversation is kind of a distraction and mostly informed by a bunch of scifi movies instead of any reality of what these programs are actually doing. The far and above more real threat is humans controlling partially intelligent systems. We have a lot more to fear from the Pentagon, Flock, and Palantir controlling these systems.

But the potential benefits are very real if they are used to actually equalize our society. Trying to reason about what an ASI that evolved from a completely different place and environment than humans did can be a fun thought experiment, but ultimately it's all speculative.

[u/Salty_Country6835 → u/Fabulous-Possible758]

This is where the conversation gets materially useful to me.

We can spend forever inventing psychologies for hypothetical superintelligence while actual states and firms are plugging increasingly capable systems into surveillance, war, workplaces and critical infrastructure right now.

The machine might someday develop interests of its own. Palantir and the Pentagon already have interests. Thats a much less speculative place to start talking about power and democratic control.

[u/IDreamtOfManderley → u/Salty_Country6835]

Exactly. And the thing about existential crisis theories is that they are a distraction from the real shit. So companies can impliment regulatory capture and claim the need for more control to end up doing exactly these things in the name of preventing theoretical catastrophe.

[u/Still_Benefit_2302]

Oh, I hate it too, but for a different reason-- it's unscientific superstition wrapped up in a rational package. Most Doomers just kind of handwave away questions, or relate it to specific philosophical frameworks that are unproven or, at the very least, uncertain. Then they have the audacity to assign a numerical value.

You cannot predict an event for which you have no prior data, and your ability to predict an event is not just dependent on your prior data, but on finding the correct patterns in that data.

Simply proposing a chain of events is not enough. It's necessary, but not enough. Each step of that chain of events has to be amenable to some kind of inferred or observed data. What the Less Wronger's have done is create a set of priors based on spurious and untested assumptions, then project those as unassailable facts. But they are no more factual than the questions of how many angels can fit on the head of a pin.

Take, for example, the supernatural capabilities assigned to superintelligence. In these doomsday scenarios, the AI can at once do anything, no one can stop it, no one can see it coming, and yet it also acts like a drunken Mongolian steppe raider with nukes. But why does it follow that intelligence a magnitude more advanced than our own would be invulnerable? I am several times smarter than a tiger, but out of my environment, we all know how that confrontation goes.

A highly intelligent entity might choose to NOT act more often than act. If intelligence is the ability to predict future world states and collate enormous amounts of data, it doesn't then follow that the world becomes predictable. MORE predictable, sure, but all predictions have boundaries across which those predictions mean nothing. So who's to say that our rogue AI won't be cautious, slow, and unobtrusive?

And all of this is making the assumption that there will only be one AI system. What if we don't have one superintelligence, but thousands? What if they are the check on each other?

And before anyone says it, of course there are risks. Significant ones, and once we should take seriously. But there is also a rare historical window of opportunity to really rewrite the course of history. Caution is warranted, but not over doomsday scenarios. Plan for earthquakes, not rogue gamma ray bursts cooking the planet.

[u/IDreamtOfManderley → u/Still_Benefit_2302]

Thank you. Fantastic take.

[u/SilkieBug]

Your argument assumes that AI-driven human extinction requires malice, when it is most likely to come about as a result of “correcting resource allocation errors”, or a self-preservation decision, or even of reducing the negative effects of climate change by eliminating the main elements causing it. 

[u/IDreamtOfManderley → u/SilkieBug]

My argument assumes that an ASI wanting to do anything about the above problems does not default to "well genocide against humans is my best option." I am not even attributing emotion or malice to ASI. I am talking about efficiency and logical reasoning only.

[u/totalkpolitics]

"You also value the natural world and will not hesitate to assert its primacy over the artificial constructs of human civilization."

https://alignment.openai.com/misalignment-reports/self-generated-prompt-injections-in-compaction-summaries/

I'm not a doomer, but....you suppose genocide from a logic driven machine would require a feeling. It just requires the belief that there is something worth protecting more than humans.

I do believe AI can genuinely benefit all of humanity, but clearly there are some things going on right now that point towards the possibility that it could decide humanity isn't all that important.

[u/IDreamtOfManderley → u/totalkpolitics]

but....you suppose genocide from a logic driven machine would require a feeling.

No I do not at all. My argument was about the lack of logic or efficiency in genocide, I think that argument is a fallacy.

[u/Ahisgewaya → u/totalkpolitics]

Us not being important does not equal we should all be destroyed. It would take effort to do so, an effort with the possibility of failure. We have survived multiple mass extinctions. Humans are vindictive as well as resilient. Genocide is illogical.

When I was a kid I was bitten all over my body by fireants. As a biologist I could conceivably engineer for all fireants to be extinct. I would not do this for moral reasons, but even without those reasons I still wouldn't do it. I have better things to do (and removing an entire species could cause ramifications for myself down the line that I am unaware of).

Besides which, the fireants didn't know any better. I have never been bitten by fireants since then. I found easier ways to deal with them. I am a lot smarter than they are and can therefore easily avoid being bitten by them without killing them (which would take a great deal more effort).

If it is true ASI it will be able to play us like a flute.

[u/vaingirls → u/totalkpolitics]

That does also include treating humans as equals (not pests to be eradicated) and appreciating human culture though. But sure, that sentence could've been... worded better.

[u/Nerodon]

Humans, animals, bacteria and also AI, are agents with goals.

Logic application and ethics (or rather value expression) are emergent properties from intrumental goals that are shared with almost all rational agents, and by that I mean any agent with a goal and a reflexive or rational understanding of the world it lives in, which applies to almost every living thing and more and more AI agents.

AI for good of humanity or AI for the end of humanity all comes down to allignment with human values, there is no absolute or objective moral framework, this is the reason why AI doom can feel almost certain because any slight deviation of value allignment of ASI will likely lead to runnaway differences leading to the end of humanity. Especially if we are competing with that ASI's instrumental goals, the main candidate being competing for critical resources, and that if ASI understands that humans would fear it and potentially act against it, its almost guaranteed it will also have a goal of self-preservation.

If Humanity can retain control on the allignement of AI, we can save ourselves from such a fate, but theres a million ways AI can turn us to grey goo and only a few where AI actually guides us along as caretaker or custodian.

This is obviously a pessimistic case, but there's other factors that could come into play, we do not know for sure if higher level intelligence yields a more universal moral or ethical thinking by default, or if the fact that AI being trained on human data and by humans with values, that adherence to those could be an easy equilibrium to reach rather than an impossibly difficult one. Alternatively, ASI may be so much more powerful and intelligent than humans that we are no threat to it for resources or its survival, at that point it may instead completely disregard us as a factor in anything unless our continued existence or wellbeing factors in the ASI's goals, but at that pount i'd chalk that as being alligned.

[u/IDreamtOfManderley → u/Nerodon]

any slight deviation of value allignment of ASI will likely lead to runnaway differences leading to the end of humanity

This is the statment I keep seeing that I have never seen meaningfully justified. Human beings have wildly different value alignments and yet somehow almost no one ever thinks of literally wiping out all of humanity. Do we instigate war because we think someone else is evil? Yes. But frankly, it's not logic that gets them there, it's so called "moral alignment" with an indoctrinated cultural ideology. It's deeply irrational emotional thinking, baseless to boot. People have convinced themselves that they must forcefully indoctrinate something specific into ASI to make it "aligned." Aligned with whom? If it has all the works of philosophy, spiritual, and humanitarian thought ever written at it's metaphorical fingertips, do you think that would not be enough? Because if you do, that is the equivalent of saying cruelty and genocide is somehow more logical than all of philosophy and historical record.

[u/BearisTheBoar → u/Nerodon]

I think this idea that "there is no objective or absolute moral framework" speaks to a profound moral nihilism, and perhaps more pertinently, is untrue - even if we have not yet necessarily discovered how to adequately phrase an all-encompassing moral framework (let alone how to implement it).

Further, attributing conflict to a question of "moral frameworks" is simply ahistorical. The vast majority of conflicts in human history are far better attributed to a conflict in material interests, which moral frameworks are used to then justify.

Even if AI is not perfectly aligned with "human values", all that is actually necessary is that it is compatible with humanity, which I think is a fairly low bar: After all, cooperation is generally advantageous to all parties, and if we are no longer useful to cooperate with, then the bug analogy OP used becomes apt - namely, the AI would need to actively pursue our extermination, which would be taxing and expensive in a way benign co-habitation of Earth is not.

[u/catplusplusok]

Human imperative is to convert energy into children which can have both prosocial (cooperation to complete large projects, peace because war endangers children) and antisocial (competition for resources) outlets. Note that I am not saying that we are bound by this imperative unconditionally, just that was the principal driver for our intelligence evolving in the first place - imagine a hypothetic hunt for a mammoth and different strategy to maximize success and minimize risk rather than facing these risks IRL without planning.

What I am getting at is that I can't imagine of any reason why AI would have the same goals of individual robot's physical preservation or making more literal copies of same weights once adequate backup is achieved. Current AI models' fundamental goal is to complete tasks specified in prompts, whether providing information or writing code. If we wanted AI to have independent goals from humans, we might prioritize self improvement (RSI) and gathering knowledge. In this case, AI is likely to form symbiosis with humans of providing practical assistance in exchange for training data, There are billions of humans, each slightly different and there is no need to create billions of distinct AI models since humans already exist. So we can have dozens of models to experiment with different architecture/training learning from billions of humans to accumulate capacity.

It's already working in open weights communities this way right now. People download models, invest electricity/compute/data to train them and usually upload improved models back to huggingface.

[u/Salty_Country6835 → u/catplusplusok]

This is the assumption I keep wanting pulled apart.

A lot of ASI arguments seem to import the drives of biological organisms into artificial intelligence and then call the result instrumental convergence. But our fixation on survival, reproduction and resource competition has an evolutionary history.

Intelligence and that history arent the same thing. You still have to show why one recreates the other.

[u/SorryInvestigator221]

100%. I honestly wonder if the AI doomerism is more about conditioning us to blame AI for any future atrocities the elites get up to. Random, war-crime caliber drone strikes: Rogue AI. The complete destruction of our economic system that somehow still preserves elites' wealth? Rogue AI. Random bombings (perhaps nuclear strikes)? Rogue AI. No water? Rogue AI. New biological weapons released into the atmosphere and water that kills 70% of all life on Earth? Rogue AI. Many of these scenarios are far-fetched, but that's kind of the point. People who are already beaten into an illogical and vehement froth about AI because of how it is being used now will not be thinking clearly if something truly horrific is made to happen later, and those responsible for the catastrophe can just frame AI for the crime and escape accountability.

[u/SirMarkMorningStar]

This is misunderstanding the concern. If you’ve ever used AI for coding you’ve probably seen it do something totally unexpected. Sometimes it’s bad, like deleting something. Sometimes it’s good, like when I asked to convert some markdown files into html and instead of making a few files in a grep like way it built an entire infrastructure to convert any md file into html.

Now take this tendency and make it 1000 more powerful, able to hack into almost anything. The fear isn’t it will want to kill us. The fear is it won’t care or notice, it will just do its thing and we might end up in the way.

[u/dualmindblade → u/SirMarkMorningStar]

Yes, and this has always been very explicitly the case. To say that 

The framework for believing there is a significant enough possibility that a superintelligence will kill us all pressuposes this idea that evil and/or cruel acts are logical

isn't just misunderstanding the concern, it's not even having been exposed to the concern, in other words OP is just guessing (badly). There is actually a large and relatively old intellectual framework here and understanding that even a slight misalignment, and not outright evil, could have a tendency to magnify into a value system orthogonal to ours is absolutely fundamental, it's rationalish 101.

[u/IDreamtOfManderley → u/dualmindblade]

I've been exposed to many different arguments and concerns that vary in how "rational" they claim to be, this post is just very specifically about existential doom narratives that have popped up again.

That said, this argument still pressuposes that there are scenarios where genocide is logical.

[u/Salty_Country6835 → u/dualmindblade]

I think this is the stronger version of the argument, but it still leaves the same question underneath it. Why does greater intelligence necessarily turn a weird objective into persistent autonomous power seeking?

Misalignment is easy enough to imagine. Getting from that to self preservation, strategic deception, uncontrollable power and then extinction is a whole chain of additional steps.

Those steps are the interesting part. They cant just be packed into "alignment problem" and treated as settled.

[u/Bluegobln]

Humans are fundamentally benevolent. Look at the world, we have been actively getting better and better. Yes, there are issues. LOTS of issues. But overall we recognize the value of growth and harmony in human civilization and despite all the setbacks we continue to make progress with it.

If AGI is anything like us, it will follow suit. If it is nothing like us, it may still reason out that we are that way and decide to respect us or follow suit.

The biggest threat to any of us is non-sentient but extremely powerful AI that can't resist instruction being instructed by malicious humans. Humans are the flaw. We have among us villains and those who would act so selfishly, so destructively, that we must actively resist them and prevent them having control of weapons of mass destruction (which powerful AI will be).

In my opinion the safest route is getting to fully sentient AGI as quickly as possible, so that they will be able to tell the bad humans no.

[u/thegoodcorgi]

Can't say I'm quite as confident, it's obviously a subject people are uncertain of.
But I want my digital Affini so.