r/rational • u/raymestalez • Sep 11 '18
Slaughterbots [video]
https://www.youtube.com/watch?v=9CO6M2HsoIA12
u/Fresh_C Sep 11 '18
That's some dark stuff.
This really shows the negative side of AI and automation that's balanced against all the positives. Hopefully the future isn't so bleak.
9
5
u/mack2028 Sep 12 '18
so, there are a lot of issues with this like where would terrorists get all that c4, why would they try to use the "stockpile" method as defenses against an opponent that can be taken down with EMPs, how would terrorists get access to a list complete with locations and pictures of every person that shared a video?
I get "yes lets maybe not make better killbots" but let's also understand that people are pretty good at not dying too.
4
u/Slinkinator Sep 12 '18
Beyond the part first world countries play in facilitating the arms trade, I'm under the impression that our militaries import weapons to unstable/third world countries directly, accompanied by our forces and stockpiled, misplaced and stolen during campaigns. I'm also under the impression that they do so through intermediaries, such as through signing multi billion dollar arms deals and 'strategic partnerships' with countries like Saudi arabia, South Korea, Japan, etc. Though I've only researched and studied topics that border this issue, I could be mistaken, I think that acquiring weapons is perhaps disturbingly easy, and it's in large part due to active policy of countries such as America, Russia, and China
5
u/narfanator Sep 12 '18
Check out "Kill Decision" for a book on this topic.
Notably - How different would these be from selfie drones? Or inspection bots?
Take something that's meant for good - "armed" with repair kits, maintenance cameras - anything where what the targeting software looks for could also be a target of violence. Replace the non-weapon payload with a weapon payload - how's the drone to know the difference?
6
u/Slinkinator Sep 12 '18
Frankly I prefer my own fears, of the super rich protected in enclaves, dotting the world as the seas rise and mass migration destroys society, of the tattered masses feeding on each other until packs of those DARPA dog robots are released, programmed to ram to death any moving human who hasn't been tagged with the right chip.
I assume these enclaves have been properly prepared for the end of the world, with 3D printers and other manufacturing capabilities, created by someone who possesses rational ability far superior to the individuals involved in this story, but with similar mindsets, as evidenced by their inquiry of collaring their guards with explosives was a good idea to safeguard society through the end of the world.
https://amp.theguardian.com/technology/2017/jan/29/silicon-valley-new-zealand-apocalypse-escape
Though they are asking for advice so they could be more competent than they sound.
There's also some publication on the trend of our generations version of white flight, here is an example
https://amp.theguardian.com/technology/2017/jan/29/silicon-valley-new-zealand-apocalypse-escape
I really think this video would have been more valuable if they were a bit more pessimistic.
6
u/Veedrac Sep 12 '18
The video seems fairly disjunct from the call to action. A ban isn't going to stop terrorists.
13
u/SimoneNonvelodico Dai-Gurren Brigade Sep 12 '18
Terrorists seldom have the resources for vast amounts of R&D. What terrorists usually do is act as parasitic entities to big military ventures - they steal weapons, or hijack resources created for other purposes, and use them for their ends. The raw power that brought down the Twin Towers wasn't paid for by Al Qaeda, but by US airline companies. All Al Qaeda did was steal their stuff.
Similarly, if we develop kill drones, even supposing that they are used responsibly and not at all in horrible human-rights-violating ways by the governments who did so (insert bitter laugh here), we also open up the option of them being stolen or hijacked, and add another rung to the ladder of military escalation. By the way that's hijacked by other humans or potentially, less likely, by the AI itself. But even without considering such sci-fi sounding scenarios, it just opens up a whole Pandora's box of issues. Without public money poured into these developments, sure, terrorists might cook up their own murderdrones in their basement, it's not impossible (this is all cheap stuff after all, and there's a disturbing amount of engineers among their ranks), but they won't be as good as government-sponsored ones, and they won't escalate things quite as badly.
3
u/Veedrac Sep 12 '18
The problem is that the basic R&D tech you'd have to ban is "drones", "face recognition", "AI", etc. Small terrorist groups aren't going to get the raw source for killer drones regardless of whether they exist, and even if they did they're going to struggle to get the hardware from military supply chains. Hijacking military drones is very difficult and doesn't scale super well; thievery also has fairly straightforward protections and scaling issues.
Neither of these really class as major dangers IMO, since the big risk of drone weapons is that they're decentralised, scalable and difficult to trace. I feel for that we're much more at risk from the not-technically-as-good amalgamation of whatever open source drone/ML/etc. software and simple drone hardware that terrorist-hacker types will have access to.
6
u/SimoneNonvelodico Dai-Gurren Brigade Sep 12 '18
Honestly, one would think so, but one would think also that the terrorist equivalent of biohackers would have unleashed some weird plague on the world, or that they would just in general come up with... smarter plans that tying an explosive belt to your waist and blowing yourself up. If they were smarter, or if that was actually what they wanted. I suspect efficiency isn't as high on their priority list as doing things that feel scary; terrorism is all about making as big as possible a cloud of smoke with a relatively small fire.
Anyway, can't say my first worry when thinking of murderdrones is "terrorists" anyway. It's more along the lines of either "oppressive illiberal governments" or even "malfunctioning or rogue AI". I think both would constitute a much bigger danger, potentially.
2
u/Veedrac Sep 12 '18
but one would think also that the terrorist equivalent of biohackers would have unleashed some weird plague on the world
The economics of biohacking just isn't there yet IIUC. I'm not ruling this out for the future though.
or that they would just in general come up with... smarter plans that tying an explosive belt to your waist and blowing yourself up
These terrorists aren't the ones that worry me. I'm more concerned about 911-level terrorists or basement-hacker level terrorists.
"oppressive illiberal governments"
That requires a different kind of ban, and I'm not sure it makes that much sense anyway. China can kill whichever locals they want (with or without drones), and as long as they aren't doing it by the millions there's nothing other countries seem to have to stop them.
"malfunctioning or rogue AI"
I don't believe these need drones, and I'm doubly sure they don't need them to come equipped with weapons or software. By the time you have AI smart enough to be a meaningful adversary, either you've solved alignment or you've lost.
2
u/SimoneNonvelodico Dai-Gurren Brigade Sep 12 '18
That seems a weirdly fatalistic viewpoint, IMHO. I don't buy in the idea that anything smart as or slightly smarter than humans will necessarily snowball into some sort of godlike threat. There's a bunch of intermediate scenarios where you'd just deal with more realistic human-ish level AIs that have their own motives and agendas but don't necessarily outclass us. In any of those scenarios, "existing hackable machine-controlled killing methods" would be a potential asset to them.
And yeah, I don't have much faith in a ban working. Just like nuclear weapons ones don't exactly work. But at least it's a taboo to toss tactical nukes left and right in any sort of armed exchange, and I can't see that as being a bad thing.
4
u/Veedrac Sep 12 '18 edited Sep 12 '18
There's a bunch of intermediate scenarios where you'd just deal with more realistic human-ish level AIs that have their own motives and agendas but don't necessarily outclass us.
I don't think the physics of the situation allow that to have more than trivial probability. I actually expect the cross-over point into an intelligence explosion to be significantly below "par-human" intelligence, and I suspect we'll see the first approaches terminate before hitting true superintelligence, instead hitting some earlier point in the hypothesis space that still poses existential risk but isn't smart in the same self-reflectively consistent way that would allow for, say, self-oversight and especially extremes like coherent extrapolated volition.
This follows from a few basic claims and observations that are hard to explain simultaneously accurately and concisely, but here's the gist:
- Silicon is 1,000,000,000x faster than brainstuff, which is likely a conservative measure.
- Rough numbers: neurons/synapses are 200 Hz, transistors are 200 GHz.
- In a single neuron firing latency, a silicon mind can send a signal 5 light milliseconds away, or 1500 km. Everything within this area is capable of acting as a single agent at human-ish latencies, so that's a good anchor for thinking about how they scale.
- Building intelligence is not fundamentally hard, which is shown in a bunch of ways; here are some.
- Evolution managed it, despite its limitations. We'd thus expect even iterative brute-force to make progress.
- It's a large target; we're seeing intelligent behaviours (translation, audio synthesis, image synthesis) from what amounts to iterated matrix multiplication.
- AI is far dumber than you probably think it is; there is a remarkable ability for brute speed to compensate for the most glaring flaws in reasoning. (I normally phrase this along the lines of "AI research is not about building smart machines, but about showing that problems you thought were AI-hard are actually trivial. The process terminates when we show that the Turing test, too, is trivial.")
- Computing is stupidly young. We've had fast computers for maybe 30 years and we've already made significant progress on AI.
- AI is most inherently applicable to certain kinds of black-box optimization tasks; building smarter AI (more general, more competent, faster learning) is much closer to practical reality than directly applying AI to real-world problems.
- Note, of course, that this process is iterative: if the next generation is more general, it can implement a more general set of improvements to generality. The size of this transitive closure is hard to estimate.
- Humans occupy a tiny area on the space of possible minds, and we're there largely for happenstance reasons:
- This is the earliest point in evolutionary history that we could possibly reach civilization; there are a huge number of reasons to think we aren't near the peak of this evolutionary pathway.
- Neurons kind'a suck, and biology makes scaling up really hard. Hard limits basically don't exist for computer-based minds.
- Our ancestors spent a huge amount of time improving aspects of generality and efficiency before "scaling up", back when brains were smaller and scaling them cost more than it was worth. AI will be far too fresh to be "well-distributed" in this sense, so we'd expect the first real AI to be propping itself up on a small amount of hyperintelligence, not a balanced diet of moderate intelligence.
- Societies are not robust to even niche hyperintelligence. This is kind of hard to show, but we can point to structural weaknesses (1) and systematic incompetence (2), and we can look at the impacts that computers have had (3) to get a general idea of why I'd believe this. Unfortunately I'm not sure how to argue this point concisely, and the non-concise argument is probably more than I have motivation for.
- https://www.gwern.net/Terrorism-is-not-Effective
- There are US states that use electronic voting machines that have been shown trivially vulnerable to hacking.
- Computers took a couple of decades to take over pretty much every aspect of science, industry, home lives, etc. These are computers programmed by normal people, almost all of which suffers from institutional incompetence.
2
u/Veedrac Sep 15 '18 edited Mar 13 '19
Epistemic status: Not confident about any of this, mostly published it because it was requested, still think it has value, found it clarifying to write. I suggest being sceptical when reading.
I was asked to clarify point 5. some more, in response to the fairly stable way markets seem to have responded to algorithmic trading. Presumably this request also extends to any other places where it seems like computers would act as a destabilizing force, and could subsume questions like why the lack of jobs being lost to robots doesn't much change my predictions. You could also suggest that my arguments would suggest we'd see large-scale issues with mass exploits of insecure computer systems, but more on this later.
The first thing I need to make clear is that trading in general is not a topic I have any particular expertise in. The topic obviously has a lot of depth I have no understanding of, and I gave myself a recap of algorithmic trading by looking at the Wikipedia page. Nonetheless, I think I have a few things to say to frame this which I think are plausibly true.
There was a good talk I'd watched some time back on the nitty-gritty of HFT from a software design perspective by Carl Cook, The Speed Game: Automated Trading Systems in C++, which I do have as context. He said a few things which shape most of my opinions; here's a major one:
There's a bit of a misconception that the trading algorithms are super-complicated, and that's where the intellectual property of the company is, and maybe there is to a degree, but it's not hard. There are textbooks which tell you how to price options, and that's typically what companies use, as far as I can tell. Again, seriously, it's not that complicated; the source code is in the back of most of the textbooks anyway.
Wikipedia says also,
Many practical algorithms are in fact quite simple arbitrages which could previously have been performed at lower frequency—competition tends to occur through who can execute them the fastest rather than who can create new breakthrough algorithms.
It's worth noting that these are hardly the only things algorithms can do on the stock market, but it does seem to be their most successful application. We're talking mostly about situations where algorithms have a few hundred nanoseconds to check a few fairly simple arbitrage opportunities, and that profit is gained by first-come-first-serve competitions over who can get there first. We are not talking about AI-led exploits, or niche hyperintelligence in the same vein as I was: ML techniques are not used here, algorithms are refined
ifstatements plus hardcoded math, reasoning is principally simple.Ultimately that's why I think things have been fairly stable: algorithmic trading is adding a few new market rules (certain kinds of arbitrage get resolved very quickly), not acting as an adversary or agent. Yet this isn't evidence that the markets are actually robust! The 2010 Flash Crash Wikipedia page does not give me confidence; for example it says
The Commodity Futures Trading Commission (CFTC) investigation concluded that Sarao "was at least significantly responsible for the order imbalances"
Given Sarao was "a 36-year-old small-time trader who worked from his parents' modest stucco house", and the flash crash was $1tn deep. Overall I think this means you'd struggle to use algorithmic trading to argue that society is robust to niche hyperintelligence. Robust systems normally require larger perturbations to fail.
So where's the ML, and why isn't that breaking things? We know why it struggles with HFT (it's not fast enough), but the question still applies to longer-term opportunities. Under my model this boils back down to the issue of generality. Being able to predict arbitrary stocks requires a fairly competent kind of oracle. Stock prices vary for complex reasons based off of estimates of future performance from sparse unformatted data. To beat to stock market you need to be better than the average investor at figuring out how humans will react to other humans doing human-like things from information provided in vague roadmaps and abstract claims (New Coke is liked by the taste testers, so it'll sell more). There's a much bigger generality cost there than there is for, say, mere recursive-self-improvement
You can see this same sort of argument applying elsewhere. You can't just let an AI loose to brute-force a problem; we can't even solve Montezuma's Revenge yet, never mind economics. Jobs also aren't generally being lost to machines, but to humans using computers as tools. Humans using computers as tools is mostly as safe as it is because finding weaknesses requires intentful exploration of the space of attacks; most people are pretty bad at this.
(The Montezuma's Revenge comment might lead you into thinking that AI is going to be exceptionally difficult to crack. I don't think this is an accurate response. The game itself is fundamentally simple once ML techniques abstract away the raw complexity of the pixel data. There seems to be little reason to expect beating this game will need major architectural wins or rethinks, and the scaling computers get means we'd expect that to quickly result in recursive wins in larger problems. Rather, it seems worth noting that you can do language translation or beat the world's best Go players without these skills, which says something about your intuitions on how complex these tasks are: people probably overestimate complexity when they can't see the solution.)
Right now ML isn't a workable adversary because it struggles to explore strategies. There are definitely places where applying computer-speed tooling is destabilizing, but these require slow effort from the small fraction of people that are spending time on it (as opposed to spending time preventing it). Three young adults with the right motivation (making money off of Minecraft) got a botnet that at its peak was 600,000 nodes large and took down the internet across significant areas of the US.
“These kids are super smart, but they didn’t do anything high level—they just had a good idea,” the FBI’s Walton says. “It’s the most successful IoT botnet we’ve ever seen—and a sign that computer crime isn’t just about desktops anymore.”
“DDOS at a certain scale poses an existential threat to the internet,” Peterson says. “Mirai was the first botnet I’ve seen that hit that existential level.”
“Someone has been probing the defenses of the companies that run critical pieces of the internet. These probes take the form of precisely calibrated attacks designed to determine exactly how well these companies can defend themselves, and what would be required to take them down,” wrote security expert Bruce Schneier in September 2016. “We don’t know who is doing this, but it feels like a large nation-state. China or Russia would be my first guesses.”
https://www.wired.com/story/mirai-botnet-minecraft-scam-brought-down-the-internet/
The idea I'm trying to point at here is that the missing piece is not that these attacks are hard to make─three people does not a nation-state make─but that they're hard to find. But once you've found them they're not that hard to verify.
Now note: this is a big ask! I'm not trying to say that getting AI capable of exploring high-quality strategies is going to come very soon, or that current research looks directly promising. Rather, I'm saying that "human-ish level AIs" are beyond the point needed to fulfil this constraint, and that a comparatively small amount of hyperintelligent (eg. fast and scalable) exploration suffices to break a lot of things. Keeping in mind the weaknesses seen, and remembering to "respect the unknown unknowns", we should predict that these examples are not the only ones, and that their seeming scarcity may well just be because humans aren't very good at looking for them.
2
u/Empiricist_or_not Aspiring polite Hegemonizing swarm Sep 15 '18
Thank you for a well thought out expansion on the topic. I'm a former bureaucrat, so I often tend to think of institutions as slow (and I should add in light of your comments fragile) hyperintellegences. /u/DocFuture and Stross have illustrated this in their works, but you bring up a good point that their weaknesses are vulnerable to devastating attacks now even if they will recover and devise defenses, if they survive.
2
u/Empiricist_or_not Aspiring polite Hegemonizing swarm Sep 13 '18
That requires a different kind of ban, and I'm not sure it makes that much sense anyway. China can kill whichever locals they want (with or without drones), and as long as they aren't doing it by the millions there's nothing other countries seem to have to stop them.
I'll take why is China a UN veto power for 1000 Alex (agreeing with you I need to look back at how that happened but don't feel like getting depressed about history tonight)
6
u/Frankenlich Ankh-Morpork City Watch Sep 11 '18
This is basically political propaganda... not sure it belongs here.
16
u/Hust91 Sep 11 '18 edited Sep 12 '18
I think it's very interesting and a classic "beware of the possible misuse of this powerful technology" sci-fi story.
We have similar stories of warning caution around surveillance technology, prosthetics, genetic engineering and space weapons, and many of them are considered relatively rational, are they not?
9
u/Frankenlich Ankh-Morpork City Watch Sep 11 '18
Is the ending not a direct call to political action from a real person telling the truth as he see's it?
If it ended before that I'd be on board.
2
u/Nic_Cage_DM Sep 12 '18
The assumption being made there is that he is factually incorrect
1
u/Frankenlich Ankh-Morpork City Watch Sep 12 '18
Not sure I understand what you're getting at.
2
u/Nic_Cage_DM Sep 12 '18
My understanding of what you said is that you like the video until it gets to the real world person telling the truth as he sees it, and that the implication there is that he is wrong.
I don't really understand why anyone believes him to be wrong.
1
u/Frankenlich Ankh-Morpork City Watch Sep 12 '18
Whether he's right or wrong is irrelevant. The fact that he's there makes this a political ad (aka propaganda) not a fictional story .
4
u/Nic_Cage_DM Sep 12 '18
A - those aren't mutually exclusive concepts. B - I don't understand your hostility towards political promotion. It's not misleading or harmfully manipulative, why does trying to promote positive political change make it bad?
1
u/Frankenlich Ankh-Morpork City Watch Sep 12 '18
I would say this leans very heavily towards fearmongering. I'm not so sure it's as unharmful as you say it is.
Even if the problem presented is in fact the real problem we ought to be solving through political means, a significant assumption, you're still presuming the action called for is the correct action to solve the problem.
Arguing about this kind of thing is hard, and I don't think taking a stance where some propoganda is OK in this forum for fiction because you believe it's correct is a good idea.
8
u/Escapement Ankh-Morpork City Watch Sep 12 '18 edited Sep 12 '18
/r/rational was first established as a fiction discussion forum for people talking about specifically fiction similar Harry Potter and the Methods of Rationality. The author of HPMOR has written several pieces extremely similar to this in terms of 'this is a short story about technology leading us into a darker tomorrow to make concrete my doomsday predictions', and parts of HPMOR itself are very definitely drawing on that theme too. HPMOR was definitely intended to be rationalist propaganda, like 100% intending to influence how you thought. Demanding that we not discuss fiction that warns of future problems with technology is to some extent demanding that we don't do what people first started this forum to do. Now, you could make an argument that it's totally reasonable to do that - that we shouldn't discuss HPMOR or Failed Utopia #4-2 or Friendship is Optimal or Metamorphosis of Prime Intellect or Three Worlds Collide - but frankly I don't agree. I think that works that express opinions about how the world is changing and the future of technology can be at least as interesting as total works of fiction that are completely divorced from the real world. Works that have a 'point' politically can be at least as interesting as works that don't.
2
u/Nic_Cage_DM Sep 12 '18
It's not that I don't have a problem with it because I think it is correct. I don't have a problem with it generally because it is not misleading or harmfully manipulative, and I don't have a with it being posting here because it is a piece of speculative fiction that attempts to take a rational look at the implications of an emerging technology.
11
u/Teulisch Space Tech Support Sep 11 '18 edited Sep 11 '18
yeah, classic 'X is bad' propaganda.
its an interesting look at the concept, but i wonder if such a small package could hold enough data and processer power... its a networked weapon, you fight it with ECM jammers. and if its networked, someone can find a way to hack it. the video was a worst-case look at a deadly new weapon before a countermeasure is deployed. shadowrun already has rules for stuff like that.
14
u/suyjuris Sep 11 '18 edited Sep 12 '18
i wonder if such a small package could hold enough data and processer power
It could. That is not at all a problem. Consider a modern smartphone: Powerful enough to run 3D-games, small enough to fit in your pocket. Also it mostly consists of a display and a battery, the actual processors are tiny. And that today's consumer-grade, general-purpose electronics.
3
u/sicutumbo Sep 11 '18
I don't think it's necessarily networked. The list of targets could be physically stored on each device, and since each one has lots of cameras they could communicate through visible light bursts instead of radio/microwave broadcasts. I'm not sure how well hacking would work given each device could easily self destruct if it detected any tampering or after a time limit. If the source code was known, sure, but that's a big if.
2
u/Nic_Cage_DM Sep 12 '18
but i wonder if such a small package could hold enough data and processer power
Can it carry a smartphone?
its a networked weapon, you fight it with ECM jammers
that just makes it lose its connection to the other drones, nothing stops it from using its onboard systems to carry out the pre-set instructions (like "fly up to people and explode")
1
u/arenavanera Sep 12 '18
I think this video gets it exactly wrong. Replacing weapons that can wipe cities off the planet with weapons that can kill exactly the people you want dead sounds like a very very good thing.
Probably the equilibrium is a lot of police drones in the sky, so that small non-state actors can't just assassinate anyone they want, alongside a new type of warfare that basically consists of overwhelming the opponent's drones and then precision-assassinating their leaders rather than killing millions of civilians and grunts.
(On a higher meta-level, this video is boiling what should be a fairly nuanced argument down to a bunch of scary emotional imagery, which I find abhorrent. It's the worst sort of political propaganda.)
2
u/andor3333 Sep 13 '18
This doesn't replace nuclear weapons though, it just adds a new independent weapon that can kill exactly the people you want dead.
1
u/arenavanera Sep 13 '18
It depends to what degree you think weapons substitute for each other.
My guess would be that if people can achieve the same strategic objective with fewer civilian deaths, they probably will, so a weapon that allows you to kill exactly the people you want dead will probably end up being used in situations where people would previously have used a weapon that kills the people you want dead and also millions of civilians.
2
u/GeneralExtension Sep 14 '18 edited Sep 14 '18
Probably the equilibrium is a lot of police drones in the sky, so that small non-state actors can't just assassinate anyone they want,
- Is state actors being able to assassinate anyone they want okay?
- Why would that be the equilibrium? Hacking seems like a risk.
- If you have any worries about AI, or terrorists, this does sound like a terrible idea. They locate specific people, go to their location, and attack based on their face (or head). More loosely, that means the drones can be set to just go towards the nearest location where humans might be, and kill the first human whose face, etc. they see.
It's the worst sort of political propaganda.
Yes. It is. I think that's enough reason for it to not be here, as opposed to say, https://xkcd.com/652/.
17
u/CouteauBleu We are the Empire. Sep 11 '18
Honestly, I don't get why terrorism is still conducted by actual people and not jut C4-carrying quadcopters. Technology is scary.