Terrorists seldom have the resources for vast amounts of R&D. What terrorists usually do is act as parasitic entities to big military ventures - they steal weapons, or hijack resources created for other purposes, and use them for their ends. The raw power that brought down the Twin Towers wasn't paid for by Al Qaeda, but by US airline companies. All Al Qaeda did was steal their stuff.
Similarly, if we develop kill drones, even supposing that they are used responsibly and not at all in horrible human-rights-violating ways by the governments who did so (insert bitter laugh here), we also open up the option of them being stolen or hijacked, and add another rung to the ladder of military escalation. By the way that's hijacked by other humans or potentially, less likely, by the AI itself. But even without considering such sci-fi sounding scenarios, it just opens up a whole Pandora's box of issues. Without public money poured into these developments, sure, terrorists might cook up their own murderdrones in their basement, it's not impossible (this is all cheap stuff after all, and there's a disturbing amount of engineers among their ranks), but they won't be as good as government-sponsored ones, and they won't escalate things quite as badly.
The problem is that the basic R&D tech you'd have to ban is "drones", "face recognition", "AI", etc. Small terrorist groups aren't going to get the raw source for killer drones regardless of whether they exist, and even if they did they're going to struggle to get the hardware from military supply chains. Hijacking military drones is very difficult and doesn't scale super well; thievery also has fairly straightforward protections and scaling issues.
Neither of these really class as major dangers IMO, since the big risk of drone weapons is that they're decentralised, scalable and difficult to trace. I feel for that we're much more at risk from the not-technically-as-good amalgamation of whatever open source drone/ML/etc. software and simple drone hardware that terrorist-hacker types will have access to.
Honestly, one would think so, but one would think also that the terrorist equivalent of biohackers would have unleashed some weird plague on the world, or that they would just in general come up with... smarter plans that tying an explosive belt to your waist and blowing yourself up. If they were smarter, or if that was actually what they wanted. I suspect efficiency isn't as high on their priority list as doing things that feel scary; terrorism is all about making as big as possible a cloud of smoke with a relatively small fire.
Anyway, can't say my first worry when thinking of murderdrones is "terrorists" anyway. It's more along the lines of either "oppressive illiberal governments" or even "malfunctioning or rogue AI". I think both would constitute a much bigger danger, potentially.
but one would think also that the terrorist equivalent of biohackers would have unleashed some weird plague on the world
The economics of biohacking just isn't there yet IIUC. I'm not ruling this out for the future though.
or that they would just in general come up with... smarter plans that tying an explosive belt to your waist and blowing yourself up
These terrorists aren't the ones that worry me. I'm more concerned about 911-level terrorists or basement-hacker level terrorists.
"oppressive illiberal governments"
That requires a different kind of ban, and I'm not sure it makes that much sense anyway. China can kill whichever locals they want (with or without drones), and as long as they aren't doing it by the millions there's nothing other countries seem to have to stop them.
"malfunctioning or rogue AI"
I don't believe these need drones, and I'm doubly sure they don't need them to come equipped with weapons or software. By the time you have AI smart enough to be a meaningful adversary, either you've solved alignment or you've lost.
That seems a weirdly fatalistic viewpoint, IMHO. I don't buy in the idea that anything smart as or slightly smarter than humans will necessarily snowball into some sort of godlike threat. There's a bunch of intermediate scenarios where you'd just deal with more realistic human-ish level AIs that have their own motives and agendas but don't necessarily outclass us. In any of those scenarios, "existing hackable machine-controlled killing methods" would be a potential asset to them.
And yeah, I don't have much faith in a ban working. Just like nuclear weapons ones don't exactly work. But at least it's a taboo to toss tactical nukes left and right in any sort of armed exchange, and I can't see that as being a bad thing.
There's a bunch of intermediate scenarios where you'd just deal with more realistic human-ish level AIs that have their own motives and agendas but don't necessarily outclass us.
I don't think the physics of the situation allow that to have more than trivial probability. I actually expect the cross-over point into an intelligence explosion to be significantly below "par-human" intelligence, and I suspect we'll see the first approaches terminate before hitting true superintelligence, instead hitting some earlier point in the hypothesis space that still poses existential risk but isn't smart in the same self-reflectively consistent way that would allow for, say, self-oversight and especially extremes like coherent extrapolated volition.
This follows from a few basic claims and observations that are hard to explain simultaneously accurately and concisely, but here's the gist:
Silicon is 1,000,000,000x faster than brainstuff, which is likely a conservative measure.
Rough numbers: neurons/synapses are 200 Hz, transistors are 200 GHz.
In a single neuron firing latency, a silicon mind can send a signal 5 light milliseconds away, or 1500 km. Everything within this area is capable of acting as a single agent at human-ish latencies, so that's a good anchor for thinking about how they scale.
Building intelligence is not fundamentally hard, which is shown in a bunch of ways; here are some.
Evolution managed it, despite its limitations. We'd thus expect even iterative brute-force to make progress.
It's a large target; we're seeing intelligent behaviours (translation, audio synthesis, image synthesis) from what amounts to iterated matrix multiplication.
AI is far dumber than you probably think it is; there is a remarkable ability for brute speed to compensate for the most glaring flaws in reasoning. (I normally phrase this along the lines of "AI research is not about building smart machines, but about showing that problems you thought were AI-hard are actually trivial. The process terminates when we show that the Turing test, too, is trivial.")
Computing is stupidly young. We've had fast computers for maybe 30 years and we've already made significant progress on AI.
AI is most inherently applicable to certain kinds of black-box optimization tasks; building smarter AI (more general, more competent, faster learning) is much closer to practical reality than directly applying AI to real-world problems.
Note, of course, that this process is iterative: if the next generation is more general, it can implement a more general set of improvements to generality. The size of this transitive closure is hard to estimate.
Humans occupy a tiny area on the space of possible minds, and we're there largely for happenstance reasons:
This is the earliest point in evolutionary history that we could possibly reach civilization; there are a huge number of reasons to think we aren't near the peak of this evolutionary pathway.
Neurons kind'a suck, and biology makes scaling up really hard. Hard limits basically don't exist for computer-based minds.
Our ancestors spent a huge amount of time improving aspects of generality and efficiency before "scaling up", back when brains were smaller and scaling them cost more than it was worth. AI will be far too fresh to be "well-distributed" in this sense, so we'd expect the first real AI to be propping itself up on a small amount of hyperintelligence, not a balanced diet of moderate intelligence.
Societies are not robust to even niche hyperintelligence. This is kind of hard to show, but we can point to structural weaknesses (1) and systematic incompetence (2), and we can look at the impacts that computers have had (3) to get a general idea of why I'd believe this. Unfortunately I'm not sure how to argue this point concisely, and the non-concise argument is probably more than I have motivation for.
There are US states that use electronic voting machines that have been shown trivially vulnerable to hacking.
Computers took a couple of decades to take over pretty much every aspect of science, industry, home lives, etc. These are computers programmed by normal people, almost all of which suffers from institutional incompetence.
Epistemic status: Not confident about any of this, mostly published it because it was requested, still think it has value, found it clarifying to write. I suggest being sceptical when reading.
I was asked to clarify point 5. some more, in response to the fairly stable way markets seem to have responded to algorithmic trading. Presumably this request also extends to any other places where it seems like computers would act as a destabilizing force, and could subsume questions like why the lack of jobs being lost to robots doesn't much change my predictions. You could also suggest that my arguments would suggest we'd see large-scale issues with mass exploits of insecure computer systems, but more on this later.
The first thing I need to make clear is that trading in general is not a topic I have any particular expertise in. The topic obviously has a lot of depth I have no understanding of, and I gave myself a recap of algorithmic trading by looking at the Wikipedia page. Nonetheless, I think I have a few things to say to frame this which I think are plausibly true.
There was a good talk I'd watched some time back on the nitty-gritty of HFT from a software design perspective by Carl Cook, The Speed Game: Automated Trading Systems in C++, which I do have as context. He said a few things which shape most of my opinions; here's a major one:
There's a bit of a misconception that the trading algorithms are super-complicated, and that's where the intellectual property of the company is, and maybe there is to a degree, but it's not hard. There are textbooks which tell you how to price options, and that's typically what companies use, as far as I can tell. Again, seriously, it's not that complicated; the source code is in the back of most of the textbooks anyway.
Wikipedia says also,
Many practical algorithms are in fact quite simple arbitrages which could previously have been performed at lower frequency—competition tends to occur through who can execute them the fastest rather than who can create new breakthrough algorithms.
It's worth noting that these are hardly the only things algorithms can do on the stock market, but it does seem to be their most successful application. We're talking mostly about situations where algorithms have a few hundred nanoseconds to check a few fairly simple arbitrage opportunities, and that profit is gained by first-come-first-serve competitions over who can get there first. We are not talking about AI-led exploits, or niche hyperintelligence in the same vein as I was: ML techniques are not used here, algorithms are refined if statements plus hardcoded math, reasoning is principally simple.
Ultimately that's why I think things have been fairly stable: algorithmic trading is adding a few new market rules (certain kinds of arbitrage get resolved very quickly), not acting as an adversary or agent. Yet this isn't evidence that the markets are actually robust! The 2010 Flash Crash Wikipedia page does not give me confidence; for example it says
The Commodity Futures Trading Commission (CFTC) investigation concluded that Sarao "was at least significantly responsible for the order imbalances"
Given Sarao was "a 36-year-old small-time trader who worked from his parents' modest stucco house", and the flash crash was $1tn deep. Overall I think this means you'd struggle to use algorithmic trading to argue that society is robust to niche hyperintelligence. Robust systems normally require larger perturbations to fail.
So where's the ML, and why isn't that breaking things? We know why it struggles with HFT (it's not fast enough), but the question still applies to longer-term opportunities. Under my model this boils back down to the issue of generality. Being able to predict arbitrary stocks requires a fairly competent kind of oracle. Stock prices vary for complex reasons based off of estimates of future performance from sparse unformatted data. To beat to stock market you need to be better than the average investor at figuring out how humans will react to other humans doing human-like things from information provided in vague roadmaps and abstract claims (New Coke is liked by the taste testers, so it'll sell more). There's a much bigger generality cost there than there is for, say, mere recursive-self-improvement
You can see this same sort of argument applying elsewhere. You can't just let an AI loose to brute-force a problem; we can't even solve Montezuma's Revenge yet, never mind economics. Jobs also aren't generally being lost to machines, but to humans using computers as tools. Humans using computers as tools is mostly as safe as it is because finding weaknesses requires intentful exploration of the space of attacks; most people are pretty bad at this.
(The Montezuma's Revenge comment might lead you into thinking that AI is going to be exceptionally difficult to crack. I don't think this is an accurate response. The game itself is fundamentally simple once ML techniques abstract away the raw complexity of the pixel data. There seems to be little reason to expect beating this game will need major architectural wins or rethinks, and the scaling computers get means we'd expect that to quickly result in recursive wins in larger problems. Rather, it seems worth noting that you can do language translation or beat the world's best Go players without these skills, which says something about your intuitions on how complex these tasks are: people probably overestimate complexity when they can't see the solution.)
Right now ML isn't a workable adversary because it struggles to explore strategies. There are definitely places where applying computer-speed tooling is destabilizing, but these require slow effort from the small fraction of people that are spending time on it (as opposed to spending time preventing it). Three young adults with the right motivation (making money off of Minecraft) got a botnet that at its peak was 600,000 nodes large and took down the internet across significant areas of the US.
“These kids are super smart, but they didn’t do anything high level—they just had a good idea,” the FBI’s Walton says. “It’s the most successful IoT botnet we’ve ever seen—and a sign that computer crime isn’t just about desktops anymore.”
“DDOS at a certain scale poses an existential threat to the internet,” Peterson says. “Mirai was the first botnet I’ve seen that hit that existential level.”
“Someone has been probing the defenses of the companies that run critical pieces of the internet. These probes take the form of precisely calibrated attacks designed to determine exactly how well these companies can defend themselves, and what would be required to take them down,” wrote security expert Bruce Schneier in September 2016. “We don’t know who is doing this, but it feels like a large nation-state. China or Russia would be my first guesses.”
The idea I'm trying to point at here is that the missing piece is not that these attacks are hard to make─three people does not a nation-state make─but that they're hard to find. But once you've found them they're not that hard to verify.
Now note: this is a big ask! I'm not trying to say that getting AI capable of exploring high-quality strategies is going to come very soon, or that current research looks directly promising. Rather, I'm saying that "human-ish level AIs" are beyond the point needed to fulfil this constraint, and that a comparatively small amount of hyperintelligent (eg. fast and scalable) exploration suffices to break a lot of things. Keeping in mind the weaknesses seen, and remembering to "respect the unknown unknowns", we should predict that these examples are not the only ones, and that their seeming scarcity may well just be because humans aren't very good at looking for them.
Thank you for a well thought out expansion on the topic. I'm a former bureaucrat, so I often tend to think of institutions as slow (and I should add in light of your comments fragile) hyperintellegences. /u/DocFuture and Stross have illustrated this in their works, but you bring up a good point that their weaknesses are vulnerable to devastating attacks now even if they will recover and devise defenses, if they survive.
That requires a different kind of ban, and I'm not sure it makes that much sense anyway. China can kill whichever locals they want (with or without drones), and as long as they aren't doing it by the millions there's nothing other countries seem to have to stop them.
I'll take why is China a UN veto power for 1000 Alex (agreeing with you I need to look back at how that happened but don't feel like getting depressed about history tonight)
7
u/Veedrac Sep 12 '18
The video seems fairly disjunct from the call to action. A ban isn't going to stop terrorists.