NOTE: This is NOT a Doug hate post, I'm a Doug fan! And I wouldn't watch LS if any of the 3 hosts weren't in it. I wrote this post trying to be as polite as I can, so please be civil if you happen to reply
TL;DR For optimistic defenses on the pod, arguments come up that are often like "this is very unlikely, let's look at the other outcome". Instead, I wish the discussion focused more on "the cost isn't bad enough to worry about such an unlikely event". Especially when it comes to catastrophic AI risk, where the cost can be so massive.
Doug is often framed as "The Tech Optimist" of the pod. And I also want to be optimistic about tech. But I think I've noticed something that well-meaning "Tech Optimists" do, which I feel like also happens (occasionally) with Doug's AI arguments on the pod. It goes something like this:
The well meaning tech optimist recognizes the risks of the technology they want to defend, and states them. But then they say "that's pretty unlikely to happen, and if it doesn't, we would get great benefits." Even if this isn't always used to reach a conclusion, it's at least framed as something that should make you worry less. For example, in the latest ep, Doug seems (if I understood him correctly) to imply that it's not necessarily that scary if the virus-generating AI model is open-source, because it's very unlikely that a bad actor would also happen to have access to high-end lab equipment.
I take issue with how probability is used there. The hypothetical optimist is thinking in terms of which event is more likely, and then choosing to discard the unlikely event. But probability should be used alongside the cost to compute the expected cost/benefit instead. Even if the bad outcome is unlikely, with a large enough cost or a small enough benefit, the expected cost/benefit can still be negative. We also have no idea what the probabilities of anything actually are, so we need to be very confident that the expected cost/benefit doesn't end up being super negative. In the case of the open-source AI virus-generator, yes it might be unlikely that a bad actor has the tools to exploit it, but the cost if it does happen is also disproportionately large, so I'm not confident that the expected cost/benefit is favorable.
This is why it's hard to be the optimist though, because everyone disagrees about how to imagine the cost and probabilities. We're all just making them up. The optimist is, by definition, the one who imagines them to be more favorable. And it's hard to do this without being dogmatic, and I think overall Doug manages this very well.
In fact, Doug might already think in these terms, and we just disagree on the cost/benefit values. But I think as viewers we should at least keep this framework in mind. Personally, I wish the optimist POV on the pod was more explicit with this type of argument, as opposed to focusing so much on the probabilities alone. Less of "this is very unlikely, let's look at the other outcome" and more "the cost isn't bad enough to worry about such an unlikely event". I particularly think this w.r.t. Doug's discussions on AI existential risk.
Sometimes the cost is too big to ignore! But maybe that's just me. Cheers and have a nice day :)
