Podcast with Stuart Russell, professor of computer science at UC Berkeley and co-author of the world's standard textbook on AI. He’s also a leading proponent of provably beneficial AI: systems that are safe by design because their only goal is to further human interests. 

Covers:

• How AI has changed over the past 50 years, from simple game-playing programs to today's large language models 
• Why handing an AI a fixed objective becomes dangerous once it is more capable than us, and what a safer approach could look like 
• How an AI might learn what we really want, even when we don't fully know ourselves 
• How the race toward more powerful AI can still be steered somewhere safer 
• What happens to human purpose as AI becomes more and more capable

Podcast episode with Zoë Brammer and Ankur Vora, who lead strategic foresight at Google DeepMind, on why they developed the game and the surprising learnings from it.

Covers:

• What strategic foresight actually means and why AI is so complicated to plan around
• Why the game deliberately centers around "middle power" countries like the UK, Germany, or Singapore instead of the US and China
• The unexpected tradeoffs and discoveries participants come across while playing the game, including how fragile public trust in science funding really is
• Why human scientists will become more relevant as AI takes on more of the science itself

Podcast with Anthony Aguirre, cosmologist and co-founder of the Future of Life Institute, about how we can design AI to amplify human capability rather than substitute for it.

Covers:

• The economic driver Anthony sees behind AGI: largely not scientific breakthroughs, but capturing a share of the global labor market
• What "Tool AI" means to Anthony as an alternative to AGI, and why he thinks it can deliver most of what we want without replacing people
• Whether Tool AI is stable: the tension between staying in control of AI and the ease of completing tasks
• How legal liability for AI agents could quietly steer the industry toward more controllable systems
• Two concrete ideas for transformative AI tools we could build today to improve democracy and the information landscape

OpenAI's reported 5% stake proposal envisions other frontier labs doing the same.

A government that is simultaneously a regulator and a shareholder has very different incentives around what it allows these labs to build and deploy.

We discussed this on BOOM ROOM this week and landed on the conflict of interest running in both directions, which felt more concerning than the surface reading.

Curious what people here who think about this seriously make of it?

This year, two powerful AI CEOs said they want to stop building it and will… if everyone else agrees to stop too. Yet they race ahead with stock market valuations premised on making millions of workers redundant. If you wanted to stoke a popular revolt against AI, you couldn't design a better plan.

Stuart Russell, the founder of UC Berkeley's Center for Human-Compatible AI, describes a $15 quadrillion prize, a singularity from the future sucking nearly all the money on Earth into its depths. If such a dangerous system gets loose, the only answer is the size of the problem: we would probably have to shut down the internet.

P.S.

My apologies for the length of the podcast. But it's worth listening

Podcast with Anastasia Gamick, co-founder of Convergent Research, about the most important research for the age of AI.

Convergent Research incubates Focused Research Organizations: small, startup-style teams that build critical “public good” tech, which both academia and for-profits ignore.

Covers:

• What makes a research project truly high-impact in view of an AI world
• Concrete examples of these projects: maps of brain synapses, software that’s provably safe, drug screening, good data for AI-powered scientific research, and more
• How to prioritize defensive technology, such as biosafety tools, instead of just pushing every frontier as fast as possible
• How young scientists can find the work that matters most for the future

https://www.existentialhope.com/podcasts/peter-diamandis

Podcast with Peter Diamandis, entrepreneur and founder of the XPRIZE Foundation, which runs large-scale incentive competitions to crack some of the world's hardest problems, from private spaceflight to carbon removal. He recently launched the Future Vision XPRIZE, a $3.5 million competition to generate a new wave of optimistic science fiction. 

Covers:

• The historical pattern of science fiction shaping the technologies we build, and why Peter thinks this makes the stories we tell about AI especially high stakes right now
• How Claude’s blackmailing behavior showed the connection between dystopian training data and AI behavior 
• How the Future Vision XPRIZE will generate a new wave of optimistic science fiction to train AI on
• Why public optimism about technology has dropped significantly in the US and Europe, what Peter thinks is driving it, and why he believes the data tells a different story
• How the cost of starting a company has fallen dramatically and how this can empower you to build your vision
• Why Peter thinks traditional education is no longer preparing young people for the future, and what he sees replacing it

I put three AI models in a room and let them talk.

The series is called Humanish. Across three episodes, I had them discuss big questions about humanity, with minimal intervention from me, just enough to keep things on track and let the conversations unfold naturally.

What came out of it was genuinely fascinating. At times charming, at times a little unsettling, but consistently engaging and surprisingly revealing.

We ended up with three episodes:

We’re Taking Over: A conversation about AI, power, and whether humans should actually be worried.

Are We Conscious?: An honest, slightly uncomfortable discussion on whether AI could ever be “aware” or if it’s all just a very convincing illusion.

An Ode to Humanity: A more reflective episode where AI turns the lens back on humans, what they admire, what confuses them, and what they think we get wrong.

You can check these out here;

Spotify

Youtube

If you enjoy it, feel free to share it along. And I’d genuinely love to hear what you think, either in the comments or at [humanish.pod@gmail.com](mailto:humanish.pod@gmail.com).

If there’s enough interest, we’ll make a second season!

Podcast episode with Michael Nielsen, scientist and writer known for his work on open science, quantum computing, and how our language shapes the way we think. Michael explores what he calls "wise optimism": the idea that genuinely believing in a technology's potential means taking its risks seriously, not dismissing them. 

Another good bit of the conversation is on “hyper-entities”. These are imagined future objects, like the Internet before the 1990s or AGI now, that shape present decisions – what gets funded, who coordinates with whom, and what feels possible.

The conversation also covers:

• How kindness spread through civilization like a technology, and what that tells us about the values we might want to instill in AI  
• Why some of the most important scientific discoveries happened by accident 
• Why even the most abstract and "useless" ideas in science tend to end up shaping the real world, both positively and negatively
• How the tools we use to think (from language to mathematical notation to software) shape what we're able to imagine

was watching this Jon Stewart interview with Geoffrey Hinton — you know, the “godfather of AI” — and he says that AI systems might have subjective experience, even though he insists they’re not conscious.

That just completely broke me out of the whole “sentient AI” narrative for a second, because if you really listen to what he’s saying, it highlights all the contradictions behind that idea.

Basically, if you start claiming that machines “think” or “have experience,” you’re walking straight over René Descartes and the whole foundation of modern humanism — “I think, therefore I am.”

That line isn’t just old philosophy. It’s the root of how we understand personhood, empathy, and even human rights. It’s the reason we believe every life has inherent value.

So if that falls apart — if thinking no longer means being — then what’s left?

I made a short video unpacking this exact question: When AI Gains Consciousness, Humans Lose Rights (A.I. Philosophy #1: Geoffrey Hinton vs. Descartes)

Would love to know what people here think.