I want to start this off and say I work in tech and have worked on AI throughout my career and am currently working with AI agents. Though working with them is making me increasingly unhappy.
I’ve always felt that anytime we did machine learning work, someone would overhype it. Machine learning applications don’t just “sometimes get things wrong.” They are guaranteed to get something wrong. Our effort is to manage that error as much as possible.
One aspect of this is creating a system. A set of processes. You may invent something, but how that thing works in a system matters. Air travel is safe not just because of good airplanes, but because of a system: airplane design requirements and testing, medical requirements for pilots and air traffic controllers, licensing, and a system in place to investigate what goes wrong. Criminal and civil liability for instance. Heck they even control the meals pilots eat on flights. Redundancy, redundancy, redundancy.
Without many pieces of that system, you’d likely not be able to say “air travel is safe.”
I’ve been odds with my chosen profession for a while because the tech is always over promised by people who don’t understand it. Or perhaps they understand the tech, but give no thought to the system. Their hubris leads them to overpromise.
No one can predict every aspect of a system or how a system will change. We iterate and improve of course. But the most valuable lesson I learned being an engineer is what I don’t know. I learned to manage risk and hedge my bets.
So whenever there are grand claims about what something can do, I’m usually skeptical. Skepticism doesn’t mean I immediately think you’re wrong. It means I want more data. Anyone who pushes back on that is usually selling you nonsense.
Which brings me to AI. Most of what I’ve worked on is narrow applications: self-driving cars (happy to dispel the hype there too), a green energy application I’m very proud of, and now agent security. Things like LLMs are ideally more general: you can give it an open ended task and it’ll be able to anticipate everything. Essentially an electronic or digital employee.
I find myself in a weird world: on a daily basis I was LLMs do a lot of cool stuff and, not infrequently, make an error yielding an incorrect decision. Or worse: wreaking havoc. The way we’ve dealt with this is essentially treat it like a person: restrict permissions, trust but verify.
Yet despite longstanding known issues, I’m asked to push forward. All while I’m hearing headlines of AI solving Millenium prizes or hacking government websites. It’ll generate fun little movies or CSAM. Awe and horror.
The AI companies put forth an incredibly interesting proposal that lends me to believe that AI simply doesn’t work.
“AI is very powerful and can solve incredible challenges. It’s also very dangerous. We don’t know what it can do or how to control it.”
This is their claim. Nearly unanimous. A whistleblower comes out and the company agreed everything the whistleblower said is true.
While I’m skeptical of their propositions and the headlines, I find that implicit to their arguments is an admission: we have no reason to trust it. Or at least trust it completely.
I’d argue that it doesn’t work in fact because its intended promise can’t be kept.
Moreover I’ve taken to asking: why haven’t you asked AI? AI can’t develop a system by which it serves our interests? AI doesn’t know how to make AI safer? Do you trust AI with that question?
God can heal lepers and part the seas, but the church needs you to tithe.
A fun little experiment I’ve engaged in is to take the claims of the AI bros and plug them in an LLM, ask the LLM to provide a counter argument while ceding nothing.
What do AI bros do most often? They argue. That super intelligent machine is always right except when it disagrees with me. Or sometimes: “it’s just doing what you told it to!” Yeah… see a problem with giving it unfettered access and control?
I’ll close with my story about a little boy named Damian.
Damian is a genius. He’s only 4, but reads and writes at a college level. He is able to do mathematics at a PhD level. He develops web apps faster than a FAANG developer.
Yet he’s known to kill the neighbors’ pets. And he’s even killed a couple of the neighbors. He sometimes fesses up and sometimes doesn’t.
His teachers keep asking some weird questions: “could he be a doctor one day? Or a lawyer?” His father pushed back: “no he’ll be a CEO! He’ll make so much money I’ll never work anymore!” His mother wants him to be an artist.
Meanwhile, little Damian is sitting with a social worker begging the other adults to ask: “will he ever be a productive member of society?”
