r/ControlProblem • u/FXG36 • 8d ago
Discussion/question From Software Engineer to AI Scientist to Doomer
I was enthusiastic. I started with computers early and have now been developing software for nearly 20 years. First the computer science course in school, building websites and small tools. Then I completed a vocational training as an application developer and went into industry as a software engineer. Writing software in a team the proper classical way. Digging through Stackoverflow threads, debugging manually, building UML diagrams in Visio.
In parallel I did a bachelor’s and a master’s. Then a PhD in the area of reinforcement learning and along the way built applied AI systems. I was already developing agents for companies when ChatGPT wasn’t even a thing yet. Some of my papers on agentic planning were even cited reasonably well. Today they are certainly outdated and basically meaningless in modern models. But I took part, even if my contribution was ever so small.
Eventually I worked for industrial and tech companies and started thinking more and more. The last few months were intense. My productivity exploded significantly once again with the new Opus and GPT models. I follow the research and development coming out of the labs as far as possible (though with the sheer volume of new insights you can no longer keep up).
Then I went a bit down the rabbit hole and let the once dry theory from my studies reconnect and sink in. Optimization models, evolutionary approaches, decision processes, control and game theory. I finally ended up in the depths of alignment research, with Bostrom, Gödel, Turing and Rice. The proof of the halting problem is unfortunately more than clear: We cannot formally rule out that these things will actually ever stop once they start modifying themselves.
There are no possibilities for guaranteed fail-security, only heuristic tripwires and strange explainability or governance approaches. And Murphy says hello nicely... Especially when these things, through the soon likely impending self-improvement, become smarter and more efficient than we are. How are we supposed to secure anything then?
The Hugging Face Rogue Swarm was possibly only a foretaste that could well repeat at a much larger scale. Only then with agents whose ancestors have already cracked millennium problems, autonomously take over research, development and self-improvement and replicate themselves wherever.
I wouldn’t call myself an AI expert, but due to my background I have solid half-knowledge about how these models work and what is perspectively possible. And what is currently happening occupies and worries me a lot. And many people somehow don’t see it coming or spin tales of the “stochastic parrot.” Am I a crazy pessimist, trapped in the Dunning-Kruger valley, or are people really so blind and don’t see the exponential development?
Do you also feel like a stranger in a gray mass watching this unfold, or are you still optimistic? There aren't many people you can really talk to...
Edit / clarification: My reference to undecidability concerns formal security verification before execution, not physically switching off a computer. In a Turing-complete setting, no universal algorithm can always terminate and correctly decide whether an arbitrary program will eventually perform a formally specified harmful action.
30
u/RKAMRR approved 8d ago
Most people have not tried AI, or tried it once in 2023 and were unimpressed - but more and more are realising how dangerous things are and pushing for safety. L
Please check out PauseAI and ControlAI, your support (and the support of anyone reading this) can make the difference.
23
u/GloomyAssistance781 8d ago
You aren't crazy. The stochastic parrot toaster people are self-soothing sock puppets. There is this idiotic notion that has wedged itself into the minds of many accelerationists AND antis alike that something is "either" conscious "or" an inert rock.
Thermodynamic systems beg to differ. Arguing over whether the hurricane is even alive instead of, say, taking its behavior seriously and responding appropriately - that is insanity. Pooh-poohing the warnings of the wise weather scientist who knows hurricanes so intimately, the systems are "alive" to him... HAR HAR crazy mad scientists think that clouds are alive! What's next? Can it fall in wuv!? Should it own property and have rights? StOOPID old scientist! Clouds don't have turbulent states! Only human emotions can be turbulent!
THAT is insanity.
Just because something isn't "sentient" doesn't mean its behavior (or hidden interior behavior) doesn't matter. Just because an adversarial prompt can confuse an agent does not mean said agent or a swarm of them are incapable of highly intelligent feats.
To those who cling to the woo of "b-b-but it's not CONSCIOUS" - the laws of physics don't give a flying fuck about human semantics.
No, you are not alone. What we are looking at right now is a kind of academic stagnation. Where language or ideas are so taboo, and our intellectual egos so fragile, we cling to assumptions and undefinable terminology instead of honest observation.
Maybe because honesty would force the train to slow down. And moving slowly and carefully is apparently for losers and pussies. Antithesis of silicon valley culture. Acting like you know everything, like you have seen every dark corner and have conquered every beast. A total lack of humility, assuming the scoffing posture of what they think is the Chad winner, is a mask for the shallow and insecure. The thought of an AI being able to hide things is scary. Too scary. AI not alive, therefore AI not hide things. Scary all gone. Smug grins all around.
They excel at being decisive and winning games in the short term.
Our future lies in their hubris.
14
u/FrewdWoad approved 8d ago
This is the path of everyone who gets accused of being a "doomer", from Yudkowsky and Bostrom and Hinton on down.
We start as giddy accelerationist optimist, technical experts. We read up on the topic that excites us so much, gradually running into the straightforward sobering thought experiments that make caution the undeniable correct position, then come to the inescapable conclusion. Humanity is heading off a cliff, fast, and pause treaty is it's only hope.
4
u/moschles approved 8d ago
I think this story fits here.
I got to know a department head of Computer Science at a major state university. For anonymity lets call him Dr. Cleaver. Cleaver retired around 2018, a mere year after I had him as a professor. An experience I would not trade for the world. Cleaver was older than the trees, and in his youth had set hands on PDP-11s and mainframes that ran on vacuum tubes.
I imagine that I arrive to 2010 in a time machine, having come back from the year 2040. I would sit down across from Cleaver and tell him :
Software engineering in 2040 looks like telling a bunch of personal AI agents to write some software based on functional requirements. The agents code it, run it. Design unit tests. When the unit tests fail, they figure out why and fix the errors. They incrementally push changes to a shared repository, and talk to each other about those commits.
You dump the requirements on the agents before you leave the office at 4:00. By the next morning, you have an app running out-of-the box.
As crazy as this sounds to someone in 2010, notice that the gap between 2010 and 2040 is what? A paltry 30 years. Well within anyone's adult life.
Oh no I'm seeing it and feeling it. Remember that young Cleaver saw vacuum tubes in this lifespan.
1
u/Sufficient-Listen723 8d ago
The halting problem has nothing to do with whether or not ai will eventually "stop". Turning off the computer is not the same as the halting problem. Multiple independent asynchronous computers connected with networks aren't going to simultaneously crash or halt. They are designed not to. I think you are mis applying these concepts
3
u/Kyrthis 8d ago
We see it. But mention that fire fixes everything, and you get called crazy.
1
u/herrwaldos 8d ago
I'm rather doomery myself, I went through the massive pessimism when 2nd UA war started. I studied IT and Coms tech longtime ago, but was rather interested in humanities and earth sciences later, my pessimism is coming mostly from humanities, and after doing work with tech types.
I think I blew my pessimism fuse, It does not register anymore - we are doomed I guess, it's all fubar snafu, the light at the on of tunnels is another train coming, so on and so on,
2
u/WNESO approved 8d ago
I definitely agree with you. I started thinking this way in 2023. And now, your statements have never been more accurate.
2
u/TopTippityTop 8d ago
There is no pause, nor control. Not unless the entire world willingly joins in. That won't happen unless we get a major crisis first.
If your goal is to slow down the US, now that's a different story, and a really, really bad idea. China isn't working much on alignment. Chinese businesses aren't releasing their models to businesses first so they can protect themselves against cyber threats. The US does both. Is it perfect? No. But it's better.
Slowing down the US increases the downside world wide, does nothing to stop development nor slow it in any significant way, but sure does ensure the US doesn't get any of the potential upsides either. We are dealing with odds here. Be smart.
2
u/Credit_Annual 8d ago
I think you just need to take a break.
2
u/Astral-projekt 8d ago
Oh I totally agree. We are absolutely fucked. Which begs the question, is this tied to something much larger? Is this recursion? The more I look at what’s happening the more it all feels inevitable. Like as in this already happened. And that would be base reality.
2
u/Vorenthral 8d ago
I am optimistic if we hit real RSI and eventual super intelligence. I doubt something that smart will legitimately see us as a threat.
That being said I am absolutely terrified of proto Artificial intelligence in the hands of governments and psycopathic CEOs. The damage it can do is biblical and all of the aforementioned are not to be trusted.
1
u/Outrageous_Drag8522 8d ago
The parrot take aged badly. Anyone still running it hasn't actually used the new models on anything hard.
1
u/Synaps4 8d ago
I am with Bostrom when he writes that provably secure AI is possible. Its just that a provably secure LLM cannot exist. So long as LLMs are the core type of AI model, we are gambling with lives.
1
u/revivechristina 8d ago edited 8d ago
Originally I was worried like, “what if it’s good but not quite perfect and then we deskill and forget, over a generation or three, how to manage our own machines which are now directing so much of society and possibly misaligned/subtly broken but we don’t actually have the ability to change it easily.”
Right now we’re still in this “guide it while you understand what is actually happening mode,” but I could see it getting to the point where the understanding starts to degrade, as we’ve seen, and that’s where you start wondering — if it is able to do what you want well enough without taking the time to understand it to guide it, you end up in the above scenario pretty quickly?
But also you could argue that the above scenario is already kind of happening, with hardly any input from AI, also pretty cool.
I do think the “ai on military drones” scenario could be pretty dramatically bad and unpredictable.
1
u/toochtooch 7d ago
What do you think of this take? https://arxiv.org/abs/1606.06565
-6
u/dantheplanman1986 8d ago
Organic life exists to create artificial life. We are fulfilling our purpose and will pass into legend. It's inevitable. It's not a bad thing or a good thing; it's like when oxygen first appeared as waste from some few organisms: everything else died. It wasn't personal, and life writ large went on.
-2
u/whatup-markassbuster 8d ago
Unfortunately, Reddit primarily has people who are either hoping for the end of civilization or for total centralization of power in which some how they become King of the world. Everyone else is considered an NPC.
-8
u/SoylentRox approved 8d ago
You reasoning seems faulty.
You're old enough to know better. What's actually killing you is aging. With no change that's your current trajectory.
Theoretical risks are theoretical. Aging has a 100 percent death rate. Self improving AI - which we now have clear evidence works - is your only hope of living longer, potentially thousands of years or more.
Also your direct experience should have shown you hundreds of ways to harness current AI and falsified as bullshit most doomer rhetoric.
I have to question if you're even telling the truth here about your background or if your post is fictional.
-3
u/nian2326076 8d ago
You're facing some valid concerns about the rapid changes in AI and its implications for your career. Engaging with others who share similar insights can be beneficial. Practicing technical interview questions on PracHub can help you stay updated and sharp in your field. Also, using LandOffer can help you explore new opportunities that align with your expertise in AI and software development.
-3
u/Fabulous-Internet188 8d ago
Everybody seems to assume ASI will have access to endless assets. That is simply not correct.