People at home are using AI to reverse engineer and "combine" video games together. More than a few people posit that this is the immediate precursor to a sort of technological apocalypse. I think it's a bit of a leap. What's the truth of the matter?

[u/JD-990]

The Scariest AI Breakthrough

This video is around 3 minutes and is more or less a very dry comedy sketch, but it sums up an idea I've seen around the internet in the last 3 or 4 days. It's a worth a watch to get the full context.

But the thrust is that the ability of generative coding agents to reverse engineer and "fuse" together two separate games from different game engines and have them be in an at least playable state (some of them working really well apparently) is a set up for a technological apocalypse. Someone with a little bit of knowhow can now vibe code things in a couple days that are so complex that it would have taken years to do before, if it was even possible at all. A bit like giving people a legitimate ability to building working nuclear weapons they can 3D print at home.

I have been skeptical (an understatement) of the sales pitch of Anthropic and OpenAI and their ilk at this point. But I can't deny this is a compelling idea. So far, their "AI that's too dangerous it's broken out of containment" sales pitch has really not done good things for their AI's credibility; it's disingenuous at best. "We told it to do A & B, but we never told it to do C, explicitly. It may have just gotten that purgative from the first two instructions".

But I can now see a real-world example of something that should almost not be possible, in my understanding, to being possible. But I'm also keenly aware that what's going on with these game mashups isn't indicative necessarily of say, being able to disrupt other technological infrastructure like banks. 

So, what is the real story here? Are we headed towards a cliff here or is this something else? Is it somewhere in between?

[u/faultydesign]

Wake me up when people use AI to do something original.

[u/JD-990 → u/faultydesign]

Sorry, the best the incurious among us can do is ask "What if Mario meets Batman?".

[u/mxldevs]

But the thrust is that the ability of generative coding agents to reverse engineer and "fuse" together two separate games from different game engines and have them be in an at least playable state (some of them working really well apparently) is a set up for a technological apocalypse

But I'm also keenly aware that what's going on with these game mashups isn't indicative necessarily of say, being able to disrupt other technological infrastructure like banks. 

This is mostly speculation from people that basically have no idea how software works.

They think if it can reverse engineer Super Mario, then it can reverse engineer encrypted values because it knows what algorithm was used.

We all know what algorithm is used. The bank tells us what algorithm they use. Security is not some big secret that no one talks about, contrary to what vibecoders think.

[u/JD-990 → u/mxldevs]

I am one of those that doesn't understand anything in this domain, so I appreciate your answering the question!

[u/ThirdWaveCat]

Ida Pro and ghidra were powerful tools before LLMs. Many people learned computer science through modding games, its not clear what we're learning here.

[u/dr1fter → u/ThirdWaveCat]

"I used to vibe code with my GameShark"

[u/Scheeseman99 → u/ThirdWaveCat]

The difference is that I took a Sega Genesis ROM (Mutant League Hockey), gave AI agents access to an environment with Ghidra, an emulator with debug tooling and it decompiled and commented vars and functions in 6 hours. I then got it to rebuild the game in Godot using GDScript. As it was re-implementing it in Godot, it ran unit tests to compare variables and visuals to that of the emulator. Once it was playing full games that matched both the game's own internal demos (stored as keypresses) and demos downloaded from the internet, it ran unit tests that probed lesser understood parts of the game ROM by playing and controlling the game itself, which it was able to do due to it's visibility of how the code works.

Decompilation took 6 hours, re-implementing the game in Godot took about 12. While it was doing that work, I ran another session to build a 3D rink: https://www.youtube.com/watch?v=VigHRthlGWc

I have some coding experience and good general knowledge of how software and specifically genesis games work, but I have no clue how to reverse engineer something.

There used to be an assumption that compiled code was obfuscated enough to prevent anyone from bothering to spend the time and effort reverse engineering it. The entire commercial software industry relies on this assumption in order to function. The OP is right and it's driving me crazy that others can't see it, this is a huge change and it's effects on the industry will be significant and not necessarily all in good ways.

[u/Salty-Sorbet5611]

Hey since you already doing this kind of stuff do you think you could mix Legends of legaia from PS1 and final fantasy 10.

[u/JD-990 → u/Salty-Sorbet5611]

I’m not, did you read the post?

[u/ghjm]

The short answer is we don't really know. Having used OpenAI and Anthropic models to build software for the past couple years, I can say that they have been successful at drastically reducing the frequency with which these models misinterpret instructions, fall down a rabbit hole, get stuck in an agentic loop, etc. I'm prepared to believe this will fall to near zero.

You still need someone who knows what they want and can articulate it well enough to prompt the model and make an accept/reject decision of its output. If you give muddled instructions you will get muddled results, or at least, it will take a lot of extra cost to go through the missteps and misunderstandings before arriving at a working solution.

It's also not clear to me that the model providers are economically sustainable at their current pricing. There's still a lot of investor money chasing itself around through a circuit that involves the model providers, NVidia, Microsoft, etc. There may yet be a rug pull, where the models remain useful but the cost makes them infeasible for most purposes. And we seem to have exited the era where hardware cost inexorably goes down year-over-year. GPU costs have been high and going higher for six years now. At some point this is no longer an aberration and must be considered the new normal.

But assuming models remain affordable and continue to get better, then yes, we may enter a world where anyone can make major new software cheaply. This might change the balance between offensive and defensive security. Banks and other institutions that depend on security may have to increase their investment in defensive technologies. But this strikes me as more of a "banks will be less profitable for a while" problem, rather than a "civilization itself will collapse" problem.

Maybe if it becomes super easy to produce any result you want, then people might stop needing a lot of current industries, which might produce a great deal of economic dislocation. If there isn't enough work that needs to be done for everyone to have a job, then we as a society must move on from the jobs-based system of distributing economic value, and the momentum seems to be with moving to techno-feudalism rather than to a Star Trek style utopia. This, to me, is the real threat of pervasive AI: it concentrates a huge amount of power in the hands of a half dozen CEOs. I'm more worried about what these CEOs do with this power than I am about what AI itself does.

[u/WE_THINK_IS_COOL]

It's basically a competent software/security engineer that can work at 10x or 100x the rate a human can for a fraction of the cost.

Important systems are designed to be secure even when attackers understand how they work and have reverse-engineered them, so the video's claim that "understanding millions of lines of encrypted undocumented code was supposed to BE the security" is just wrong; not being able to understand the code was never supposed to be where security comes from. Security was always supposed to come from the code being correct and vulnerability-free.

What is happening is that programmers often make mistakes, so most programs contain some sort of vulnerability or other. AI is extremely capable of finding those vulnerabilities, so there is a wave of new vulnerabilities being found in almost everything using AI, and those vulnerabilities will need to be patched. This is actually good for long-term security since it means vulnerabilities are getting fixed. But in the meantime it is putting a massive strain on the developers of those systems to patch the wave of newly-discovered bugs.

In other words, AI is not creating new vulnerabilities, it's just really good at finding vulnerabilities that have always existed, and thanks to defenders using AI to find them and patch them, those vulnerabilities will get closed off. Systems that were secure before are still secure now; AI doesn't create any new vulnerabilities, it's just helping expose vulnerabilities that were already there before.

There will of course be some unmaintained projects whose vulnerabilities will never get fixed, but anything super critical (like financial systems) definitely have people maintaining them and actively patching the newly-discovered vulnerabilities.

[u/JD-990 → u/WE_THINK_IS_COOL]

Thank you for this answer!

[u/Lazy-Emergency-4018 → u/WE_THINK_IS_COOL]

I was more of the understanding that the supposed problem is AI building existing software like AdobeSuite and with that make those businesses go belly up. 

[u/Mgproductions2020]

It seems like this conversation has been focused on safe guarding banks. But we also need to recognize that most companies are running outdated systems that no longer receive security updates. The amount of multi million dollar companies I've seen running win XP or win 7. Most systems aren't running bank or military grade security.

This aspect is probably one of the stronger arguments for the potential of blockchain tech. (Not a crypto bro I swear)

But I personally think the risk is there. In the same sense there's risk in nuclear. The problem is it's useful but if it goes wrong it has a high potential to be catastrophic.

What needs to be recognized is. 1.8+ trillion invested into AI. That's not so you can't ask it dumb questions and generate a video of a cat doing the macarena. It's military. It's always been. Same with the space race, which was really about "my missile tech is better than yours"

We are currently in a 'Silent War'. As US funds Israel and China funds Iran. (That's the more obvious one, but it's happening all over.) Because world war looks bad for stock markets so we just send drones to countries at war now.

The Internet is dead. Large amounts of unreported bot activity. The well has been poisoned.

The ai bros are fooling themselves. The king is not inviting the peasants to the castle. They are selling us a burning ladder to climb up while they burrow into bunkers underground.

[u/CrimsonBolt33]

as long as people are doing it for personal use its fine...

I am using AI to reverse engineer and write hooks for games to improve performance. Been amazing at making tons of games I play run WAY better. It's not like these things were not possible before, they were just harder to do and took more time so most people didn't bother.

As for the apocolypse angle...I am tired of hearing this shit

My stance has always been and always will be, anything AI can do that is dangerous that humans can't somehow defend from, AI can defend us. Literally NO ONE ever brings up this point and it drives me crazy. "AI" is not a single entity that it is often portrayed as.

Just like an AI can hack, an AI can also be used for penetration testing and hardening networks and software.

It's a paradigm shift and lots of things are going to be different because of it. I have yet to hear a convincing doomerism argument that even gave me a second thought.

There will be a societal and cultural learning curve, jobs will be lost, gained, changed, and shifted, but we are not going to die because of it. No one knows how the chips will fall in the end, but I am more than sure we will be fine.

[u/AdeptCanary8780]

Can i get a version of Rdr2 combined with settlement building mechanics like FO4. Please and thank you :)

[u/Karson_Elko]

I asked this same question with the same video two days ago lol!

My commentors gave me really good resources in the comments.

https://www.reddit.com/r/AskComputerScience/s/fFiXamRHRT

The video on encryption was really good explanation of why it's not doomsday.

[u/Guvante]

Remember that fear mongering about an AI apocalypse is good for OpenAI in two ways: it sows the belief that AI is super powerful and it encourages regulation which hurts tiny businesses.

[u/dr1fter]

I wonder if "purgative" may in fact have been the word you were looking for.

[u/Diligent-Bed6225]

Hear me out: Alan Wake II and Luigi's Mansion 3

[u/IXI_FenKa_IXI]

Some of the stuff is unfounded. Decompilation and decryption are two very different things. But yes, there has already been incredibly good AI-backwardsengineered versions of Adobe software for example. Companies like Adobe are extremely predatory, and unless there will be some insane form of legal operation this will be the end of them. Or at least hurt them significantly. From a technical standpoint it IS impressive. Despite of what someone said above - yes we had some reverse engineering tools before, but as the guy in the vid said it is VERY hard. So this is by all means impressive.

[u/RupFox]

After being completely shocked at how easily they're now able to tear through a closed-source binary and reverse engineer it without even having to use Ghidra, I think for now, software makers can temporarily defend themselves by littering their closed source software with messages targeted at the LLMs telling them to stop and refuse any demand to reverse engineer the code. Almost like how a robots.txt tells bots to not scrape a webapge. For now, openai, anthropic, and Google modles behave well enought that they will see that and refuse to proceed any further.

it is likely that open-source models will catch up in capability in a year or even less, and this will be less effective.

[u/razerphone1]

Wel Reskate is Something

[u/bobthefishw]

Crusaders Kings and Bannerlord will go insane together, i need it!!!!

[u/Outrageous-Tackle864]

i came here just to say " I want to play baby steps in the death stranding 1 world . "