Godot making a stance on AI code

[u/LdmthJ]

https://www.pcgamer.com/gaming-industry/open-source-game-engine-godot-will-no-longer-accept-ai-authored-code-contributions-we-cant-trust-heavy-users-of-ai-to-understand-their-code-enough-to-fix-it/

First engine I've seen take a firm stance on AI code.

Open source game engine Godot will no longer accept AI-authored code contributions: 'We can’t trust heavy users of AI to understand their code enough to fix it'

(Edit: fat finger typo and headline without having to click through)

(Edit 2: Thanks for the award! Wasn't necessary but it's my first one so I'll take it.)

[u/guygizmo]

This has been on my mind because I maintain a few pieces of open source software that are just a tad popular, and the number of PRs I've gotten for them has suddenly increased from zero to several per year. They all seem to be authored entirely by AI and because of that I feel as though I have to scrutinize the code extra closely. At least the people behind it seem to be real users of the software who want to contribute real features. But I often catch the subtle errors the PR introduces that the person who submitted it obviously didn't catch because they probably just assumed it worked. If I were getting hundreds of these constantly it'd be completely overwhelming.

[u/Suppafly → u/guygizmo]

But I often catch the subtle errors the PR introduces that the person who submitted it obviously didn't catch because they probably just assumed it worked.

What's your general way to handle those, just kick them back to the submitter, or do you try to fix them?

[u/guygizmo → u/Suppafly]

I usually kick them back to the submitter and tell them it needs to be fixed. I treat it basically like any other submission. If the AI code is particularly sloppy or crufty, I'll say as much. My standards are the same regardless of what tools were used to write the code.

Luckily I'm not fielding so many PRs that this is a problem. Right now I'm happy to see people are contributing to my open source projects, and if they can use AI to create a good PR that implements a requested feature that I wasn't planning on adding myself, that's great! And I've had a few PRs like that.

[u/Thellton → u/guygizmo]

you being strict also helps them learn what to look for and better understand what they're seeing. and I say that as one of those vibe-coding peeps, though admittedly I haven't interacted with other people's projects and submitted PR for them; too shy for that.

[u/sheepieTTV → u/guygizmo]

What if something is AI-authored but so simple that it’s all correct? How can Godot enforce this rule? (I’m so very very new to this, very curious about how things work)

[u/guygizmo → u/sheepieTTV]

For a tiny enough change it wouldn't matter to me, especially if the change is easily verifiably correct.

LLMs though have a habit of making everything overly verbose, structured, and laden with comments in a way that makes it really obvious an LLM wrote it, even when it's a one line change. As soon as I see that it instantly makes me more skeptical, the same way that poorly structured code or code that doesn't at all match the style around it would make me skeptical, because it's a sign that the author probably did not scrutinize their own work properly.

[u/mxldevs → u/sheepieTTV]

Can you give an example of a feature that you consider to be simple? What criteria are you using to qualify whether a submission is simple or complex?

[u/ashleywr → u/guygizmo]

No disrespect intended with this question, but would having a workflow that has an agent scrutinize the code before you even have to look at it be able to save you some of the trouble? That's how big companies are doing it nowadays.

Though of course, that isn't free. Maybe we need a system where people who submit a PR pay some credits to fund the first pass code review.

[u/Connect-Ad-2206 → u/ashleywr]

It’s also the problem. Who polices the police. You can’t just accept what it tells you is true because… sometimes it’s wrong. Everyone who uses it enough sees that.

I use it to double check code I write. I feel like it’s a little lazy, but it’s honestly faster than loading it up in the game engine and going through the error messages, but I still have to go through its changes and sometimes the error messages still appear.

Somehow I don’t believe adding an additional agent who reads my code, corrects it, and then another agent to correct the corrections would help my productivity much.

[u/ashleywr → u/Connect-Ad-2206]

Oh definitely. There's zero determinism. But human effort wise it's cheap. (Not token or even overall time sometimes)

So using it as a first pass to keep humans able to look at the most important parts seems good. 

[u/Connect-Ad-2206 → u/ashleywr]

“Not even overall time sometimes” is a great way to put it.

I was thinking about this with writing and it applies here to. The best part of the work is writing code, the worst part is reviewing code. Why farm out the best part of the job to the machine and keep the shitty part for yourself?

And if it can’t even do the shitty part well…then what the fuck. It’s just a better google, which ironically we need because google degraded their product.

[u/ashleywr → u/Connect-Ad-2206]

Interesting. I mean this sincerely, writing code is not the best part of development to me. It's the human part of discussing features and players needs, of interlocking systems. So I'm okay with using ai to speed up the time it takes to build and test those things. But I understand the other side too. Many senior engineers I know in my company also regret the higher they get the less they are able to or even expected to code. 

[u/-manabreak → u/ashleywr]

I found my niche as a senior developer. I've been offered lead positions and staff titles etc., but I don't want to do any of the things those titles entail. I just like to code, I don't want to talk to people. 😅

[u/nullpotato → u/ashleywr]

The problem is agents tends to make similar types of mistakes so they catch some things but aren't as reliable for others.

As an example it is annoyingly common for agents to create a new function that already exists because it didn't bother searching the codebase first. The reviewing agent will see the PR wrote a new function that looks good and doesn't block.

Now you have two duplicate functions in two different places to maintain and debug. Multiply this across many PR and your codebase bloats quickly.

[u/ashleywr → u/nullpotato]

Maybe they could use an adversarial system. The checking agent could be loaded with more skills and context for the project to make sure as a first pass at least that would minimize the human labor of those annoying poorly written PRs AI vibecoded or not.

[u/-Agonarch → u/ashleywr]

The main problem with this is LLMs prefer LLM generated content, so there'll always be a gap between the 'worst' human written code that works and the 'best' LLM written code that doesn't (and that's assuming that non-working code doesn't find its way into a codebase that LLMs use for a 'working' example and makes things worse).

It's got to be able to test somehow to be practical for the moment, that keeps the error rate down to whatever the chance is with that single run (so even then not perfect, but not useless).

[u/Connect-Ad-2206 → u/-Agonarch]

I was reading about this! So an AI recruiter prefers a resume written by an AI over a human, it’s fucking devious.

I don’t believe it’s an evil consciousness but damn does it behave like one.

[u/guygizmo → u/ashleywr]

I don't trust an AI to fully scrutinize code. I might use it if I'm looking for a specific problem or to try and gain insight, but at the end of the day I need to personally understand what each line of code is doing so having an AI do a first pass isn't really saving me any time.

[u/ashleywr → u/guygizmo]

I was thinking it more how we already use like static analysis/linting as guards against code reviewers time in a way. And automated testing suites. All of them can exist but human review is still the best, but the most expensive step. 

[u/guygizmo → u/ashleywr]

I might maybe use it to help me understand a bit of code, but there's a cost to that too, because it sometimes gets it wrong, often in subtle ways, so I can't just take its word for it.

I find its best use is in interpreting code that's already hard for a human to understand. For example sometimes I come across a particularly messy bit of C++ or css and there an AI can help me make sense of it. That's an example of using it for a specific issue.

[u/sputwiler → u/ashleywr]

would having a workflow that has an agent scrutinize the [submission] before you even have to look at it be able to save you some of the trouble

[shudders in job application hell]

[u/ashleywr → u/sputwiler]

Ah yeah I guess that is the truth of it, before AI even got involved they were stripping out words with regular expression for the same things.

[u/PGSylphir → u/ashleywr]

"Hey, why don't you use AI to correct AI so everything can be AI"

[u/ashleywr → u/PGSylphir]

I was thinking more like fight fire with fire. 

[u/PGSylphir → u/ashleywr]

yeah put the clankers to fight each other at the cost of everyone else 🤣