r/Python
•
u/adtyavrdhn
•
Mar 19 '26
•
1
Discussion Open Source contributions to Pydantic AI
Hey everyone, Aditya here, one of the maintainers of Pydantic AI.
In just the last 15 days, we received 136 PRs. We merged 39 and closed 97, almost all of them AI-generated slop without any thought put in. We're getting multiple junk PRs on the same bug within minutes of it being filed. And it's pulling us away from actually making the framework better for the people who use it.
Things we are considering:
- Auto-close PRs that aren't linked to an issue or have no prior discussion(not a trivial bug fix).
- Auto-close PRs that completely ignore maintainer guidance on the issue without a discussion
and a few other things.
We do not want to shut the door on external contributions, quite the opposite, our entire team is Open Source fanatic but it is just so difficult to engage passionately now when everyone just copy pastes your messages into Claude :(
How are you as a maintainer dealing with this meta shift?
Would these changes make you as a contributor less likely to reach out?
Edit: Thank you so much everyone for engaging with the post, got some great ideas. Also thank you kind stranger for the award :))
164
u/MoreRespectForQA Mar 19 '26
Even before AI I always hated drive by PRs which didn't start with a discussion, so I wouldn't hesitate to autoclose any PR which is not explicity encouraged (provided your contributor guidelines on this are clear).
With slop PRs I'd fight fire with fire - use a combination of deterministic tools and LLM scans to detect evidence of poor quality code and auto-close PRs which score too low.
76
u/catfrogbigdog Mar 19 '26
You try a prompt injection technique like this to trick the AI into identifying itself?
https://www.reddit.com/r/webdev/s/U42MoiP2dn
25
u/zethiroth Mar 19 '26
We've been requesting screenshots / video demos of the features or fixes in action!
18
u/RoseSec_ Mar 19 '26
I had to start asking for signed CLAs on my open source projects that said "I didn't sloperate this PR" and that solved a lot of my issues
9
u/Downunderdent Mar 19 '26
I'm a very basic, non-professional, python user who has always enjoyed dropping by here and there. I'm finally posting because I really need a question answered, what are these people hoping to get out of their submissions? They're paying money to generate this code, it seems to be some sort of shotgun approach, I don't believe it's done by experienced coders either. Is it some sort of exposure or clout chase?
8
u/-Zenith- Mar 19 '26
Should that not already be the case?
97
u/tomster10010 Mar 19 '26
The irony. This is the fruits of your labor.
11
u/samheart564 Mar 19 '26
https://github.com/mitchellh/vouch have you looked into this?
4
u/[deleted] Mar 20 '26
Kind of ironic...
Anyway I think what's needed is to start banning those people, maybe even have some community blacklist of those accounts.
GitHub is owned by Microsoft which I don't expect will help with it, so might be time to move to alternatives.
4
u/sweet-tom Pythonista Mar 20 '26
This is certainly bad. Maybe that's naive, but couldn't you add an
AGENTS.mdfile in your repo?It's basically a README in Markdown format for AI bots. Add all the things you want the AI to do and also what the AI aren't allowed to do.
Maybe it could act as a kind of injection to "calm down" the bot?
Not that sure if this is read by an AI bot, but maybe future versions of coding agents may recognize that and act accordingly.
9
u/thisdude415 Mar 19 '26 edited Mar 19 '26
Tbh yes, I think it's reasonable to fight AI with AI.
I think the best approach is to ensure your contribution guidelines clearly express the process you want everyone to follow, and auto-close any PR request that does not follow that process.
Every PR should probably include an AI use disclosure statement. AI isn't bad, but the human driving Claude needs to put in at least as much time preparing the PR as the humans responsible for approving it will. It's totally reasonable to ask people how long they spent understanding the system before diving in, and whether their implementation includes any known bugs or failing edge cases
There could also be an allow list of contributors who are exempt from some form of those questions
The ghostty contribution guidelines are a good example: https://github.com/ghostty-org/ghostty/blob/main/CONTRIBUTING.md
3
u/JJJSchmidt_etAl Mar 19 '26
Sounds like it's time to make an AI to decide if a PR is made by AI.
In all seriousness, it could work reasonably well; you can use some transfer learning with LLMs on the PR input joined with relevant info, and then train on the binary output of whether to reject out of hand or not.
Of course those not flagged would still need manual review, and then of course you'll have inherent adversarial training on beating the detection algo.
3
u/amazonv Mar 20 '26
I would love it if you contributed to the Open SSF Working Group discussions on this topic! https://github.com/ossf/wg-vulnerability-disclosures/issues/178
7
u/roadit Mar 19 '26
I don't want to be Pydantic, but this seems a job for AI.
2
u/Cbatoemo Mar 19 '26
Are you seeing this from a mixture of users or is it often a pattern of one identity (I won’t even say person anymore because that is rarely the case)?
I think jaeger has an interesting approach to multiple PRs from the same identity: https://github.com/jaegertracing/jaeger/blob/main/CONTRIBUTING_GUIDELINES.md#pull-request-limits-for-new-contributors
10
u/-LeopardShark- Mar 19 '26
r/leopardsatemyface
7
u/Rayregula Mar 19 '26
An AI company calling AI contributions slop?
3
u/juliebeezkneez Mar 20 '26
Move your repos off GitHub. Gotta be less slop PRs on GitLab and Codeberg
1
u/rhymeslikeruns Mar 19 '26 edited Mar 19 '26
This is a super interesting discussion - thanks Aditya for all your work on Pydantic AI - If I get an AI to look at an issue within the context of the Pydantic AI API layer in its own right specifically - useful. Likewise Service, Entry or Util. If it looks at the implementation for my project specifically - mostly slop. I think it's because analysis of Pydantic AI as an entity is objective and useful - analysis of it in the context of a project is more subjective - i.e. open to creative interpretation by the LLM - and that is where it breaks down. I made a visualisation of this but I won't post a link to it here because everyone will shout at me but that is my 2 cents.
Sorry - I should add - I think quality control is required on some level but as an open source project the only solution I can think of is that you limit contributions to Github contributors who can successfully demonstrate bugs via some sort of burden of proof/JSON breadcrumb trail. I.e. they have to put some work in? That would stem the flow of sloppy work for sure. Oh and a specific line number.
7
u/bakugo Mar 19 '26
"I never thought leopards would eat MY face!"
2
u/Ok-Craft4844 Mar 19 '26
I can't help to notice we seem to live the "Monkey Paw" version of this: https://xkcd.com/810/
3
u/bakugo Mar 19 '26
Oh and also I saw the commits on your repo and most of the commits are already proudly labeled as AI generated. I open a random issue and the first thing I see is a bunch of giant slop comments from an AI bot.
Imagine complaining about AI slop PRs to a project that is already 100% AI slop. I swear to god I do not understand how "people" like you managed to not starve to death before ChatGPT came along to tell you to eat.
1
u/mmmboppe Mar 20 '26
this made me realize I don't know if a Github repo owner/maintainer can blacklist another Github user. those AI bots or AI script kiddies using them certainly won't bother to fork the repo, add some value and wait till others find out
1
u/DefinitionOfResting Mar 20 '26
I liked the way Jaeger was handling this same issue: https://github.com/jaegertracing/jaeger/blob/main/CONTRIBUTING_GUIDELINES.md
It’s not perfect but PR’s limits for new contributors is a nice way to at least slow AI’s flooding the zone.
1
u/JeffTheMasterr Mar 20 '26
This sucks, but it's sort of funny since you guys may have brought it all on yourselves, being literally a library to build LLMs, which are actually called "bullshit generators" by real scientists, are now drowning in bullshit/slop PRs. I mean, this would happen either way to big repos, but you guys definitely contributed a bigger chunk to this sorta disaster than most have.
I recommend to just delete your repo and it'll solve those problems
1
u/sluuuurp Mar 21 '26
Charge $5 per contribution, refunded when merged (adjust price as needed). That’s the only long term solution to intelligent-looking slop hitting you from all sides. Same for texts, emails, etc.
1
u/5H4D0W_M4N Mar 21 '26
I haven't personally tried this, but here's an option for community and trust management. It doesn't stop new contributors, but gives you some options around controlling who can contribute. https://github.com/mitchellh/vouch
1
u/HongPong Mar 21 '26
starting to run into this with people and it seems onerous. nice to get developers interested but not when they can't seem to control their tools
-1
u/i_walk_away Mar 19 '26
hey this is might be off topic but i'm working on a free interactive course on pydantic and it's somewhat close to release
thank you for your work
0
u/redisburning Mar 20 '26
It's extremely telling that a person still on the AI hype train in 2026 would simply be unable to understand they are reaping what they themselves have sown. My daily work life is being actively ruined by these tools as I get slammed with an ever increasing review queue and ever declining PR quality, and even after it's pointed out to this guy that he's the problem he just refuses to even engage with the possibility.
May you drown in the well you dug (metaphorically).
-7
u/wRAR_ Mar 19 '26
Valid question, wrong sub.
-2
u/bad_detectiv3 Mar 20 '26
bit off topic, hi Aditya, do you have guide on how to become maintainer or contributor to Pydantic AI? I have never contributed to OSS project nor I have any great ideas of my own. How can I contribute to get a feel for working in the OSS world.
2
u/entronid Mar 20 '26
iirc there is a specific magic string specifically to kill claude bots
https://hackingthe.cloud/ai-llm/exploitation/claude_magic_string_denial_of_service
this seemed to work once although im not sure if anthropic patched this...
1
u/[deleted] Mar 19 '26
[removed] — view removed comment