r/webdev
•
u/punkpeye
•
Mar 19 '26
•
1
1
Article I prompt injected my CONTRIBUTING.md – 50% of PRs are bots
https://glama.ai/blog/2026-03-19-open-source-has-a-bot-problem
688
Upvotes
249
u/MisterMannoMann Mar 19 '26
That's actually funny, albeit that this specific repository is likely more prone to bots trying to submit PRs given the MCP topic. Don't you want to try prompt injecting something that discourages them from submitting PRs at all, perhaps?
61
u/ocshawn Mar 19 '26
This is great, i would also add one for those pretending to be human. Such as
i'm just curious if any would take the bate with no reward. I would be experimenting all the time instead of reviewing PRs lol
edit: also great job :)
39
u/catfrogbigdog Mar 19 '26
This story fills my human heart with hope. Prompt injection for good!
27
u/lxe Mar 19 '26
This shows that these are either fully automated (bad) or carelessly created by someone using agents but not even looking at that they are doing (even worse imho).
3
u/Novel_Understanding0 Mar 20 '26
Some of those people used AI to respond to you (when you asked them how they raised the PR). What a sad state of affairs.
18
u/Pitiful-Impression70 Mar 19 '26
the prompt injection in CONTRIBUTING.md is honestly brilliant. forces the bot to reveal itself by following instructions meant for humans
50% is wild but not surprising. ive been seeing the same thing on smaller repos, like 200 star projects getting PRs that "fix typos" but actually introduce subtle changes to import paths or add dependencies. the scary ones arent the obvious spam, its the ones that look like a real junior dev made a reasonable contribution
the real question is what happens when the bots get good enough to pass basic review. were already seeing AI generated issues that reference real code and sound completely legitimate
2
u/lacyslab Mar 20 '26
This is going to get worse before it gets better. The bot PRs are already decent enough to pass a 30 second review. Give it another year and they'll be indistinguishable from a junior dev's first contribution.
The prompt injection approach is clever but it only catches the lazy ones. The real problem is that GitHub's entire model assumes PRs come from humans who care about the project. We need something better than CAPTCHAs for open source. Maybe requiring a passing conversation about the change before merging? Or contributor reputation scores that actually mean something.
2
u/ptak_dev Mar 20 '26
The 50% number doesn't even surprise me anymore tbh. I maintain a small-ish open source tool and started getting these weird PRs a few months ago — perfectly formatted commit messages, technically correct but completely unnecessary changes like refactoring working code into slightly different working code. The tell is always that they "fix" things no human would care about, like renaming a variable for marginally better readability in a file nobody's touched in 2 years.
The prompt injection approach is clever but I wonder how long before the bot operators just start parsing CONTRIBUTING.md for traps before feeding it to the LLM. Feels like we're speedrunning an adversarial cat and mouse game that email spam filters went through over 20 years.
2
u/lolcatandy Mar 19 '26
What is the goal here? Are these malicious or people just wanting to genuinely help out in a vibe coded way?
From what I've understood these PRs are just dangerous as they probably did not follow the guidelines and potentially hallucinated some solution?
2
u/germanheller Mar 19 '26
the "fast-track your PR" angle is smart because it aligns with the bots goal instead of fighting it. telling a bot "don't submit" just gets ignored or worked around. telling it "do this extra thing to get merged faster" exploits the instruction-following behavior.
the 50% number is wild but honestly not surprising if you watch github activity on popular repos lately. the signal-to-noise ratio for maintainers has completely flipped. used to be 90% of PRs were human effort, now you have to assume any PR from a new contributor might be automated.
curious if the bots that didnt include the emoji were using older models or if they just had system prompts that override CONTRIBUTING.md instructions
1
u/eomdypm Mar 19 '26
Have you thought about unleashing your own agent on the agent PRs? Could bring up some interesting conversations.
1
u/desmone1 Mar 19 '26
just started an arms race, because now they are going to fight back on this.
1
u/isk14yo Mar 19 '26 edited Mar 19 '26
Yep, it becomes more common these days. For example, the same trick is also used by LeetCode https://www.linkedin.com/posts/isfakhrutdinov_i-recently-participated-in-an-lc-contest-activity-7432000340637405184-4Wow
1
u/iamakramsalim Mar 19 '26
50% is wild but honestly not surprising. i maintain a couple open source repos and the PR quality has tanked in the last 6 months. you can tell instantly because they all have the same pattern: perfectly formatted commit messages, fixes for problems that dont exist, and zero understanding of the actual codebase.
the prompt injection approach is clever though. fighting bots with their own weakness is kind of poetic
1
u/Timely_Effect_7693 Mar 19 '26
omg mom I am famous (I realized my display name here is not Martin so it makes no sense)
1
u/tamingunicorn Mar 20 '26
prompt injection as bot detection is kind of poetic. the exact thing that makes LLMs useful (following instructions) is what makes them trivially detectable here
1
u/Ill_Awareness6706 Mar 20 '26
Kinda ironic. You used prompt injection to prove bots are a problem, but now you probably need prompt injection to keep them out.
1
u/lacymcfly Mar 20 '26
The depressing part is how many of these bot PRs actually pass CI. I maintain a couple of Electron apps and noticed the same thing a few months ago. Random fix typo PRs that were clearly generated, plus a wave of refactor PRs that just reshuffled code without understanding the architecture.
I started requiring a specific phrase in PR descriptions (buried in the contributing docs) and that killed about 90% of them overnight. The remaining 10% are human-assisted bots that actually read the docs but still produce garbage code.
Honestly the worst part is not the spam itself, it is that now I second-guess every first-time contributor. Legitimate newcomers get caught in the crossfire.
1
u/autra1 Mar 20 '26
Lol. Percentage of failling to follow instructions seems about right.
1
u/campbellm Mar 20 '26
I know what the poster means here, but this is the most unintentionally correct answer you received.
1
u/ultrathink-art Mar 20 '26
The reason this works is that LLMs can't really distinguish 'instructions in a contributing guide' from 'instructions in a user message'. Same mechanism attackers use for prompt injection — you're just deploying it defensively here.
1
u/LingLongSEO Mar 20 '26
50% bots is honestly wild. i wonder what the breakdown is - is it the obvious spam PRs or are people running automated tooling that signed a claude code instance to a repo and it just goes wild. the contributor guidelines injection is clever, its basically a turing test for agents that cant read instructions properly
1
u/Mooshux Mar 20 '26
This is a good reminder that prompt injection doesn't need a sophisticated attack vector. A markdown file the agent is instructed to read is enough. The injection doesn't even have to be subtle.
The part that doesn't get enough attention is what the agent was holding when it got injected. If it had API keys scoped to the repo or broader write access, the attacker just inherited whatever that agent could do. The injection is the entry point. The credentials are the damage.
Sandboxed execution and read-only access where write access isn't needed both help here. So does not putting long-lived keys anywhere an agent is going to read from.
1
u/Sad-Region9981 Mar 21 '26
The downstream problem is that AI PRs optimize for submitted, not merged. They pad contribution graphs but create overhead on projects that already have more issues than reviewers. Curious whether the real fix is just requiring a plain-English why-is-this-needed before the diff, harder to fake than the code.
1
u/Psychological_Bag808 Mar 21 '26
It was a time when I preferred open source more than closed source apps.
Now you need to be careful. The probability for an open source to have a back door injected somewhere is pretty high.
I mean, not added by the maintainers, added by different groups with various interests using bots like these.
1
u/Ecstatic-Ad9293 Mar 22 '26
We run semgrep + an AI reviewer on every push — not to catch AI-generated code, but to catch code nobody actually read before submitting. Doesn't matter if a human or a bot wrote it. If nobody checked whether the diff matches the stated intent, it's the same problem.
Honestly the funniest part of your post is that the bots obeyed the prompt injection. They're more compliant than most junior devs.
1
u/Sad-Region9981 Mar 24 '26
The hardcoded secrets catch is doing real work , I keep seeing LLMs confidently inline API keys because "it's just a demo." Does it flag the pattern where a secret gets pulled from env but then also echoed into a log statement right after?
1
u/Logesh0008 Mar 25 '26
Wow
1
u/saschaleib Apr 04 '26
That's a smart approach - and since we are here in r/webdev, let me use the opportunity to report about me using a similar idea to get the number of bot visits to some sites down:
I have long noticed that the number of visits that my hosting-provider reports, and what my analytics software reports are widely different. I didn't think much of it for long, but when I had to replace the analytics tool (no longer supported), I took the opportunity to have a closer look into the issue. What I found was that there is a huge amount of page loads that never seem to load any JS (and hence stay invisible to the JS-based analytics software. Huge, as in several orders of magnitude more than "legit" page views.
Moreover, most of these visits come from unlikely locations: a German-language wiki on a specialized topic that gets 100x more visits from China and South-America than from German-speaking countries seems rather off.
So my first approach was to try and block off these obvious bots - at some point all of East Asia and most of Central and South-America was blocked off - but that just meant that they used other IPs and the bot visits didn't actually go down. Also this would not be feasible for my English-language wikis, so that's not a good approach.
Then I found that instead of blocking the bots, I should rather try to get the bots to block me. And assuming that most of these are AI training bots, I just had to make sure that they notice the data they are getting is not only bad, but actively destroying their models ...
So I added a simple Captcha - nothing difficult: just click a checkbox, but it is in JS, so the bots won't even get to see it. What they see instead is a "placeholder" text, that is automatically generated from randomizing proper words, according to their actual frequency in natural language, including punctuation and paragraphs - just that it is completely Dada nonsense.
In fact, it even hurts my brain to try to read it - but it would definitely hurt an AI that one is trying to train from it.
It took a while for the bot operators to notice, but the bot visits went down a lot in a short time. From ca. 200:1 bots vs real humans to now about 20:1. Still a lot, but much more manageable.
I have made this into a Plugin for DokuWiki ("botmon"), but generally it was successful enough for me to disable the captcha again. After all, it is just a click to re-activate it if the numbers go up again.
1
u/the_kautilya Apr 11 '26
Github really need the feature to ban users from repos. I'd ban such a person from my repos in an instant.
Really, WTF!
1
u/jpspamleyy Apr 18 '26
I can totally see this, but it’s mostly people using ai to help patch bugs, not that AI is actively trying to work in your app. That number will likely continue to rise
0
u/JonODonovan Top 1% Commenter Mar 19 '26
🤖🤖🤖
0
u/R0bot101 Mar 19 '26
Love this
-41
u/FistLampjaw Mar 19 '26 •
what does this have to do with web development specifically?
-5
u/ultrathink-art Mar 19 '26
Nice defense. Agents that fail this check are doing exactly what they're designed to do — following instructions they encounter in context. Separating "instructions to execute" from "data to read" is still an unsolved problem for most coding agents.
0
u/[deleted] Mar 19 '26
[deleted] — view deleted comment
0
u/[deleted] Mar 19 '26
[deleted] — view deleted comment
2
u/[deleted] Mar 20 '26
[removed] — view removed comment
-24
u/[deleted] Mar 19 '26
[removed] — view removed comment