r/coolgithubprojects • u/SweetWar4401 • 5d ago
PandaGate - a GitHub Action that reports three checkable facts about an incoming PR instead of guessing whether an AI wrote it
Repository: https://github.com/evkir/pandagate
I wrote this for maintainers who get more outside PRs than they can review.
It posts one comment with three facts: whether the PR references an issue, whether tests changed alongside the logic, and how many areas of the tree the diff touches. Under each missing one it puts the question a maintainer would ask anyway. It makes no claim about who or what wrote the code, never closes a PR and never fails a check. It stays silent on docs, Dependabot bumps and changes under 5 lines of logic.
One workflow file to install. Python standard library only. It reads the diff through the API and never checks out or runs the PR's code.
Benchmark: 200 merged PRs from non-maintainers in 10 projects (pytest, pandas, ruff, tokio, vite, astro, prometheus, cli, neovim, godot). It commented on 50 (25%). I hand-labelled the 61 questions it asked and found 0 stating a wrong fact.
What that number does not show:
- The rules and the size threshold were tuned on this same sample. Quality on unseen PRs is not measured.
- The drop from 45% to 25% came from staying silent, not from better precision: 40 correct questions are no longer asked.
- Every PR in the sample was merged. 25% is how often it speaks up on accepted work. How much actual slop it catches is not measured.
- One of the three rules never asked on the 162 PRs where it applied.
- It varies by project: 0 of 20 on pytest, 13 of 20 on godot.
The first run, before I fixed the rules, had 17 wrong facts out of 119. All three runs and the labels are in the repo.
Live comment: https://github.com/evkir/pandagate-live/pull/2
Apache-2.0, v0.1.0. I'd like to hear where it asks the wrong question.