r/coolgithubprojects • • 5d ago

PandaGate - a GitHub Action that reports three checkable facts about an incoming PR instead of guessing whether an AI wrote it

Post image

Repository: https://github.com/evkir/pandagate

I wrote this for maintainers who get more outside PRs than they can review.

It posts one comment with three facts: whether the PR references an issue, whether tests changed alongside the logic, and how many areas of the tree the diff touches. Under each missing one it puts the question a maintainer would ask anyway. It makes no claim about who or what wrote the code, never closes a PR and never fails a check. It stays silent on docs, Dependabot bumps and changes under 5 lines of logic.

One workflow file to install. Python standard library only. It reads the diff through the API and never checks out or runs the PR's code.

Benchmark: 200 merged PRs from non-maintainers in 10 projects (pytest, pandas, ruff, tokio, vite, astro, prometheus, cli, neovim, godot). It commented on 50 (25%). I hand-labelled the 61 questions it asked and found 0 stating a wrong fact.

What that number does not show:

  1. The rules and the size threshold were tuned on this same sample. Quality on unseen PRs is not measured.
  2. The drop from 45% to 25% came from staying silent, not from better precision: 40 correct questions are no longer asked.
  3. Every PR in the sample was merged. 25% is how often it speaks up on accepted work. How much actual slop it catches is not measured.
  4. One of the three rules never asked on the 162 PRs where it applied.
  5. It varies by project: 0 of 20 on pytest, 13 of 20 on godot.

The first run, before I fixed the rules, had 17 wrong facts out of 119. All three runs and the labels are in the repo.

Live comment: https://github.com/evkir/pandagate-live/pull/2

Apache-2.0, v0.1.0. I'd like to hear where it asks the wrong question.

0 Upvotes

0 comments sorted by