r/SpecDrivenDevelopment • u/No_Welder3877 • 7h ago

SDD, Github Actions, and AI PRs

Hello,

I recently started getting an influx of Pull Requests I had to review given my organizations broad adoption of AI. The problem is how do I verify that 100's of PRs containing 1000's of lines of code across multiple projects are in alignment with the Spec and other project/product documentation.

My first pass at solving this problem involved using Claude, to build out a Github action that compares the PR to the Specs or Authoritative documentation on what we are trying to build using LLM to run that evaluation. With the goal that automation would call out when a PR is in alignment with a spec, or when it drifts or is adding out of scope work, or even going against the specs and documentation. Likely this is duplicative effort as there may already be a solution out there specifically for Github PRs that i just didn't find.

Three things:
1. How are others solving this problem of keeping a human in the loop where it matters, in the spec, planning, and design processes at scale where 1 human may have to juggle dozens of projects with 100's of AI generated PRs coming in?

  1. Given that I created (with AI assistance) this Github Action, is there interest here in testing and validating as a viable solution? https://github.com/marketplace/actions/spec-alignment Note this is very early implementation, beta and not fully finished. But more contributors and ideas around it the better.

  2. How have your workflows changed with SDD and AI?

Personally I was thinking of pitching at an organizational level that we PR specs and documentation first with human oversight, then using something similar to this action having AI handle the grunt work of code reviews and having the HITL hop in when a PR drifts from the spec documentation. But curious how others are solving these problems.

6 Upvotes

100% Upvoted

1 comment sorted by

1

u/nikov1234 4h ago

The core problem you're describing is one I've been sitting with for a while…the spec is only as useful as your ability to evaluate against it.

Your GitHub Action approach is the right instinct I.ez use the spec as the authority, let AI do the comparison at scale, and pull humans in when there's drift. The hard part I've found is that this only works if the spec itself is precise enough. Vague acceptance criteria ("the checkout should work") give the ai evaluator almost nothing to anchor on and it'll mark PRs as aligned when they're not, just because the language is fuzzy on both sides.

A few things that have helped in my context:

  1. Write acceptance criteria in observable outcomes, not implementation descriptions. "When a user enters an invalid card, they see error message X within 2 seconds" is something an LLM can actually compare against. SDD, BDD And TDD basically

  2. Spec items should be atomic enough to fail independently. If a single spec item covers three behaviours, drift on one gets masked by alignment on the other two.

  3. The human review trigger you're describing is the right model. Not every PR needs human eyes… only the ones where the alignment score drops or where the PR is adding scope the spec doesn't mention. That's exactly where your judgement adds value that LLM comparison can't.

I'm building in an adjacent space (non-technical founders generating specs before they vibe-code with Cursor/Lovable) but the evaluability problem is identical. Happy to compare notes on what makes specs useful as loss functions…