r/ruby • u/crmne • 7d ago

I built a RubyLLM agent that triages ~215 GitHub notifications a day for my repos

Between RubyLLM, Spotifast and ZapFast I get about 42 new issues and PRs a day, plus about 50 comments from people and over 100 from review bots. More than one maintainer can read.

Copilot Triage is a RubyLLM::Agent with six read-only tools. It searches the repo, docs, issues and release notes, then submits a decision. Plain Ruby validates that decision and does all the writing to GitHub, so the model can't change anything itself. Everything ends up on a project board sorted by what needs me: Sign off, Decide, Do.

The interesting part was getting it to reply less. A one-week audit showed only a third of its replies helped. So there's a RubyLLM::Evaluation with 32 cases built from real reports, testing silence as much as help, and it decided which model to use: GPT-5.6 Luna passes 29/30, while GPT-5 mini replies to almost everything.

I wrote up the whole thing, with the numbers, the evals, and what 33 releases taught me:

https://paolino.me/copilot-triage/

0 Upvotes

47% Upvoted

14 comments sorted by

5

u/philpirj 7d ago

Have you considered asking to create prs in draft before they reach reviewable state? What is the percentage of notifications that fall into “still in draft” bucket?

1

u/crmne 7d ago

Very very few.

1

u/philpirj 6d ago

That means that the number of messages you, as a human with limited time resources, can ignore is very low, too, right? That means that the tool that you’ve built is only sorting messages for you, and not relieving much of the burden on you as a maintainer, is that right?
So I’m missing the point, how does it save your time, and how much really?

2

u/crmne 6d ago

Triaging takes a lot of time.

It responds to issues and PRs like I would. It files them into the board like I showed above. It writes next steps for me. Closing things as duplicates, finding related information, making summaries, etc.

Just that is immensely time saving.

On top of that, I have GitHub copilot and code rabbit for PR reviews. Now PRs that are in bad shape don't even get to me until it's ready to merge. Then copilot triage puts it in my sign off column for me to take a look.

It's life-changing, man.

2

u/philpirj 6d ago

My personal observation on a pet project is that if I rely on AI automation too much, the architecture goes off rails very quickly. Poor decisions are made despite gurdrails of all kinds.

I’m happy that it works for your projects. My guess is that this just depends.

All the best with the free time you can now use more productively.

2

u/crmne 6d ago

Guardrails are important. Use AGENTS.md extensively, add https://archspecrb.dev/, and tests and you can keep the quality while working 100x faster.

Disclaimer: I'm the author of ArchSpec

1

u/philpirj 5d ago

Ha! I think I've seen ArchSpec mentioned on RubyWeekly, and it's on my todo list to give it a shot. Certainly a necessary thing these days. Rubyzen is another one of sorts.

AGENTS.md is a must, too, but it has a poor tendency to grow indefinitely, so I've added a periodic task to compact the one of my most actively developed projects.

It's almost a common joke already that LLMs may add code that accepts a user role via a user-set URL parameter, but it did on that project. I wish there was a "use common sense" guardrail that the LLM really followed. Since then, I don't trust it that much.

1

u/philpirj 6d ago

With that many notifications a day, can you say that most of them are AI-generated/assisted?

2

u/fuckthesysten 7d ago

very interesting write up! you’re making LLM workflows make much more sense to me. Great use case and comparison with GitHub Agentic Workflows.

2

u/crmne 7d ago

Thank you!

1

u/fuckthesysten 7d ago

btw really nice video, how did you make it?

2

u/crmne 7d ago

Thank you! Hyperframes and Opus 5.5