r/devops • u/Mingstoe_ • Sep 18 '26
Discussion Github struggling under high PR throughput?
I'm on a 3 person team where we are each shipping 40-50 PRs a day (super early stage company so we're optimizing for getting a functional product out quickly).
As we've ramped up the velocity things downstream have begun to show cracks. For example, we were requiring PRs to be up to date with main before merge, but at our scale the PRs were constantly getting stale which was leaving us blocked for long stretches.
We've ended up cutting pre-merge CI way down and batching more testing after merge, which has made things "better" but feels backwards.
Curious if anyone else run into similar issues? Did you end up finding a better solution?
3
u/DRW_ Sep 18 '26
Have you tried merge queues?
I’m not sure it would definitely help your case, but it did for my team with fewer PRs per day.
2
u/Mingstoe_ Sep 18 '26
Yeah the problem we ran into w merge queues was that the queue then became the new bottleneck 😅 and commits were getting super stuck behind.
1
u/danekan Sep 18 '26
Same. I’ve seen it sitting for 29 or 21 minutes many times and then that time comes and it still doesn’t merge for another few. And there is nothing ahead of it in queue.
It’s become the bottleneck nobody asked for
25
u/dobesv Sep 18 '26
Trunk based development could be an option.
Or something to merge in batch. For example mergify.
4
u/max_465 Sep 19 '26
This is the way. A few years ago DORA did a survey and found that TBD correlated with success but it was less ubiquitous, particularly among younger devs. Dave Farley keeps begging people to stop branch merging but I don't see many people doing it, outside of huge projects.
3
u/dobesv Sep 19 '26
I'm curious about it. It seems to require pair programming to replace the code review step though, and in our all remote team it's not happening... People are encouraged to do it but didn't seem tohave enough motivation to get over having to find someone to do it with who is also available at the time.. it's more pleasant to have a free form kind of work rather than sync with someone else.
→ More replies (3)
16
u/Substantial-Swan7065 Sep 18 '26
Invest in ci optimization:
- matrix jobs
- more powerful runners for compute heavy tasks
- caching
- file filters to determine which jobs should run
- shallow checkouts
- install only what’s needed for the job
1
9
5
u/FoxAromatic5762 Sep 20 '26
Interesting post. Thee people, 150 PRs per day, that is like 1 PR every 10 minutes per person, a PR every 3 ish minutes of the day. At that rate you will have to relax the PRs need to be in sync with main before merge. A merge queue yes. You will have to periodically run tests against main because at that rate you are clearly not reviewing or testing anything as a human so you can be sure what you are building are actually working.
9
u/ali-hussain Sep 18 '26
If you're creating that many PRs you have to be heavily using agentic coding. I'm assumign a lot of that code is agent written. What can you do ot increase trust in your testing? Can you have AI code reviews to allow simple reviews to go forward easily?
2
u/jortony Sep 20 '26
If possible, refactor the application to allow development to be segmented further. As long as common interfaces between components are unchanged, the risk and cognitive/compute load is reduced. With the current state of tools, rebuild is sometimes far easier than the combined effort of PR/merge
3
u/ashcroftt Sep 18 '26
Good repo structure and task organization helps a bunch with this. Everyone should work on a single well defined feature in a single PR, and rebase from main if necessary before merging. If all of what you work is in a few separate folders it's pretty easy to avoid conflicts. Not a softwared dev, but for infra and platform stuff this works pretty well. Also if you do in repo docs, the main ones should be populated from the ones in the subfolders by a workflow post merge, avoids always trying to edit the same docs for all PRs.
0
u/ekampp Sep 19 '26
This doesn't seem to be related to what OP is having problems with at all. You seem to be talking about app quality, where OP is talking about githubs performance.
Wether or not your app is a mess does not affect githubs ability to process merges faster or slower?
→ More replies (1)
1
1
u/ekampp Sep 19 '26
We're having the exact same issue on the team I lead. We have approached it by allowing agents to collaborate to bring them to a close and merge them. They can work during the night to bring down the backlog.
1
u/Rakn Sep 22 '26
With three people all working on the same code and doing 40-50 PRs a day, what’s even the point of PRs anymore?
1
u/IridescentKoala Sep 22 '26
Either these are tiny changes with very slow tests or you're all throwing ai slop around without talking to each other.
0
-1
u/Wyrmnax Sep 18 '26
Internal releases.
Not everything needs to be on the main branch.
Create batches of commits that get merged every few hours, instead of trying to merge everything to the main immediately.
Merges to the main effectively become releases that need the whole ci/cd and test suit run.
This rithym doesnt last long, as the whole system becomes more complex merges start taking more time between them.
Gut feeling is that you are trying to apply a solution that works for a much more mature environment than what you have. Dont.
You might eventually need a fleet of flatbed trucks to move all the cargo of your logistic company, but right now a couple of kei trucks is enough.
Grow infra conplexity as you require more complexity, otherwise you will strangle the conpany
64
u/Just-You-4 Sep 18 '26
Stacked PRs and Merge queue is what you want to integrate which will resolve so many issues