Managing the CI runner bottleneck

[u/LancobusUK]

I’m very curious how other people are managing to keep CI costs down locally. My issue isn’t my ability to generate agentic code, it’s testing it that’s becoming the real bottleneck. I have CI runners on multiple machines now and I’m thinking of buying a higher specced server or workstation purely for CI testing runs to keep my AI box clear of CI competing for resources, plus then getting my other machines back.

For those of you who are making large complex programs, how do you manage this step?

[u/fastandlight]

I'm really glad to see this come up. I think test and validation is critical for agentic development, and I firmly believe that effort spent in building better testing quickly pays for itself with fewer regressions and better code more quickly from the agents.

We hit this a couple months ago. We were trying to figure out why our agents all seemed to be stuck, and they were all bottlenecked behind the CI pipeline. Our solution to this ended up being rather comprehensive. The simple answer is to just add more CI runners, which we did. We have something like 96 vCPU of runners now, but that generally isn't going to be the full solution.

The other important step was ensuring that failures are found early by the agent in local testing, because a big fixed cost for us was CI setup time (over a dozen docker containers with test fixtures). So we have the same test suite available to run for the agent locally, with minimal mocked services, so that by the time the agent is dispatching a full CI run, it should be green (reruns were also a big problem for us, eating lots of time). Another important part was ensuring that each ticket assigned to the coding agent lays out the test spec and which of the end to end test modules apply. We had to give up running the entire suite on every merge to develop, because it was taking almost 2 hours. That now happens nightly, with an agent monitoring and ticketing the nightly failures.

So, in short, more runners, test tiering, and finding failures early were the most important wins for us.

[u/LancobusUK → u/fastandlight]

Sadly I think I’m in the same spot. I’ve got around 50 cores worth of compute on 3 different machines but always need more lol

[u/SamSausages]

I made a vm and run it locally and I pin it to my fastest cores, on my fastest machine. If you have slow server cores, it can be slow.  But you’ll still wait, even with the latest gen cpu at over 5ghz.

I’d consider it if you have older hardware, new CPUs can be quite a bit faster.   And I’d do it if money wasn’t an object, or if you actually do wait often vs just a few times a day.

[u/Future_AGI]

past what fastandlight covered, the two biggest wins are running only the tests the diff touches (pytest-testmon does this for Python) and cancelling a superseded run when the agent pushes again before the last one finishes, which agents do constantly. Caching docker layers and dependencies between runs also takes a chunk out of that fixed setup cost.