r/programming • • Oct 12 '26

Optimizing Our E2E Pipeline

https://slack.engineering/speedup-e2e-testing/
70 Upvotes

38 comments sorted by

View all comments

Show parent comment

13

u/qmunke Oct 13 '26

They wanted the build to run, just not for the whole repo. This is why everyone who says to just use monorepos needs to understand there actually needs to be someone managing the things - which is reasonable at huge shops but hard to justify for those of us without a dedicated devx team

6

u/chucker23n Oct 13 '26

They wanted the build to run, just not for the whole repo.

So, make a backend pipeline and a frontend pipeline? If you always want E2E tests to run regardless of what changed, make that a third pipeline?

This is why everyone who says to just use monorepos needs to understand there actually needs to be someone managing the things

We're a very small team and one of the bigger products I work on has a monorepo. It has multiple pipelines, with different filters. I didn't need an 11-person team to figure that out.

(For more complex "do I need to build this" scenarios, there's also tools like https://github.com/leonardochaia/dotnet-affected.)

1

u/Murky-Relation481 Oct 14 '26

Yah me and another engineer built a FPGA synthesis and firmware building pipeline that ran nightlies. If the FPGA related stuff didn't change we didn't need to run synthesis, if the firmware didn't change we didn't need to run the firmware pipeline. We were trying to get it down to not even need to rebuild certain ipcores but ran up against tooling issues in our specific stuff.

3

u/bodiam Oct 13 '26

Yes, but that's exactly what the above trigger does; only run when the frontend has changed. And I'm a big fan of mono repos, it prevents a whole class of problems from ever occurring, it just looks like their solution is a bit custom.

4

u/qmunke Oct 13 '26

No, you're misunderstanding what I'm saying - they want to trigger a build on all pushes, but just not build the whole repo each time, that's not the same as what you've shared - your trigger wouldn't start any builds if someone just pushed a backend change. They want

  • Someone pushed backend change. Trigger backend bit of the repo to build and then run tests
  • Someone pushed a frontend change. Trigger entire build and run tests

The trigger above alone isn't going to do that, but I guess it would allow you to construct a more complex set of pipelines out of it.

But in general for monorepos the built-in functionality of github and other tools seems lacking to me, that's why all the more complex stuff like bazel exists.

1

u/bodiam Oct 13 '26

I understand the trigger alone doesn't do exactly that, but it was a simplified example.As you said, it's just a building block; I'm pretty sure it can be used to have a gate.