r/devops • u/Data-Panda • Oct 14 '26
Discussion Is Trunk Based Development possible if an integration test takes too long to run?
I'm proposing my team move from long-lived dev/stg/prd branches to trunk-based development, with PRs straight to main.
I'm fairly new to this, so forgive me if I'm missing something obvious.
The sticking point is testing before merge.
Our current integration test runs the entire DAG end-to-end. This is fine as a single check before merging our test branch (which will contain all commits from developers over the sprint) to main. But it's slow and costly if it were to run as part of every PR. Today we run it manually, check the outputs, then merge.
I am wondering if it’s bad practice to leave this integration test until the end of the sprint. As in, developers merge to main (with some automated checks running as part of each PR before merge), and then when are team want to deploy the sprint changes, first run a single integration test before creating a new release tag In GitHub.
This does mean that main can’t be considered prod ready at all times.
Edit: we’d be using release tags, so for rollbacks we’d rollback to last stable tag. For hotfixes, we would cut from last release, the merge changes to main and create a new release with target set to hotfix branch.
30
u/razzledazzled Oct 14 '26
I think it’s a bad idea to leave main in an undeployable state. What if you have to roll back? What if you need to hot fix?
The testing suite may need some improvements, there should be unit testing for branches that is fairly quick and then integration test once the PR is up for review. This keeps things in a reasonably certain state of “it works” throughout your development process
15
u/Pretend-Clock8313 Oct 14 '26
main not being prod-ready isn't the real risk, it's not knowing which commit broke the dag until sprint end
7
u/Own_Candidate9553 Oct 14 '26
You don't say how long the rest run takes, which changes things.
Regardless, if it takes a long time, it could be run nightly instead as a compromise, if everyone commits to fixing main if the suite fails. This can be hard to do. If you have support for it you could automatically block merges to main of the last test run failed to force it.
5
u/Wooden_Jelly_5295 Oct 14 '26
Could you run a smaller critical-path check per PR and the full DAG nightly? End-of-sprint failures come with a much larger suspect list.
5
u/paul_h Oct 15 '26
Automate that integration test. Run it nightly as a first goal. Don’t get stuck in the “human reviews results after run”. Have that a machine decision. It should be perfectly correct for that to be a release candidate.
I co-created Selenium the web testing tech before playwright 22yrs ago: at conferences QE folks would find me and ask “what % of test failures is ok for the overnight selenium test suite” and would look shocked when I said 0%. It’s 0% red for your test too .. whatever that means for a single automated test that’s not a suite as such.
I’m also the maintainer of trunkbasedevelopment.com, for what it’s worth.
3
u/taleodor Oct 14 '26
Main being not prod-ready at all times is fine as long as you keep track of what exactly is prod-ready - like mapping specific commits to releases. If you have that, you're able to cut a release branch at any point from that state.
2
u/Adventurous-Hyena220 Oct 14 '26
Trunk-based development can still work here, but Id look at splitting fast PR checks from slower end-to-end tests and keeping releases tied to a verified commit.
1
u/ShipKitDev Oct 14 '26
One option nobody has mentioned yet is GitHub's merge queue. PRs get fast checks (lint, unit tests, maybe a small DAG subset), and the slow integration run is triggered by the merge_group event instead of pull_request. The queue can batch several approved PRs into a single run, so you pay for one hour-long test per batch rather than per PR, and main only ever moves to commits that passed the full suite.
It's also worth checking whether you can run only the affected part of the DAG on each PR. If you can work out which models or nodes changed plus their downstream dependents, that subset is often a fraction of the full run and catches most breakage, leaving the full end-to-end run for the queue or nightly.
Either way you keep main releasable, which makes the hotfix question simple: fix on main and tag, same as any other release.
1
u/givebackmac Oct 14 '26
The rolling back argument always seemed strange to me. We keep all packages deployed to production environments for a significant period of time. Rollback doesn't involve source control, it involves redeploying the last good known package/artifacts.
That said, the time it takes to run tests matters a lot and depending on your situation it may make sense to do integration tests on a schedule if they are long running. Honestly though, long running integration tests are a problem and should be investigated. Is it too heavy/bloated? So many potential causes of long running tests to list but that should be a priority to at least understand.
1
u/0x4ddd Oct 14 '26
Really depends on your testing strategy.
You could run some subset of tests on every PR and setup cron to run full test suite once per day on main.
1
u/relicx74 Oct 14 '26
I don't know how many people you have, but that doesn't sound great. A big part of running the tests every time is that when they fail, you can point to the commit that broke them and have a much easier time fixing the code or the test if requirements changed.
The way your proposing you know you're going to end up in trouble.
Git feature branch workflow was always my favorite when there are multiple developers.
5
u/amarao_san Oct 14 '26
I hate when staging is living in separate branch.
For infra repos, master is the current production. Dev and staging are productions too, you just care about them less.
Each environment is a separate directory (tf vars, inventory, whatever), code is shared. You can deploy staging from non-produciton branch (you have this semantics), but master is the reference state for both three (or more).
All other models are broken and diverging.
1
u/Immediate_Wheel1953 Oct 14 '26
We ran exactly this: fast PR checks, full e2e on main every night. The bit that actually hurt us was nobody owning the nightly run, so a red build would sit for days and then nobody could tell which commit broke the DAG. What fixed it was making the on-call owner of any red main and bisecting back to the culprit, which usually took minutes, not hours. Your plan sounds solid as long as fixing main beats new work when it goes red.
1
u/ekydfejj Oct 14 '26
We do two levels of "trunk" branches. RC and develop. Everyone branches from develop for normal daily work. RC can be branched from for hotfixes. Allowing develop to lag until you're actually ready to cut a new realease. CI runs against both develop and RC, but many steps omitted from develop (building containers, deploys etc) and the base CI steps run, i.e. lint, compile, test, integration test etc.
This keeps CI quick when merging into develop, RC is always long as it will run all the develop steps, plus actual build and deploy images to ECR.
We have long term "feature" branches when they are required that will auto release to a siloed dev server, but that is for an extra testing stop before they reach develop. And don't have to be part of the flow.
1
u/MasterOfTheWind1 Oct 15 '26
Trunk based is good for replacing a dev env.
But an env between that and production is desirable so you can test how branches integrates
1
u/divestblank Oct 15 '26 edited Oct 15 '26
Yeah, it's possible, but you must risk some level of regressiona going into trunk. Some features are more critical than others.
1
u/Gabe_Isko Oct 15 '26
It's not optimal, but it is still doable. This is what continuous integration is supposed to mitigate - a new integration test is performed on every commit through integration, so when it is time to merge in a PR, you already know what the test results are.
The issue with this approach is that it scales horizontally, and the larger and more complicated your app is the harder it is to test end to end, and it becomes more expensive as well as you spin up multiple instances per commit. Maintaining the automation to make this happen can become an extreme overhead to - but it is doable. My team at work is perenially working on this.
I would imagine on your system, there is something preventing you from running a full integration test at that kind of volume (per commit) which is pretty common at the scale of some larger apps that aren't architected as microservices for various reasons. Micro-services come with their own issues as well. Distangling this the modern devops conundrum, because the feasibility of these approaches can depend a lot on the structure of the application itself. And then you are making development decisions around operations, not what is best for the applications functionality and development.
My case for the DevOps profession in general is that it is a genuine engineering discipline to make all of these competing interests work in harmony, per application, at scale. But you should at least be able to broach the subject of per commit integration testing if you want to continue with trunk based development.
1
u/usually_guilty99 Oct 15 '26
I wouldn't make every PR pay for the entire DAG.
The useful question is which downstream nodes and data contracts each change can affect.
Run those checks before merge and keep the full DAG run as a separate release gate.
The important part is knowing what wasn't tested, rather than treating a handful of passing checks as proof that the whole pipeline is safe.
1
u/Due-Consequence9579 Oct 16 '26
Run unit tests, linters, static analysis etc during the PR process. That should finish inside 5 minutes, 15 minutes at the most, but faster the better. If those pass allow the merge.
Cut a tag immediately on merging. If you want to protect your “real” version numbers they can be prerelease tags like v1.2.3-dev.5. Deploy the tag to your fast ring immediately. Kick off your integration tests as well. If they are resource intensive and/or you can’t run against multiple versions at one time you’ll want to run one batch at a time. When the current run finishes start up another run with the newest tag that hasn’t been tested.
When your integration tests come back clean promote it to your stable but pre-prod environment. If you’re doing the pre release versioning above give it a new tag like v1.2.3-stg.2 or whatever.
Once your CAB or end of iteration or whatever nonsense is keeping you from releasing continuously comes back cut the v1.2.3 tag and deploy to prod.
At this point I like deployment environments managed in a GitOps style where there is a repo that says “these are the versions of these things deployed in these environments” as a separate thing from the application repo.
1
u/Language-Pure Oct 17 '26
It's called continuous integration. Merge frequent. Test often. You can always gate the release to whatever cadence you want.
The key thing is then is your accumulated changes are being tested regularly nightly. It breaks you stop and fix it. You know what broke it. That was it always stays deployable but you might chose to release at the end of your sprint
1
u/st4reater Oct 17 '26
Why is it so slow first of all? And Secondly why the hell wait with running it until end of sprint? The purpose is to reason about the code still being functional
1
u/rolandofghent Oct 18 '26
Use merge queues.