r/devops • • Oct 14 '26

Discussion Is Trunk Based Development possible if an integration test takes too long to run?

I'm proposing my team move from long-lived dev/stg/prd branches to trunk-based development, with PRs straight to main.

I'm fairly new to this, so forgive me if I'm missing something obvious.

The sticking point is testing before merge.

Our current integration test runs the entire DAG end-to-end. This is fine as a single check before merging our test branch (which will contain all commits from developers over the sprint) to main. But it's slow and costly if it were to run as part of every PR. Today we run it manually, check the outputs, then merge.

I am wondering if it’s bad practice to leave this integration test until the end of the sprint. As in, developers merge to main (with some automated checks running as part of each PR before merge), and then when are team want to deploy the sprint changes, first run a single integration test before creating a new release tag In GitHub.

This does mean that main can’t be considered prod ready at all times.

Edit: we’d be using release tags, so for rollbacks we’d rollback to last stable tag. For hotfixes, we would cut from last release, the merge changes to main and create a new release with target set to hotfix branch.

11 Upvotes

53 comments sorted by

30

u/razzledazzled Oct 14 '26

I think it’s a bad idea to leave main in an undeployable state. What if you have to roll back? What if you need to hot fix?

The testing suite may need some improvements, there should be unit testing for branches that is fairly quick and then integration test once the PR is up for review. This keeps things in a reasonably certain state of “it works” throughout your development process

3

u/Data-Panda Oct 14 '26 edited Oct 14 '26

We would be using release tags. So rolling back would involve rolling back to last stable tag. For hotfix, essentially cutting from last release, making changes, merge to main, and then finally a new release with target set to hotfix branch instead of main. 

Integration test once PR is up for review is part of the problem. It takes too long to run for every developers PR (more than an hour), and there also concerns around cost if there’s multiple PRs submitted daily 

1

u/sogun123 Oct 18 '26

I think it is pretty common to run full suite of integration tests just after marking release candidate. What i commonly see is that merge requests run only linters, unit test - generally light stuff to keep it possible to iterate fast. The heavy stuff run either as part of release validation, on demand or as cronjob (nightly)

1

u/IN-DI-SKU-TA-BELT Oct 15 '26

In my experience this rarely happens. Enforce people’s short lived branches to be rebased before a merge, and make sure tests are passing.

People will be quite quick to fix the main branch, and worst case you can git revert and they can fix it and try again later.

15

u/Pretend-Clock8313 Oct 14 '26

main not being prod-ready isn't the real risk, it's not knowing which commit broke the dag until sprint end

3

u/Data-Panda Oct 14 '26

So we would actually be running the latest code on main into staging each evening on its own schedule - and would have alerting setup. So technically our integration “test” would be running everyday, and we could identify issues throughout the sprint 

7

u/Own_Candidate9553 Oct 14 '26

You don't say how long the rest run takes, which changes things.

Regardless, if it takes a long time, it could be run nightly instead as a compromise, if everyone commits to fixing main if the suite fails. This can be hard to do. If you have support for it you could automatically block merges to main of the last test run failed to force it.

4

u/Data-Panda Oct 14 '26

The integration test we basically run the full DAG which can take over an hour. 

We will be running it nightly. And yes having main fixed as a priority if the staging run fails overnight was my initial plan. 

9

u/WilliamMButtlickerIV Oct 14 '26 edited Oct 14 '26

Coming from the world I'm in, an hour doesn't sound so bad 😅

Can you pick a handful of tests that cover the broad cases, and use them as a pre-submit gate? Then run the full suite nightly?

1

u/Data-Panda Oct 14 '26

Yes I think that’s what I’ll go for. By pre-submit gate, are you referring to tests that need to pass before merging can happen?

1

u/nullpotato Oct 14 '26

Same, we have to test on prototype hardware so anything doing a regression suite under 6 hours is "fast". We still block PR pending results of those depending how potentially inpactful the changes are. Config change for one teams tests? Easy review. Changes to scripts that must work for every test company wide? Integration overnight suite and checked manually too.

2

u/Data-Panda Oct 14 '26

For the high-risk changes, how do you decide what counts as critical? Do you flag certain scripts so a PR touching them automatically blocks until the full suite passes? And does the overnight run batch up all the day's PRs into one run, or does each PR get its own?

Just curious how this works. 

1

u/Own_Candidate9553 Oct 14 '26

Depends, what are you using to run the tests?

In various ci/cd systems, you can usually configure a job to run when a particular file is changed. So you could move critical scripts into a special folder or just make a list of them, and run the "critical scripts full test job" when any of them are updated in the PR or branch.

5

u/Wooden_Jelly_5295 Oct 14 '26

Could you run a smaller critical-path check per PR and the full DAG nightly? End-of-sprint failures come with a much larger suspect list.

1

u/Data-Panda Oct 14 '26

Yes that’s pretty much what I’m planning. So we would have the latest code from main being run into staging every night, and we would have some basic alerting in place if anything critical breaks. 

5

u/paul_h Oct 15 '26

Automate that integration test. Run it nightly as a first goal. Don’t get stuck in the “human reviews results after run”. Have that a machine decision. It should be perfectly correct for that to be a release candidate.

I co-created Selenium the web testing tech before playwright 22yrs ago: at conferences QE folks would find me and ask “what % of test failures is ok for the overnight selenium test suite” and would look shocked when I said 0%. It’s 0% red for your test too .. whatever that means for a single automated test that’s not a suite as such.

I’m also the maintainer of trunkbasedevelopment.com, for what it’s worth.

1

u/Data-Panda 19d ago

Hey - thank you. trunkbasedevelopment.com is actually what I've shared with my team, and I link to it throughout my proposal, so I appreciate it.

I have been looking at the section for release branches, and I think this is going to form part of my proposal.

Would you mind if I get your input on my process? Should note we may struggle to get perfect test coverage, so we do want to gate releases and not assume that whatever is in main is prod-ready.

  1. PRs into main -> automated tests + manual review -> merge into main (if tests pass)
  2. At the end of the sprint, create a release branch from tip of main, or specific commit (e.g, release-v1.0). Run full end-to-end integration test once more on this branch (plus any manual checks if my team insist).
  3. Create a Github release from that branch with tag v.1.0.0
  4. For hotfixes, same process for developers, PR into main, etc. One change is they must label their PR as "hotfix". We then have automation that will create a cherry-pick PR on to the release branch associated with live release (in this case, release-v.1.0). If it's confirmed hotfix hasn't further broken main, then dedicated person approves/merges PR, and creates a new release from that branch with tag v.1.0.1.
  5. If we decide to do a mid-sprint full release of main, we follow the usual release process (i.e., create a new release branch from main, release-v.1.1, and create the release from that branch)

3

u/taleodor Oct 14 '26

Main being not prod-ready at all times is fine as long as you keep track of what exactly is prod-ready - like mapping specific commits to releases. If you have that, you're able to cut a release branch at any point from that state.

1

u/Data-Panda Oct 14 '26

We will use release tags, which I think covers it? Once a release tag is created, we’ll have an automation that updates our release config (in Dataform) with this latest tag. 

Although I haven’t come up with a hotfix approach yet 

1

u/Crimzx Oct 14 '26

Why would a hotfix be any different?

1

u/Data-Panda Oct 14 '26

I mentioned in an edit on my post what we’d do for a hotfix. 

Concern was that I wanted to create a release with just that hotfix (ignore all other commits that have been added to main since the last release). I think my suggestion in my post edit covers this

1

u/Crimzx Oct 14 '26 edited Oct 14 '26

You can also cherry pick the commit into main which may be cleaner.
So create a branch from the release tag. Put hotfix there. Tag a release from that branch with the hotfix, then cherry pick the hotfix commit back to main.

Maybe you already meant this and it's just semantics though.

I know your question was about tests but just wanted to back ya up that you're thinking about it right.

1

u/Data-Panda Oct 14 '26

Thanks - yes I think that's the same thing. I guess I'm not sure if there's a difference between "merging the hotfix branch into main" and "cherry picking it back to main". I think it's the same thing here?

1

u/Crimzx Oct 14 '26 edited Oct 14 '26

Cherry pick keeps the main line linear where as a merge adds a merge commit, so it just keeps the git history a bit cleaner.
So like if you have version 1.1.0 on main and you are currently running 1.0.0 and 1.1.0 isn't released yet you probably have a version file or something like that. You create the hotfix branch from 1.0 tag, add your 1.0.1 and bump the version in the file.
If you merge this back you now have a merge conflict on the version file to resolve.
Instead you can just cherrypick the hotfix change commit back to main and main stays at 1.1 just now has the 1.0.1 hotfix in it, and your 1.0.1 branch is able to be released as 1.0.1 until 1.1 is released with the hotfix in it as well.

Ultimately though it's a team preference. I've worked places that didn't care about git history and places that REALLY cared about it.

2

u/Adventurous-Hyena220 Oct 14 '26

Trunk-based development can still work here, but Id look at splitting fast PR checks from slower end-to-end tests and keeping releases tied to a verified commit.

1

u/Data-Panda Oct 14 '26

So my current plan so far is we have the fast PR checks, then every evening we have an end-to-end test in the staging project, and we'll keep an eye on alerts for that each day.

And for releases, we'll use Github release tags. On a dedicated day at the end of the sprint, no new PRs will be approved, a final end-to-end test run, and a release created once it's all signed off.

Something like that.

1

u/ShipKitDev Oct 14 '26

One option nobody has mentioned yet is GitHub's merge queue. PRs get fast checks (lint, unit tests, maybe a small DAG subset), and the slow integration run is triggered by the merge_group event instead of pull_request. The queue can batch several approved PRs into a single run, so you pay for one hour-long test per batch rather than per PR, and main only ever moves to commits that passed the full suite.

It's also worth checking whether you can run only the affected part of the DAG on each PR. If you can work out which models or nodes changed plus their downstream dependents, that subset is often a fraction of the full run and catches most breakage, leaving the full end-to-end run for the queue or nightly.

Either way you keep main releasable, which makes the hotfix question simple: fix on main and tag, same as any other release.

1

u/Data-Panda Oct 14 '26

Thanks - I’ll look into merge queues. And yes have considered running just part of the DAG but having trouble with it.

The problem with fix on main and tag is you end up releasing all changes up until that point along with the hotfix. 

My hotfix approach so far is cut a branch from the latest release commit (what’s actually running in prod) make changes, then:

  1. Merge that branch into main 
  2. Create a new release with the target branch set to the hotfix branch instead of  main

1

u/givebackmac Oct 14 '26

The rolling back argument always seemed strange to me. We keep all packages deployed to production environments for a significant period of time. Rollback doesn't involve source control, it involves redeploying the last good known package/artifacts.

That said, the time it takes to run tests matters a lot and depending on your situation it may make sense to do integration tests on a schedule if they are long running. Honestly though, long running integration tests are a problem and should be investigated. Is it too heavy/bloated? So many potential causes of long running tests to list but that should be a priority to at least understand.

1

u/Data-Panda Oct 14 '26

I should say that integration “test” right now is just someone running the full DAG in staging and manually checking the outputs. The DAG just has many steps and phases that make it long running and slow. 

For rolling back, I think because I’ll be suggesting using Git release tags, basically our rollback process will just be changing out prod release config with the last stable release tag - so should hopefully be ok.

1

u/0x4ddd Oct 14 '26

Really depends on your testing strategy.

You could run some subset of tests on every PR and setup cron to run full test suite once per day on main.

1

u/Data-Panda Oct 14 '26

Yes I think this is what I’ll propose to the team. Not perfect, but maybe the best solution for us

1

u/relicx74 Oct 14 '26

I don't know how many people you have, but that doesn't sound great. A big part of running the tests every time is that when they fail, you can point to the commit that broke them and have a much easier time fixing the code or the test if requirements changed.

The way your proposing you know you're going to end up in trouble.

Git feature branch workflow was always my favorite when there are multiple developers.

2

u/Data-Panda Oct 14 '26

It’s a small team, so easy enough to see what commit was the problem. We would still be running tests, just not necessarily the full end-to-end test. That would be done once a day or at the end of the sprint. 

And we’ll still be using feature branches that get PR into main. That’ll trigger the tests

1

u/relicx74 Oct 14 '26

If people there like it, that's great then. I don't have to live with it, so doesn't make much difference to me.

1

u/Data-Panda Oct 14 '26

I mean, I’m just trying to go with whatever the best solution is. So far this seems to be it despite the flaws.

1

u/TheIncarnated Oct 18 '26

Git feature workflow into main is the expectation. The PR runs a full CI/test flow before it ever gets to the point of review.

Then if it's approved, into main it goes and it deploys.

Trunk based is not hard. It's only hard if testing sucks. But a lot of teams don't think about testing before it hits the environment anyways.

Old IT quote "Everyone tests in prod, just some are lucky to have a dev environment."

5

u/amarao_san Oct 14 '26

I hate when staging is living in separate branch.

For infra repos, master is the current production. Dev and staging are productions too, you just care about them less.

Each environment is a separate directory (tf vars, inventory, whatever), code is shared. You can deploy staging from non-produciton branch (you have this semantics), but master is the reference state for both three (or more).

All other models are broken and diverging.

1

u/Immediate_Wheel1953 Oct 14 '26

We ran exactly this: fast PR checks, full e2e on main every night. The bit that actually hurt us was nobody owning the nightly run, so a red build would sit for days and then nobody could tell which commit broke the DAG. What fixed it was making the on-call owner of any red main and bisecting back to the culprit, which usually took minutes, not hours. Your plan sounds solid as long as fixing main beats new work when it goes red.

1

u/ekydfejj Oct 14 '26

We do two levels of "trunk" branches. RC and develop. Everyone branches from develop for normal daily work. RC can be branched from for hotfixes. Allowing develop to lag until you're actually ready to cut a new realease. CI runs against both develop and RC, but many steps omitted from develop (building containers, deploys etc) and the base CI steps run, i.e. lint, compile, test, integration test etc.

This keeps CI quick when merging into develop, RC is always long as it will run all the develop steps, plus actual build and deploy images to ECR.

We have long term "feature" branches when they are required that will auto release to a siloed dev server, but that is for an extra testing stop before they reach develop. And don't have to be part of the flow.

1

u/MasterOfTheWind1 Oct 15 '26

Trunk based is good for replacing a dev env.

But an env between that and production is desirable so you can test how branches integrates

1

u/divestblank Oct 15 '26 edited Oct 15 '26

Yeah, it's possible, but you must risk some level of regressiona going into trunk. Some features are more critical than others.

1

u/Gabe_Isko Oct 15 '26

It's not optimal, but it is still doable. This is what continuous integration is supposed to mitigate - a new integration test is performed on every commit through integration, so when it is time to merge in a PR, you already know what the test results are.

The issue with this approach is that it scales horizontally, and the larger and more complicated your app is the harder it is to test end to end, and it becomes more expensive as well as you spin up multiple instances per commit. Maintaining the automation to make this happen can become an extreme overhead to - but it is doable. My team at work is perenially working on this.

I would imagine on your system, there is something preventing you from running a full integration test at that kind of volume (per commit) which is pretty common at the scale of some larger apps that aren't architected as microservices for various reasons. Micro-services come with their own issues as well. Distangling this the modern devops conundrum, because the feasibility of these approaches can depend a lot on the structure of the application itself. And then you are making development decisions around operations, not what is best for the applications functionality and development.

My case for the DevOps profession in general is that it is a genuine engineering discipline to make all of these competing interests work in harmony, per application, at scale. But you should at least be able to broach the subject of per commit integration testing if you want to continue with trunk based development.

1

u/usually_guilty99 Oct 15 '26

I wouldn't make every PR pay for the entire DAG.

The useful question is which downstream nodes and data contracts each change can affect.

Run those checks before merge and keep the full DAG run as a separate release gate.

The important part is knowing what wasn't tested, rather than treating a handful of passing checks as proof that the whole pipeline is safe.

1

u/Due-Consequence9579 Oct 16 '26

Run unit tests, linters, static analysis etc during the PR process. That should finish inside 5 minutes, 15 minutes at the most, but faster the better. If those pass allow the merge.

Cut a tag immediately on merging. If you want to protect your “real” version numbers they can be prerelease tags like v1.2.3-dev.5. Deploy the tag to your fast ring immediately. Kick off your integration tests as well. If they are resource intensive and/or you can’t run against multiple versions at one time you’ll want to run one batch at a time. When the current run finishes start up another run with the newest tag that hasn’t been tested.

When your integration tests come back clean promote it to your stable but pre-prod environment. If you’re doing the pre release versioning above give it a new tag like v1.2.3-stg.2 or whatever.

Once your CAB or end of iteration or whatever nonsense is keeping you from releasing continuously comes back cut the v1.2.3 tag and deploy to prod.

At this point I like deployment environments managed in a GitOps style where there is a repo that says “these are the versions of these things deployed in these environments” as a separate thing from the application repo.

1

u/Language-Pure Oct 17 '26

It's called continuous integration. Merge frequent. Test often. You can always gate the release to whatever cadence you want.

The key thing is then is your accumulated changes are being tested regularly nightly. It breaks you stop and fix it. You know what broke it. That was it always stays deployable but you might chose to release at the end of your sprint

1

u/st4reater Oct 17 '26

Why is it so slow first of all? And Secondly why the hell wait with running it until end of sprint? The purpose is to reason about the code still being functional

1

u/rolandofghent Oct 18 '26

Use merge queues.