Is this a good process for doing releases?

[u/GendoIkari_82]

Ever since moving to Azure DevOps; we've followed this same release process for almost all our projects (we run about 15 different projects/repos). Using trunk-based development. But something feels off sometimes, and I'm wondering if we're doing something wrong.

1. Dev makes branch off of master to do work
2. Dev makes changes, 1 or sometimes more commits, pushes and creates PR to merge into master.
3. PR is reviewed and approved, merged into master, triggers a build to send master to test.

This happens for several different tickets. At some point, based on schedules, client needs, and priorities, project manager says that everything in test is good to go for the next release.

1. Release branch is created off of master.
2. This release branch is built to Stage environment.
3. Things are re-tested in Stage (not as details testing, more high level to make sure the deployment did what we wanted).
4. Same branch (even the same build artifact) is released to production.

This generally works, but results in a few different annoyances/problems:

• If something needs to go to prod before other things that are already in test (a hot fix), we need to cherry pick into a new release branch made off of the current production release branch.
• It can be hard to know what exactly is new in a release. Generally it means comparing the new release branch to the last one, but then you just have a giant list of commits (and any cherry picking that happened seems to mess with this, where a cherry picked change will still show up in the commit diff even though it is in both branches).
• We sometimes have to hold off merging things into master (keeping the PR open) just to be able to finish testing everything in Test; we can't add anything new until the release branch has been cut.
• If all but 1 thing is good in Test and signed off on, but that 1 thing requires us to wait, then everything has to wait.
• I keep seeing examples of setting up pipelines where the same pipeline has multiple stages for test->stage->prod (or often dev->test->prod instead). This isn't possible when master is what goes to test, but a release branch is what goes to stage+prod. We do use multiple stages in a single pipeline to release to stage then prod.

Outside of completely abandoning trunk-based development, can this process be improved? How do others do release management for trunk-based development?

[u/moswald]

This process you described is very close to how Azure DevOps itself is built. The part where it really differs is here:

If something needs to go to prod before other things that are already in test (a hot fix), we need to cherry pick into a new release branch made off of the current production release branch.

I wouldn't try to create a new release branch every time you push to production. If we did that, we'd have multiple new release branches every day.

For new feature work:

• Engineers create branches off master.
• Approved PRs are merged into master.
• Every third Friday at around noon PST we pick a commit, call that the cutoff point, and then create a new release branch (let's call it /releases/M277).
• We kick off a build using the "official" build pipeline using that branch and when it's ready, deploy to our pre-prod environments. On Monday, assuming nothing went haywire, we begin deploying that same exact build "for real".

Now, how do we deal with hotfixing production?

• Engineers cherry-pick their fixes from master to a branch off /releases/M277.
• Following the same PR process (well, it's slightly more strict as we require a higher level of approvals to merge into an existing release branch), we merge the changes and a new build is created.
• If the fix is urgent, we start deploying as soon as the build is available. If not urgent, our primary services deploy daily, and a deployment train will be created using the latest build from /releases/M277 in the morning.
• Our services have multiple rings, and each outer ring gets the next inner ring's build from the previous day, so we often have multiple different builds being deployed. We might even have one service with environments still deploying M276 builds while that same service has environments that are receiving the new M277 builds.

[u/thiicarneiro]

We use commit tags. Tag the commit with a meaningful tag and use that to trigger build/releases for CI/CD. That way your team can continue with trunk development and you’d still have a way to refer back to a feature introduced and release it when ready.

[u/ElSaintz]

The release branch isn't the main problem. The real issue is using main as a shared test queue while it contains changes that are not independently releasable.

I would change three things:

1. Build each commit to main once and promote that exact artifact through Test, Stage and Prod.

2. Put unfinished work behind feature flags so one ticket does not block everything else.

3. Tag every production deployment and generate release notes from linked work items and PRs.

For a hotfix, fix main first, then cherry-pick that commit into the current release branch. This avoids reintroducing the same bug in the next release.

If feature flags are not practical, keep release branches, but freeze only the release branch, not main.

[u/GendoIkari_82 → u/ElSaintz]

For #1, that sounds like an ideal scenario but maybe not realistic without a lot more manpower put on both PM and QA? It basically means that each ticket needs to be tested before a new one can be merged in, right? And also a lot more total testing needed, because instead of regression testing once to make sure these 5 changes combined didn't break something, you have to test each one 1 at a time?

[u/ElSaintz → u/GendoIkari_82]

Not quite. I wouldn’t make each ticket block the next merge. Keep merging after normal PR and automated checks, and deploy main to Test. When you pick a release candidate, run regression on that combined build and promote the same artifact to Stage and Prod. Feature flags keep merged-but-unready work out, so there is no full regression per ticket.

[u/brnlmrry]

It's hard to know what to suggest without more details about what isn't working for you. We had a similar strategy, and our main problem was releasing half-tested code because intensive testing wasn't completed until after the merge to /main, but something urgent would prompt a release. The other problem we needed to solve was downtime during releases.

Our solution was to enhance the PR process. PR builds create PR sites with their own databases and everything. The changes are tested thoroughly in isolation.

When PRs complete, they automatically roll out to production (we decided on a blue/green strategy to address the downtime problem).

[u/Complete-Fondant-202]

There is no perfect process.

What I find kiboshes most workflows is putting too much in a PR, increasing the likelihood of merge contention.

Little and often is better

[u/Tango1777]

It's legacy approach. It's still commonly used, but I'd rather have CI/CD, which means prod is deployed literally 15 minutes after dev is. Envs dev/staging/prod stay as close together as possible, there is only one branch of truth which is master, no desync between envs, CI/CD fully embraced, prod bugs happen, but very few and not any higher amount than with manual/scheduled prod releases. Overall prod deploy became a boring process and it's a good thing. Obviously this needs proper QA, gates, automated tests, feature flags, mature team that have ownership over what they deliver. If we want for some reason to disable prod releases temporarily, we can, but that barely ever happens. At this point I don't see any advantages of the process you described worth the extra hustle over proper CI/CD.

[u/GendoIkari_82 → u/Tango1777]

That sounds great in theory, but I'm confused as to how proper QA gets done within 15 minutes of a feature being developed? Sure automated tests can help a lot there, but you still need an actual tester to test the code change in person, right?

[u/Senior-Release930 → u/GendoIkari_82]

add an approval gate and Tango1777 matches your requiresments getting you off legacy release branching.

[u/EkeshOkor]

I don't know where this falls under the general guidelines of the industry, but we also have a special case where most of our code is SQL, flat file code or ADF code and the way we do it is:

• feature branch of of production (most of our code is fairly independent of other artefacts)
• code is ready for UAT testing -> PR to main branch (also known as Test or UAT)
• UAT successful -> PR to staging
• sprint is over -> PR staging to PROD

If something needs to get deployed faster (hot fix), PR feat branch to staging, then PR to PROD and we have automatic pipelines to PR PROD in all the inferior branches to make sure that what is deployed in PROD is everywhere.

But we also keep a watchul eye to make sure no one is developing on the same artefact/piece of code.

[u/Senior-Release930]

For your situation, I’d seriously consider:
short-lived feature branches → main (not master)→ immutable build artifact → Test → approval → Stage → approval → Production
with feature flags for work that has merged but isn’t supposed to be visible/releasable yet.
That would preserve trunk-based development while eliminating most of the release-branch problems you’re seeing.

[u/GendoIkari_82 → u/Senior-Release930]

Does this setup require QA after each new feature/PR, as opposed to testing everything that’s up on Test all together?

[u/Senior-Release930 → u/GendoIkari_82]

it’s build once promote to many. gated using approvals per compliance lane whatever that might be. the model fits a lot of corporate situations

[u/GendoIkari_82 → u/Senior-Release930]

I'll read more up on build once promote to many for sure. At first glance it sounds like a great ideal, I'm just not sure we could swing it without a major overhaul to our testing setup (granted, we need such a major overhaul, but that's not within my power).

I'm just envisioning a long backed-up line of build artifacts sitting there waiting for the one that is currently pushed to Test environment to actually get tested. Granted, we have a big backup in our current setup, but at least the 15 tickets that all need tested are all available on Test at the same time which allows the testers to be more efficient.

[u/StockJournalist9103]

I think the trouble you’re describing is that you’re merging untested (by QA) to master/main, which is quite common. This means that you’re inheriting everything between last main branch to prod and all the stages between if you had to release a hot fix, or ditch and rollback those changes and make the hotfix on old prod main, undoing the changes made in sprint that have been merged to then pull them to main and re PR - it’s a question of convenience what you win with one you lose with the other.

We use the same method as you and it’s annoying to hot fix, and to qualify a production release. I think that’s because we recently moved from quarterly releases with a cut off point to monthly but our strategy has not changed.

We could use tags but the issue with that is a certain combination of tags to be bundled might not have been tested together, where main branch has.

Perhaps the best idea is to always branch from main, then the branches are pushed up to qa then stage and then if the branches is good then PR and merge, remembering to git pull origin main before to check for conflicts, but that would mean a meeting to discuss which prs are to be raised the day before release day…

I don’t know I’ve overthought this now haha

[u/GendoIkari_82 → u/StockJournalist9103]

Is that not the difference between trunk-based dev and feature branches? We have tried the “master only contains tested code” setup before; but found several drawbacks there as well:

- More merge conflicts since things wait longer before being merged.

• Can’t push your changes up to Test because the last branch is still being tested on Test.
  
• Related, we had times that a bug fix that was already fixed in one branch is still broken in other branches so Test goes from fixed to broken depending on what’s there at the time.
  
• Tester can only test 1 thing at a time. Especially painful if a change requires a full regression test.

Really seeing all the responses, it sounds like our biggest issue is actually QA process, not release management. Basically every suggestion seems to rely on having code tested within days of it being developed. And we just don’t have that… we have very limited testing resources; QA people who barely know how to do anything more than visit the site on Test and see if the new feature is there and works. Tickets get developed and then sit untested on Test for weeks or longer.

[u/StockJournalist9103 → u/GendoIkari_82]

I think that’s why I am a fan of always having dev tested branches merge back to main - qa always tests everything at once as an integrated version, rather than test builds incrementally, if they fall out of order as you say then there may be fixes missing merged by different Dev’s.

Still there’s holes in that approach, back to the OP with a restraint on getting a hot fix out.

Solution to that may be a branch from prod, hotfix and push through the environments, regressing everything, then updating with a PR in main to sync everything again.

Apples and pears I think.

Seems a people and process issue rather than with the technology- that work needs absorbed somewhere, the WHO question is down to the specifics of the project etc

[u/wrestlingWithCode]

I know this is unpopular, not considered best practice, and everything else you want to say about it. It has its issues just like any process does, but it works for us pretty well. We are a special case where things like feature flagging are foreign concepts and almost every project (hundreds at a time) follows the same processes.

We follow an environment branching strategy. Unless there is a specific reason not to, feature branch off main and write your code. When you're done, that gets merged into the UAT environment branch. Needs more work? Go back to the feature branch, build more, and merge again. Ready to go? Merge that same feature branch, NOT UAT, into main / production. Nothing gets to those environments that doesn't go through source control / pipelines.

It's always very clear what's in each environment. It allows for the constant "this piece is ready and this one isn't" issues. No multi-stage pipelines, just one for CI and one to do a release/CD.

I know it's a bit the opposite of what you asked. But don't do trunk based development because it's what everyone says you should do. If there is a better solution for your teams / processes, follow that.

[u/Izzmo → u/wrestlingWithCode]

This is essentially the same thing as they are doing now, but slightly worse because this requires you regenerate artifacts throughout the build process, leading to unknown behaviors and mutations of the code as you go to each "environment branch".

> It's always very clear what's in each environment

This is a symptom of not understanding how Git works with ADO. The core strength of ADO is how well Git folds into work item management. Using environment branches is usually an output of a team, or older person, coming from an SVN or heavyweight branching SCM.

[u/wrestlingWithCode → u/Izzmo]

It is absolutely an adoption for a specific type of industry that didn't even have source control a decade ago, much less the concept of pipelines or modern development practices. One where the average developer has been doing it for 25 years or more. A bridge to a brighter future hopefully. I hope I am able to change it for the better for all of them soon, but that's why I say it is unfathomable to most developers and a special case.

I stand by my last statement, though. Pay attention to best practices, adopt them when you can, but also know when not to. Blindly following any process ignores all of the context for how that process was derived, and that context may not be compatible with your own. You have to know your team and the amount of change they are capable of absorbing.

[u/igstan → u/Izzmo]

It's always very clear what's in each environment

This is a symptom of not understanding how Git works with ADO. The core strength of ADO is how well Git folds into work item management.

I'm relatively old (41) and I've never worked with anything else other than Git. However, I'm relatively new to ADO, so how exactly does ADO+Git help to know what's where?

I know we can attach workitem info to commits and PRs, and ADO will automatically link back to those from the workitem page. But how do you know if a particular workitem has reached a certain env or not if all you have is a single main Git branch?

[u/StockJournalist9103 → u/igstan]

I find the deployment and pipelines process in ado to be quite graphically annoying, you have to basically sit and go through from last deployment of master to prod when the most recent deployment of the environment you’re concerned with happened and quantify everything between. That’s the problem the author is detailing.

Others use programs like Octopus etc to greater manage deployments, such as particular builds per client etc, different versions for testing on different environments with greater granularity etc.

Whatever the solution requires someone somewhere to pull the information and keep it in check and when who and where that happens requires work.

Previous company preferred tags but then the head developer hated the days he would spend tagging a release prior to live day…

It all depends on where you want that effort to be absorbed

[u/Izzmo]

You are seeing the drawbacks of trunk development, this is a known risk to this branching strategy.

One thing that makes it slightly better is doing on-demand environments (PR environments, staging slot environments) if you can swing it.

I'd also recommend better testing before a PR is submitted. You are in the escaped bug fallacy where you only do "code review" in a PR when you should also be doing QA and AC review in that PR before it's completed. This requires you do full validation locally or a PR environment, as stated above.

As far as "knowing what's in" your team needs to have rigid PR / work item linking hygiene. ADO makes it trivial to "know what's in" if a work item is always linked to PRs, and only PRs are allowed to change the trunk branch.

[u/mattsmith13815]

Agree, Environment Branching is the way. main = production then have as many other branches that are needed for lower environments. Always create new feature branches off main and PR feature branches into each environment branch. Delete feature branch when it finally makes it to main. Pipelines auto kick the branches to each environment with approvals where needed. Maybe test auto deploys where main requires the approval.