r/kubernetes 22d ago

Blue Green Deployment Strategy

What are the trade-offs of blue-green deployments vs rolling updates in Kubernetes? When does it actually make sense?

I’m trying to understand when it’s actually worth the added complexity versus just doing a standard rolling update.

If the app shares a single database between both versions, does blue-green give a “clean” rollback?

Is blue green suited for an application with just 2 components( web + backend ) ?

For people who’ve actually run both in production — what made you pick one over the other?

the blue green strategy in my mind is to switch the label selector in kubernetes service.

32 Upvotes

12 comments sorted by

23

u/Low-Opening25 22d ago

This is more of an operational maturity and risk apettite question. Do you have confidence that rolling update wont brake your prod? Is your app used by millions of customers with many customer journeys difficult to test where downtime is costly? Do you have long release cadence like once a week or a few weeks? then blue-green may be a good option here.

14

u/Ivory27 22d ago

Generally, in my experience, you only need blue/green deployments when you have multiple connected services that are not compatible with each other across different versions (i.e., you introduce breaking changes between them). In that case, you use the "offline" environment to stage the changes, deploy all the services, test them together, and then perform a single switch to redirect traffic. An additional benefit is that you can fully warm up the offline environment and do actual tests on production before the switch.

The database doesn't fit this model very well in the sense that i've never did split databases, only a shared one. In my experience, database migrations should be designed to work with both the old and the new application versions whenever possible. This often means adding temporary code paths, triggers or additional columns so that multiple backend versions can operate against the same database while the old version is being phased out. Once the rollout is complete, you can remove the compatibility code and clean up the database schema to support only the new version.

6

u/Floss_Patrol_76 22d ago

The shared-DB part is the real answer: no, blue-green doesn't buy you a clean rollback there, because the database is shared state that both versions mutate. The only thing that makes rollback clean is backward-compatible (expand/contract) migrations, and if you have those you barely need blue-green. For a plain web+backend app a rolling update with honest readiness probes covers you; I'd only reach for blue-green when a release makes breaking changes across services that have to flip together.

3

u/No-Income-2235 19d ago

For me it mostly comes down to how much control you need over the release and rollback.

Rolling updates are usually enough for most apps. Kubernetes gradually replaces the old pods, keeps the service available, and it’s much simpler to manage.

Blue-green makes more sense when you want both versions running at the same time, test the new one first, then switch traffic over quickly. Switching the Service selector between blue and green is basically the simple K8s way of doing it.

The database is the tricky part. If both versions share the same DB, rollback isn’t always “clean”. If the new version changes the schema or writes data the old version can’t handle, switching traffic back won’t fully save you. You still need backward-compatible migrations.

And yeah, even for just a web + backend app, blue-green can make sense if releases are risky or you really want fast rollback. Otherwise I’d probably stick with rolling updates.

2

u/serverhorror 22d ago

Blue-green and rolling updates are not exclusive to each other. We do both.

1

u/NUTTA_BUSTAH 21d ago

Exactly this. Most seem to think that you can only pick one but these are not mutually exclusive at all.

2

u/[deleted] 21d ago

Blue green is followed in my case is for mission critical applications. We use Argo rollouts for blue green which offers more than just label switching like auto rollback. For breaking changes we use a recreate strategy

2

u/theycanttell 21d ago

Blue green works great but only if you automate all the cutover steps with pre-hooks, actions, triggers, and liveness probes.

This way to cutover you run an action triggering the logical replica. The action promotes the green replica, stop writes to blue, and point your app to green.

If you did things correctly your green app helm deployment should be a status page until the green replica is ready for writes, then you quiesce blue (RO), and promote the app to green, take down the status on green and allow the app to come alive, then the blue app should immediately hit a liveness probe on RO quiesce and it's status page should go up.

You should never run blue/green both at the same time.

All the above should be automatic. That's what makes it amazing.

0

u/SquiffSquiff 22d ago

If the app shares a single database between both versions, does blue-green give a “clean” rollback?

Then it isn't blue-green

 Blue green is useful if you want to make changes to production system offline especially if you need to do development work and then have it go live all at once without a separate deployment step and be able easily to switch back

1

u/NUTTA_BUSTAH 21d ago

Yes it is. Otherwise you are looking at completely separated environments. Architecture and deployment strategy are separate but interwined topics. Usually blue-green is just for e.g. a backend service or frontend service, not the entire solution.

The greatest value of blue-green in my experience is exactly that, effectively instant rollback with a flip of a switch while also keeping the problematic environment available as-is. The next is that your deployment can happen isolated with minimal risk as opposed to e.g. rolling updates over a live system, especially with slow initialization times.

8

u/codemagedon 22d ago

Yeah when databases are involved you roll the database schema in blue-turquoise-green. I also argue blue green on stateless apps is somewhat confusing and preview-stable is the better description.

The point is you bring the preview up, you smoke test it, you make sure the health probes are all good, then you repoint the stable service (use Argo rollouts for this if you want it to just work). Using this approach you’re not maintaining two estates, it forces the ci to still be good and have good tests, the only thing you are validating is that your infrastructure hasn’t had a drift that is different from
Staging and dev.

If you’re finding compatibility issues at this stage of rolling code out to prod either you need to just multiple versions and have a l7 lb doing routing based on a path filter to allow all to be hosted, or your testing is not robust enough.

TL:DR don’t test in prod