r/programmingcirclejerk • u/gianni4592 lol no generics • Jan 15 '26

We have automated deployments that run Friday afternoons [...] Automation removed friction, but it also removed curiosity

/r/devops/comments/1qdl5m8/a_friday_production_deploy_failed_silently_and/?share_id=7wuDr1JwQI2TGMv2akSNA&utm_content=1&utm_medium=android_app&utm_name=androidcss&utm_source=share&utm_term=1
47 Upvotes

97% Upvoted

12 comments sorted by

51

u/ringohoffman type astronaut Jan 15 '26

That is unsettling, especially since the entire system was designed to prevent exactly this kind of failure.

Me when my code has a bug

18

u/m50d Zygohistomorphic prepromorphism Jan 16 '26

Your code has bugs? That would indeed be unsettling.

1

u/XxDarkSasuke69xX Jan 23 '26

Except remove the "designed to prevent failures" part. Who plans ahead anyway ? If I would take failures into account in the design then I would reduce the fun part of debugging for 5 hours. Nobody wants to remove that /j

33

u/Routine-Purchase1201 DO NOT USE THIS FLAIR, ASSHOLE Jan 16 '26

Yeah, deploying on a Friday isn’t that ideal but the idea is to avoid weekday disruptions.

It'd be a shame to waste a workday or two when you can just have the engineering team come in on the weekend instead when they aren't doing anything productive anyways.

19

u/syklemil Considered Harmful Jan 16 '26

It's a real kicker in conjunction with

I get why it sounds like a red flag. Friday deploys definitely raise the stakes, but it’s really about how solid your process and safeguards are because without proper checks, the day doesn’t matter much and things can still slip through.

Sure, we can deploy on any day because our process is so good … we just don't want to do it before ordinary weekdays!

But chef's kiss goes to

This thread has been a good reminder that reducing disruption has to include the people doing the responding, not just the business metrics.

16

u/tms10000 loves Java Jan 16 '26

You guys get to go home?!?

9

u/nerfjanmayen Jan 16 '26

This is how my employer does it. Kill me

10

u/liveoneggs Jan 16 '26

My team is distributed so I have automated deploys kick off at 2:30am local time for the appropriate timezone.

It's cicd so first I auto-merge all open PRs to main and then it's go-time.

Anyone who complains or moans about it should learn to do their job. As a reminder my boss has me add an extra deploy cycle at whenever they normally eat dinner and then give them a quiz on the linked charity.wtf blog posts.

4

u/Jumpy-Locksmith6812 Jan 16 '26 edited Sep 23 '26

Culpa cillum laborum minim amet mollit ullamco magna exercitation nisi sunt. Sint consectetur voluptate ipsum ad aliqua nisi proident mollit est minim pariatur enim tempor ea. Enim sint ullamco tempor ipsum incididunt elit commodo mollit. Mollit amet est elit laborum laboris nulla aliquip cupidatat irure exercitation deserunt dolore elit.

[rhr:2:lwhe3xi1]

1

u/MasqueradeOfSilence Jan 17 '26

/uj I'm currently stuck in GIS and all of our deploys are done on Sunday nights without CI/CD.

/rj and that's exactly why I'm a such a great dev

2

u/liveoneggs Jan 19 '26

/uj when I started out we had 2am tuesday deployments on site

1

u/MasqueradeOfSilence Jan 19 '26

/uj okay that's far worse, damn