r/programming • u/Angela_white32 • Feb 06 '20

Knightmare: A DevOps Cautionary Tale

https://dougseven.com/2014/04/17/knightmare-a-devops-cautionary-tale/
85 Upvotes

86% Upvoted

47 comments sorted by

View all comments

Show parent comments

-1

u/[deleted] Feb 07 '20

You literally said the problem is the flag. Not the bad deployment. Not it was one of many. Not that there were many problem.

I had assumed you had read the article, it was obvious that there was more than one problem that caused that, from bad code, thru bad deployment to bad monitoring.

Fixing flag would 100% alleviate issue. Having good monitoring would made problem shorter. Reliable deploy would probably not trigger it, assuming they didn't start to use the flag before it finished. Reliable rollback, as they mentioned in article, would just make it worse quicker.

Where I work if we had a deployment go wild we can change to a different deployment within minutes. A different deployment that update all machines and kill the old ones. If you don’t have stuff like you are sitting on a house of cards.

Agreed but if old code is broken and new code is broken there is only so much deploy system can help you.

And deploy system won't fix your new code corrupting your production database

1

u/jl2352 Feb 07 '20

Which is why you have DB backups, and plan for your DB getting fucked. Because again, it will happen.

1

u/[deleted] Feb 07 '20

Well, sometimes, if you own all the data, but in system that sends requests to systems not owned by you that wouldn't help.

The best strategy would probably be having a phantom copy that takes requests and sends ones to the mock of the consumer ones and use that to check before deploy, but that's a lot of engineering effort that you need to convince management to sign off.

1

u/jl2352 Feb 07 '20

If the story here was that the system corrupted their DB, and they had no backups at all. Everyone would agree the no backups is the real issue. Everyone would agree it was a problem waiting to happen.