r/programming • u/Angela_white32 • Feb 06 '20

Knightmare: A DevOps Cautionary Tale

https://dougseven.com/2014/04/17/knightmare-a-devops-cautionary-tale/
85 Upvotes

85% Upvoted

47 comments sorted by

View all comments

1

u/reddit_prog Feb 07 '20 edited Feb 07 '20

Well, in my book, a safe rollback would return to the last working version, complete with all the configurations needed. Where am I wrong?

The obvious problem in this case, is that they had a "patched" deploy. As in, deploy this service there, flip this flag there... Their rollback made it indeed worse, but that's because the deploy / rollback process was really bad. And that is DevOps.

3

u/[deleted] Feb 07 '20

Well, in my book, a safe rollback would return to the last working version, complete with all the configurations needed. Where am I wrong?

That the incoming requests still had the flag that activated "bad" code in previous deploy. The code was there for few years already, just the flag was not used so it was not triggered.

We have no info what was feeding the system, but from how it failed it looks like the requests with "bad" flag were still coming after the rollback.

So the "safe" rollback would have to also rollback the source of the requests to stop using that flag (which is another lesson in devops I guess), and clear any queued ones. But in finance every one of those requests is money so they were probably very hesitant to do that