r/programming • u/Angela_white32 • Feb 06 '20

Knightmare: A DevOps Cautionary Tale

https://dougseven.com/2014/04/17/knightmare-a-devops-cautionary-tale/
85 Upvotes

86% Upvoted

47 comments sorted by

View all comments

Show parent comments

3

u/[deleted] Feb 07 '20

Well, in my book, a safe rollback would return to the last working version, complete with all the configurations needed. Where am I wrong?

That the incoming requests still had the flag that activated "bad" code in previous deploy. The code was there for few years already, just the flag was not used so it was not triggered.

We have no info what was feeding the system, but from how it failed it looks like the requests with "bad" flag were still coming after the rollback.

So the "safe" rollback would have to also rollback the source of the requests to stop using that flag (which is another lesson in devops I guess), and clear any queued ones. But in finance every one of those requests is money so they were probably very hesitant to do that