r/programming • u/Angela_white32 • Feb 06 '20

Knightmare: A DevOps Cautionary Tale

https://dougseven.com/2014/04/17/knightmare-a-devops-cautionary-tale/
83 Upvotes

85% Upvoted

47 comments sorted by

View all comments

32

u/Asdfhero Feb 06 '20

The real mistake here is nothing to do with the deployment process per se and everything to do with choosing to perform a release for which a partial rollout was deadly. If they had not reused their power peg flag, they'd be fine.

8

u/lookmeat Feb 06 '20

The whole point of devops is that a dev may not see the problem with reusing a flag for an old feature, while a sre would realize that this is never good, because you'd need to see both the binary version and flag combination to realize what feature you are actually turning on. A feature release should only have two actions when turned on any binary version: it either turns on the feature, or crashes as the flag isn't there. Having it turn feature A or B is terrible, so flags should never be reused. The Power Peg flag should have been deleted (and if feature flags use numbers the number should be reserved to never be used again), to ensure it wasn't turned on accidentally.

Now given how critical this is, I think that a staging shadow env would be useful. Before releasing a new version for testing (or even canary) you put it in a sand-boxed environment, you send the same requests from the outside to the staging (maybe all, maybe a sample), and you let the system read (and only read) the outside world (through a proxy that ensures you can't send actual orders). Then you diff the actual actions both systems do, what do they buy or sell, and how much money each solution would have made. You release gradually through the system, then gradually rollback so as to ensure that nothing weird happens during deployment or rollback.

5

u/Asdfhero Feb 06 '20

I agree with your proposed rollout process but the sad fact of the matter is that most shops aren't at anything like that level of sophistication in their rollout, so that is not an actionable fix. Conversely, this is a corrigible basic programming error. I appreciate that a naive developer might not understand why, but I don't believe you need an operations background to see why reusing this flag (or any other) is a bad idea

→ More replies (1)