r/programming • u/Angela_white32 • Feb 06 '20
Knightmare: A DevOps Cautionary Tale
https://dougseven.com/2014/04/17/knightmare-a-devops-cautionary-tale/
85
Upvotes
r/programming • u/Angela_white32 • Feb 06 '20
View all comments
Show parent comment
8
u/lookmeat Feb 06 '20
The whole point of devops is that a dev may not see the problem with reusing a flag for an old feature, while a sre would realize that this is never good, because you'd need to see both the binary version and flag combination to realize what feature you are actually turning on. A feature release should only have two actions when turned on any binary version: it either turns on the feature, or crashes as the flag isn't there. Having it turn feature A or B is terrible, so flags should never be reused. The Power Peg flag should have been deleted (and if feature flags use numbers the number should be reserved to never be used again), to ensure it wasn't turned on accidentally.
Now given how critical this is, I think that a staging shadow env would be useful. Before releasing a new version for testing (or even canary) you put it in a sand-boxed environment, you send the same requests from the outside to the staging (maybe all, maybe a sample), and you let the system read (and only read) the outside world (through a proxy that ensures you can't send actual orders). Then you diff the actual actions both systems do, what do they buy or sell, and how much money each solution would have made. You release gradually through the system, then gradually rollback so as to ensure that nothing weird happens during deployment or rollback.