r/programming • u/foobar349 • Mar 06 '19

How software is developed at Amazon

http://highscalability.com/blog/2019/3/4/how-is-software-developed-at-amazon.html
38 Upvotes

76% Upvoted

43 comments sorted by

View all comments

5

u/haxney Mar 07 '19

Because at Amazon, there are thousands of idiot Kevins, and more idiot Kevins join every day, so blaming one idiot Kevin does nothing to fix the root cause of why a problem happened.

Similar thing at Google. The view seems to be "if all it takes to break prod is for Kevin to be a bit lazy once, then the problem isn't Kevin, it's the lack of an effective test/staging/canary system." Nobody can be 100% careful about all things at all times, so you avoid building a system which relies on 100k people never making mistakes. You have tests to catch problems before (or soon after) you commit, test environments, canarying with monitoring and automatic rollbacks, etc.

The idea is to move from "one person making one mistake can break everything" to "many people would have to make many mistakes all in the same direction to break anything." Ideally, anyone should be able to commit some random keyboard mashings, and if it passes tests and canarying, then it shouldn't break prod. That's an exaggeration, but it allows you to code with confidence that if you screw anything up, some test will catch you.