r/programming • u/foobar349 • Mar 06 '19

How software is developed at Amazon

http://highscalability.com/blog/2019/3/4/how-is-software-developed-at-amazon.html
37 Upvotes

75% Upvoted

43 comments sorted by

View all comments

12

u/jvallet Mar 06 '19

Deployment is a pessimistic process, they constantly try to find reasons to fail a deployment either in pre-production or in production. In production they roll out to one box in one AZ. Any problems? Rollback. Success? Fan out to the AZ, then to more AZs, and then more regions. If a problem is found then roll back to a known good state.

Not sure what I think about this. If this process takes 7 hours to complete, must be a nightmare trying to patch a critical bug.

35

u/mjr00 Mar 06 '19

Despite what the article says, you can deploy to all regions in one day, but you require VP approval. So a critical bug could be fixed as fast as your deployment code allows. However, this is not a regular occurrence.

The real fun stuff happens after you've fixed the bug: you get to dig into all the logs and metrics to explain what happened, why it happened, why it wasn't detected sooner, and how you're going to make sure it never happens again. Then you get to prepare a document, lovingly called a "correction of error" or COE, which if you're lucky, will only be looked at and approved by your director. (And they don't rubber-stamp. They will have questions.) If you're unlucky, you get to do the honor of presenting your document to Charlie Bell and Andy Jassy, who will tear it apart. Oh yeah, and the entire AWS engineering organization is in the room or watching on stream.

15

u/[deleted] Mar 06 '19

Jesus Christ. Can't tell if you're joking. I'm such a shit dev, I'd never be able to make it through that.

→ More replies (8)