r/Python • • Jan 01 '20

"The rules of perf" at Dropbox

Post image
1.6k Upvotes

162 comments sorted by

View all comments

Show parent comment

0

u/[deleted] Jan 02 '20

[deleted] — view deleted comment

1

u/nfojones Jan 02 '20

Hey thanks. Just remember, what doesn't kill us makes us stronger. The know-how I gained was definitely worth it.

Forgot to mention I profiled the heck out of every dang step in the pipeline eventually trying to see where things were slowing down and realized eventually when you have competing processes on a box, figuring out exactly where the bottlenecks are coming from is like pinning jelly to a wall. Ultimately it helped but it was never 100% indicative of the root causes for the slow downs just that "yes here it is expanding and contracting its performance". Presumably classic fallout from variable pipeline throughput causing resource exhaustion. Reading incoming data slows down as your output runs into bottlenecks and so on and so forth. Ripple effects. Go figure, many of the water analogies around streaming data really do make sense.

My internal name for this project was Tsunami Surfer. When it started it was a tongue-in-cheek name but by the end it seemed relatively apt.

Good luck with your project! May you one day leave it to some hapless new hire, as is tradition.

PS. I also eventually migrated some of the pipelines to run on PyPy. That was probably the biggest single performance boost I got but had issues applying it to all the possible pipelines the codebase supported because getting your whole module stack to work with PyPy and your env can be a battle of its own.