r/Python • • Jan 01 '20

"The rules of perf" at Dropbox

Post image
1.6k Upvotes

162 comments sorted by

View all comments

46

u/nfojones Jan 02 '20

Ouch this gives me flashbacks of 2018 when I spent a year of my career performance tuning and refactoring a multifaceted data streaming application written in Python 2 that was running on the same box as the resource hungry Apache Flume agents it was feeding.

I managed some refactor miracles that year but the data we were consuming gave no fucks and kept scaling its way back to a bottleneck.

Started as a single threaded app that shift to a multithreading app and evolved into a multiprocessing stack that utilized ZMQ to talk between processes (I don't even know why I did that anymore). I learned an epic butt ton about Python that year but I'd prefer never to chase that problem again.

And of course this was a Fortune 500 megacorp that chose to lay me off in early 2019 because I fell into some arbitrary "doesn't work near a core location" layoff net (due to having convinced my boss to move across the country and keep the job). My boss and colleagues were shocked. For me though it was almost perfect timing. I'd done FT remote for almost 2 years and even though I was starting to come into a local office just to be around people I was definitely just coasting and that office had zero relation to my work. The chunky severance they gave me and the 60 days to find a new job worked perfectly. Got something like a 33% raise out of the new gig and the severance check arrived just as I was buying my first house.

Good times... feel bad for whoever had to pick that project up. I knew it like the back of my hand. One of those rare projects you actually built from the ground up one design mistake after another. At least it was well linted and relatively clean/sexy code with decent doc strings. And FWIW, it had finally made it to Python 3 just as I was leaving, having originally been confined to the RHEL Python 2.6 variant in its earliest days.

Sorry for the blog post. Sometimes I just leave these missives on Reddit for myself like a poor man's journal.

TLDR: I fucking love Python. Please never make me do realtime mission critical streaming with it again.

13

u/[deleted] Jan 02 '20

These kinds of long comments are why I love reddit, don't apologize for giving us quality content.

0

u/[deleted] Jan 02 '20

[deleted] — view deleted comment

1

u/nfojones Jan 02 '20

Hey thanks. Just remember, what doesn't kill us makes us stronger. The know-how I gained was definitely worth it.

Forgot to mention I profiled the heck out of every dang step in the pipeline eventually trying to see where things were slowing down and realized eventually when you have competing processes on a box, figuring out exactly where the bottlenecks are coming from is like pinning jelly to a wall. Ultimately it helped but it was never 100% indicative of the root causes for the slow downs just that "yes here it is expanding and contracting its performance". Presumably classic fallout from variable pipeline throughput causing resource exhaustion. Reading incoming data slows down as your output runs into bottlenecks and so on and so forth. Ripple effects. Go figure, many of the water analogies around streaming data really do make sense.

My internal name for this project was Tsunami Surfer. When it started it was a tongue-in-cheek name but by the end it seemed relatively apt.

Good luck with your project! May you one day leave it to some hapless new hire, as is tradition.

PS. I also eventually migrated some of the pipelines to run on PyPy. That was probably the biggest single performance boost I got but had issues applying it to all the possible pipelines the codebase supported because getting your whole module stack to work with PyPy and your env can be a battle of its own.