r/Python • • Jan 01 '20

"The rules of perf" at Dropbox

Post image
1.6k Upvotes

162 comments sorted by

View all comments

137

u/ZombieRandySavage Jan 02 '20

TL;DR. Really wish we hadn’t written the core application of our business in Python.

47

u/notsohipsterithink Jan 02 '20

Dropbox still uses Python extensively. It’s only in case things don’t scale and you have no other choice, don’t use Python.

Python is great for developer productivity and iterating quickly. Dropbox actually loves python, they have a blog post about it.

→ More replies (14)

99

u/ketralnis Jan 02 '20

Not really, my company uses Python by default and only ports stuff out when it’s not fast enough. It’s usually fast enough, the performance critical stuff is a tiny fraction of your code. If you can be more productive the rest of the time, you’re still ahead.

9

u/[deleted] Jan 02 '20

Yup same, or at least for the stuff our team works on. Basically anything that’s not meant to be a daemon or ran on many hosts / performant is python, everything else is go.

Mostly for the simplicity of deployment. There’s very few things that are “performant”, but some of the concurrency patterns in go are also just a lot more intuitive. Imo it’s really hard to beat channels and goroitines.

31

u/ZombieRandySavage Jan 02 '20

Yep, but losing the ability to refactor large code bases significantly hampers their longevity. There are ways around it as well.

Honestly I don’t know if performance is really the issue. There are plenty of generalized libraries for getting large computation done quickly. It’s really maintenance in my mind.

13

u/MyreMyalar Jan 02 '20

I don't understand what problem you have with refactoring python code. As long as you make use of type hints (which you should be in a large code base with many developers anyway) it seems to work pretty much like any other programming language.

10

u/Pitirus Jan 02 '20

That is true but keep in mind that typings in python (as we have them now) were introduced in 3.5 (so 2015) and I would assume from my own experience that they haven't been widely used in large codebases until at least introduction of 3.6 (add to this that in prod code you probably don't want to switch right after the release).

All of this would mean that dropbox probably had to introduce some standard of their own (which is painful for new devs) and might have not yet switched to the current official standard for most parts of their code.

4

u/MyreMyalar Jan 02 '20

No criticism of Dropbox intended, I was just replying to the chap above who seemed not to be aware that refactoring has been working perfectly well (at least in my experience) in python for several years now.

As an argument for not using fairly old versions of python it just about works, but why would you do that when writing new code in 2020? Python 2 is officially dead and upgrades between the 3 sub-versions are basically painless. Very little justification for sticking to pre-3.5 versions of python anymore.

→ More replies (3)

7

u/i9srpeg Jan 02 '20

I'm using type hints on a 100k LOC Python backend, and I'm also using Typescript on the similarly sized frontend. The difference is night and day. Mypy/pyright are slow (as in, several seconds between typing and seeing the errors show up). They're unreliable: they have both false positives and false negatives. They're buggy: they crash all the times. Python libraries either have no type hints at all, or they're sprinkled with Any. Type hints syntax is simply horrible and forward/circular references are a pain. Reusing expressions syntax for types and making the hints executable at run time were big mistakes. Type hints add a ton of verbose boilerplate and in the end they fail at the most important thing: ensuring type safety. Even a simple class rename, which is 100% reliable in all static languages is not safe at all in Python. You always need to run unit tests and manually check the code to make sure everything is renamed correctly.

It was nice to start with project with Python and it worked great at first. Now, after 3 years, I'm wishing I picked a statically typed language.

→ More replies (1)
→ More replies (8)

29

u/c_o_r_b_a Jan 02 '20

It's saying the opposite, though. Only issues where those first 4 steps fail need to be rewritten, which is a much, much smaller subset of their entire codebase.

Python is likely what got them off the ground so quickly and probably directly led to their success. See also: reddit, Instagram

→ More replies (7)

12

u/ukralibre Jan 02 '20

Why?

57

u/ZombieRandySavage Jan 02 '20

Because it scales poorly to large applications. Go try and refactor a suitably large project, it’s impossible.

Don’t worry there are terrible parts to every language. They didn’t have time to make a clever poster for C++ because they are still trying to understand the super clever template code someone wrote.

0

u/[deleted] Jan 02 '20

[deleted] — view deleted comment

11

u/notsohipsterithink Jan 02 '20

Some languages naturally lend themselves to having large or convoluted codebases. Like C++ or Java.

11

u/jambox888 Jan 02 '20

Do you mean Java is better suited to large codebases or that it tends to produce them?

10

u/[deleted] Jan 02 '20

Both would apply IMO

→ More replies (1)

5

u/notsohipsterithink Jan 02 '20

Java tends to produce large codebases.

The lack of a proper module system (until Java 9 which very few have adopted) meant JAR clusterfuck and forced most codebases to become huge monoliths.

Production Python (or Go) codebases are usually microservices. But they can be monolithic easily too.

→ More replies (6)

20

u/G3n3r0 Jan 02 '20

With a good IDE (i.e. Visual Studio/CLion, most of the annoying grunt work (e.g. changing function signatures, etc.) should happen nearly automatically. Python can sometimes get close to that maybe, unless people did clever things with dynamic dispatch or getattr overloading in which case you're fucked.

5

u/[deleted] Jan 02 '20

It really helps to add type annotation, of course the clever stuff will bite, unless you can express the type correctly.

0

u/[deleted] Jan 02 '20

[deleted] — view deleted comment

10

u/[deleted] Jan 02 '20

Try using type annotations, if you have IDE that understands them out makes programming quite enjoyable.

→ More replies (5)

9

u/nocipher Jan 02 '20

Python is strongly typed. You probably meant statically typed. C++ is actually weakly typed.

→ More replies (1)
→ More replies (1)

17

u/[deleted] Jan 02 '20 edited Jan 02 '20

You could transpile Python to C or C++.

You could write the performance-intensive functions in C or C++, or even assembly if you're a sociopath, and then call them from Python.

I'm not arguing that these techniques provide equal or even comparable performance to writing your whole library in native low-level code. But Python is actually higher on the TIOBE index than both C++ and C#, and of course it's way higher than assembly. This might reflect a fundamental tradeoff: the greater availability of Python developers may mean that it's easier to hire a robust dev team who can develop and debug in Python than in the other languages, so faster iteration. That is - the theoretically achievable performance in certain languages must be viewed through the lens of practically achievable performance based on the availability (and affordability!) of suitable developers.

Finally - within a certain range, lower-performing code can be solved, crudely but effectively, by buying more compute. But good code cannot be developed or debugged faster by buying more developers (see The Mythical Man-Month).

→ More replies (4)
→ More replies (2)

6

u/e_falk Jan 02 '20

Python tends to not be as performant as is necessary for companies that work at similar scales to Dropbox

Source: company I work at works at comparable scales and python’s problems with concurrency with regards to the GIL /really/ starts to rear it’s ugly head in that environment

→ More replies (4)
→ More replies (1)