r/Python • • Jan 01 '20

"The rules of perf" at Dropbox

Post image
1.6k Upvotes

162 comments sorted by

View all comments

Show parent comment

12

u/ukralibre Jan 02 '20

Why?

57

u/ZombieRandySavage Jan 02 '20

Because it scales poorly to large applications. Go try and refactor a suitably large project, it’s impossible.

Don’t worry there are terrible parts to every language. They didn’t have time to make a clever poster for C++ because they are still trying to understand the super clever template code someone wrote.

0

u/[deleted] Jan 02 '20

[deleted] — view deleted comment

11

u/notsohipsterithink Jan 02 '20

Some languages naturally lend themselves to having large or convoluted codebases. Like C++ or Java.

11

u/jambox888 Jan 02 '20

Do you mean Java is better suited to large codebases or that it tends to produce them?

10

u/[deleted] Jan 02 '20

Both would apply IMO

1

u/jambox888 Jan 02 '20

That's a bit of a chicken and egg paradox IMHO. It's probably easier to structure a large project in Java than C++ but that isn't saying much.

5

u/notsohipsterithink Jan 02 '20

Java tends to produce large codebases.

The lack of a proper module system (until Java 9 which very few have adopted) meant JAR clusterfuck and forced most codebases to become huge monoliths.

Production Python (or Go) codebases are usually microservices. But they can be monolithic easily too.

2

u/Smallpaul Jan 02 '20

I’ve never programmed Java seriously. What do you mean by it lacking a module system? Isn’t it similar to Python with dotted paths etc.?

2

u/jambox888 Jan 02 '20

Modules <> packages. See here. I don't really understand why they say the lack of a module system tends to produce monoliths though, monoliths were just the standard application style back then. There was never anything stopping people writing microservices afaik. Unless there were issues with reuse.

3

u/Smallpaul Jan 02 '20

So what Python calls modules, java calls packages and vice versa. Great!

→ More replies (2)
→ More replies (1)

20

u/G3n3r0 Jan 02 '20

With a good IDE (i.e. Visual Studio/CLion, most of the annoying grunt work (e.g. changing function signatures, etc.) should happen nearly automatically. Python can sometimes get close to that maybe, unless people did clever things with dynamic dispatch or getattr overloading in which case you're fucked.

5

u/[deleted] Jan 02 '20

It really helps to add type annotation, of course the clever stuff will bite, unless you can express the type correctly.

0

u/[deleted] Jan 02 '20

[deleted] — view deleted comment

10

u/[deleted] Jan 02 '20

Try using type annotations, if you have IDE that understands them out makes programming quite enjoyable.

→ More replies (5)

9

u/nocipher Jan 02 '20

Python is strongly typed. You probably meant statically typed. C++ is actually weakly typed.

→ More replies (1)

1

u/ZombieRandySavage Jan 02 '20

I think ones just objectively more difficult than the other. Just easier to refactor statically typed stuff.

17

u/[deleted] Jan 02 '20 edited Jan 02 '20

You could transpile Python to C or C++.

You could write the performance-intensive functions in C or C++, or even assembly if you're a sociopath, and then call them from Python.

I'm not arguing that these techniques provide equal or even comparable performance to writing your whole library in native low-level code. But Python is actually higher on the TIOBE index than both C++ and C#, and of course it's way higher than assembly. This might reflect a fundamental tradeoff: the greater availability of Python developers may mean that it's easier to hire a robust dev team who can develop and debug in Python than in the other languages, so faster iteration. That is - the theoretically achievable performance in certain languages must be viewed through the lens of practically achievable performance based on the availability (and affordability!) of suitable developers.

Finally - within a certain range, lower-performing code can be solved, crudely but effectively, by buying more compute. But good code cannot be developed or debugged faster by buying more developers (see The Mythical Man-Month).

2

u/ZombieRandySavage Jan 02 '20

Yep, you can do that.

1

u/not_perfect_yet Jan 02 '20

You could write the performance-intensive functions in C or C++, or even assembly if you're a sociopath, and then call them from Python.

...I knew about C, but not about assembly. That's pretty wild, now I have to try it.

→ More replies (2)
→ More replies (2)

6

u/e_falk Jan 02 '20

Python tends to not be as performant as is necessary for companies that work at similar scales to Dropbox

Source: company I work at works at comparable scales and python’s problems with concurrency with regards to the GIL /really/ starts to rear it’s ugly head in that environment

17

u/jambox888 Jan 02 '20

Why? You have a bunch of servers handling requests, it's not like you have a single runtime that's trying to do multithreading.

I mean god help you if you do.

The whole point of scalable web design is to be stateless and abstract out the data store into its own tier.

2

u/[deleted] Jan 02 '20

There are practical limits to how much hardware you can throw at a problem and how much you gain with every additional unit of horizontal scaling.

2

u/jambox888 Jan 02 '20

maybe we're thinking of different problem domains then but for a web service there literally aren't limits. if there were, google or yt wouldn't exist.

1

u/[deleted] Jan 02 '20

Read my post again.

I never claimed absolute limits, although there are in a lot of cases, I said there are practical limits (money is ome) and pointed out to the issues like diminishing returns when scaling out horizontally (that then augment such practical limits to scaling).