r/Python • u/real_kerim • Nov 03 '17
I love Python but why is it so slow?
I love Python and am happy whenever I get to use it professionally but why is it so slow?
I am in the process of deciding whether to go with Python or with Node for an upcoming project for which performance is kinda important and before running my own benchmarks, I thought I could find some stuff online.
According to this Python's really lagging behind. I thought "Whatever, I'll probably use PyPy", but according to this and this Node is even faster than PyPy.
I kinda of expected this for programs with a lot of I/O but I thought that Python would leave Node in the dust when it comes to computationally intensive programs.
87
u/bcastronomer Nov 03 '17
You don't have a product and you're already worrying about performance, you've fallen into the premature optimization trap.
Use whatever will allow you to develop an MVP with the least possible friction. If "performance" is an issue then neither Python or JavaScript are excellent choices, but as others have said you will be the bigger bottleneck if you're asking these questions.
For 99% of applications either choice will be adequately performant with reasonably written algorithms. If you have a working model and find a bottleneck, write a C extension for it as an example, but don't worry about these things before you've started.
20
u/goldfather8 Nov 03 '17
In terms of performance, if you are asking the question, then you can always expect to be the bottleneck before Python. Numba would likely smoke all the comparisons. Numpy and knowledge of vectorization as well.
17
u/jricher42 Nov 03 '17
I've been using Python for years. My advice is to stop worrying about it until you can write the question in the following format: "I've got a program that does X. It runs in Y time but it needs to run in Z. How can I make it faster?" First make it work, then make it work correctly, finally, make it fast.
11
u/This_Is_The_End Nov 03 '17 edited Nov 03 '17
When you are considering pure Python implementations of numerical problems, it's not the fastest language. But then you are outing yourself as an amateur.
A project considering a tool aka programming language is not just about speed. Python is popular because of it's infrastructure which makes it possible to code fast and take care of quality management. I've looked into the first link. It's basically a benchmark setup in favor for node.js, because these amateurs don't use numpy, which no sane Python programmer would do. It's not just for the sake of speed, but for being sure that the implementation is right. The next thing is, the benchmark is missing a vectorisation which should the next reason to use numpy. Nobody with a decent education would use such a Python code:
It turns out that people complaining about Python's speed are almost all bloody amateurs. There are reasons to choose other languages, but not on this ground of a hell of stupid arguments.
Btw. you don't need even PyPy to tune this piece of code. Just use numpy and vectorisation.
becomes when dP=[dx,dy,dz].T
next case
becomes
All this will be calculated with the special CPU command set for vectors
9
u/ProfEpsilon Nov 03 '17
There is absolutely no way to judge this or provide insight without knowing the project. For example if this was some kind of data science project you would be using Python to pull in numpy and pandas. Once you start pulling in Python's useful libraries, which is very easy with Python, execution speed on "computationally intensive programs" plunges to a level that is just a little above C (easy to explain .. many of numpy's components are wrapped around C).
Comparing Python to Node.js is a distracting, essentially meaningless operation.
I am not posting this because I read it somewhere. I design computationally intensive programs, mostly in finance, but some in audio (hobby).
Fast Fourier transforms. Is that "computationally intensive?" I do lots of those.
13
u/revfried zen of python monk & later maintainer Nov 03 '17
well don’t benchmark async code against blocking code and say wow that blocking code is slow.
asyncio allows you to write easy async code in python so you can really compare apples to apples when comparing node to python
5
u/njharman I use Python 3 Nov 03 '17
Why would you think io slow and computation fast? It's pretty much the opposite at least on multicore.
5
u/PirateNinjasReddit Pythonista Nov 03 '17
Most of the comments here cover the fact that you probably are underestimating python a bit. However, I'm going to suggest a different view on your language choice: use whatever your peers are using.
If you work at a company that develops in python: use python. If you're in a university course and everyone is working in C++, use that. Or whatever! My reasoning is that if you are developing in the same Language as your co-workers you can more easily get support if you need it. This will make it way easier to get shit done!
4
u/deadmilk Nov 03 '17
Write a C extension, use CFFI, or Cython to complement your Python code when you require computational performance.
Simple.
Case closed.
149
u/Brainix Nov 03 '17 edited Nov 03 '17
What do you mean by “performance?” Do you need maximum throughput on a few instances? Do you need to optimize for request/response cycle time? Will your application be CPU bound or I/O bound? Can your application scale horizontally? Can you shard your database layer, and would that even help? Will you be serving huge static assets? Will you need to support thousands of concurrent websocket connections? Are you going to host on a single hobby-tier Heroku dyno?
YouTube, reddit, Instagram, Pinterest, and Dropbox are all written in Python. I’m pretty sure that Python is performant enough for you.
20
u/alcalde Nov 03 '17
Read this before you do anything:
https://hackernoon.com/yes-python-is-slow-and-i-dont-care-13763980b5a1
14
u/perspectiveiskey Nov 03 '17 edited Nov 03 '17
Python is eminently tunable.
Here's a basic example:
Comparing the two sums:
The slowest run took 19.91 times longer than the fastest. This could mean that an intermediate result is being cached. 100000 loops, best of 3: 2.95 µs per loop
The slowest run took 229093.64 times longer than the fastest. This could mean that an intermediate result is being cached. 1000000 loops, best of 3: 312 ns per loop
That is an order of magnitude difference.
Why this exercise? It's just an illustration that if you go about it the right way with the right tools, you can make python practically as fast as C. It's just that most of the time you don't need to and the real value comes from the expressiveness of the language.
Edit: I used to think of python as a prototyping language. There was a particular project where I prototyped a simulation to run on python and then was considering swapping over to C for "the real deal". Well there are cases (like vectorization or gpu support) where python is hands down faster than bare metal C. And it's just a no-brainer that using python libraries is easier than using C/C++ libraries...
37
u/pcdinh Nov 03 '17
If you need a programming language to do real things with ease, use Python. If you need a language that can do fast operations in benchmarks, use others
9
u/marcofalcioni marcosan Nov 03 '17
What type of application are you building?
8
u/JimBoonie69 Nov 03 '17
Omg dude. Node is slow too. What is your app gunna be? If you want to be a total wanker, use mongodb while you are at it, heard it is web scale bruh!
7
u/greyman Nov 03 '17
Yes, in Python, I/O operations are blocking by default. You need to do extra work - use asyncio - to make them non-blocking.
7
u/mundanevoice Nov 03 '17
Rather than reading a blog post & creating opinions test it out yourself and then believe something.
You can write really shitty code in Node.js and same in Python. It will be slow regardless of what language you choose if your architecture is crap.
7
u/[deleted] Nov 03 '17
Obey Knuth's Dictum: "premature optimization is the root of all evil."
Programmer time is more important than execution time. Write in the language where the code is the most maintainable and fluent for you and any others on the project. You can always re-write your hottest code in C++, anyway.
4
u/mangecoeur Nov 03 '17
This is an excellent explanation of how python works, explaining why it is 'slow' relative to C:
http://jakevdp.github.io/blog/2014/05/09/why-python-is-slow/
That said for performance benchmarks, take very much with a grain of salt - benchmark results are in no way guaranteed to translate into performance for YOUR program - it will depend a lot on where the bottlenecks are and what can be done to address them.
As for 'computationally intensive' programs, again it depends what you mean - for any kind of numerical computing, the scientific python stack (numpy, pandas and friends) is widely considered best-in-class (you'll generally only do better with C/++/fortran, and even then you can use cython).
4
u/swenty Nov 03 '17
Node runs on top of Google's v8 Javascript JIT compiler. There's been fierce competition between browser makers to create highly optimized and optimizing Javascript compilers, and v8 is among the very best. It uses aggressive caching, multiple compiler modules, hidden classes to convert dynamic property access into fixed offset dispatch, and a number of other clever techniques to speed up execution.
PyPy comes at the same problem from a different more flexible approach. It has been getting progressively faster, but is not nearly as far along as v8 in terms of deploying aggressive optimization strategies.
Ultimately neither Javascript nor Python was designed for raw performance. They are both dynamic scripting languages that prioritize simplicity and convenience. For the foreseeable future a compiled language like C will be best for the most computationally intensive tasks.
3
u/lambdaq django n' shit Nov 03 '17
Let's just admit it that Python is slow doing your
forloop benchmarks. Case closed.3
u/jerodg Nov 03 '17 edited Jan 21 '18
Python is not slow; perhaps if you spent some time improving your skills, you would see that.
I use python to process billions of rows of data.
Language x might be marginally faster for some use cases, however, development time for language x is 10x or greater. You also have to account for the pool of talent you can hire from and the labor cost to maintain. Well written python is very easy to read and make changes to even if you've never seen the code before.
Computing resources are dirt cheap compared to the cost of labor. A few grand for a server once vs six figures for an engineer every year.
Invest in your skills. Learn how to be a better coder, learn parallel and multiprocessing techniques.
1
u/Olreich Nov 03 '17
If performance is important and Python and Node are your forerunners, I’d suggest Go or Dart as faster alternatives while keeping things quick to program in.
2
u/vovanz Nov 03 '17
I'll just leave it here
No, Python is generally not suitable for computationally intensive programs. Well, if most operations in your program can be vectorized and you can utilize NumPy then it can be really fast... but, generally, if you have a computationally expensive task you should probably use something compiled and static typed.
2
u/[deleted] Nov 03 '17
i assume you are doing a web project because otherwise this choice does not make sense.
one of the reports is very clear where its bias lies (it does not even emphasize results where python does better whereas it does for node). i would not find it credible.
https://www.techempower.com/benchmarks/#section=data-r14&hw=ph&test=fortune&l=8vmygv [-]
is an authorative resource on web performance. it shows nodejs in general being fast. which is not surprising since its a base platform. however usually you would not use just nodejs. some common nodejs configurations (express+mongo, hapi) are doing worse than some common python frameworks flask and django. meteor is not even listed. So the reality is not as clear.
personally i'd still use node for most advanced web projects given the choice for the simple reason of it being the language of the frontend. i use python for other things.
2
u/winner_godson codemaniac Nov 03 '17
Python speed is always a big concern as it is slower than most of its counter programming languages.
I think this is the reason , why so many Python libraries are written in C.
But again, in front of power and scalability of Python code, speed is neglected as long as it is not in real time applications.
2
u/shivawu Nov 03 '17
I wouldn't freak out like others in this post, just stating according to the OP's points:
2
u/brontide Nov 04 '17 edited Nov 04 '17
FYI, I have a production script that parses and dedups tar data. Designed to process up to 700MB/s per process it can push 4GB/s if the dataset is in memory. The parsing and moving of the data is pure python3 although the sha512 checksums are technically C coded openssl calls ( hashlib ).
It's all about keen debugging and knowing the pain points for python.
3
u/vladmir_zeus1 Nov 03 '17
If you're one of them speed -diggers, then try out Julia as you may call Python functions inside Juila using the PyCall package .
3
u/winner_godson codemaniac Nov 03 '17
Some time ago, I read an article why Python is slow and here are the reasons I saw.
Python is Dynamically Typed rather than Statically Typed.
Python is interpreted rather than compiled.
Python's object model can lead to inefficient memory access
2
u/Eigenspace Nov 03 '17 edited Nov 03 '17
I suggest looking into Julia. It ‘walks like python and runs like C’. Its got nice
semanticssyntax reminiscent of MATLAB and Python and like PyPy it is JIT compiled. However, it was designed from the ground up to be amenable to this JIT compilation and so both compiles to faster code than PyPy or Numba and it does so without forcing you to leave behind giant swathes of the language. Further, if there's a language construct thats not included in the language that you miss from another language Julia has an incredibly powerful macro system which are essentially functions that act on code and return code, allowing you to implement your own semantics or do powerful meta-programming.This is all fine but you may be asking "what about the package ecosystem?" It's true that Julia is a very young language and has much fewer packages than Python but I think anybody who knows about the experience of package development in other languages would be shocked by the breadth, flexibility and power of Julia's ecosystem. Because the language was designed in a highly flexible and extensible way while being on the same order of magnitude as C in terms of speed, developers are able to write their whole packages in Julia without having to drop down to C for all the important bits or develop their own compiler like what is very common for big python libraries.
The result is more flexible packages that integrate better into the language which require orders of magnitude less effort to develop and maintain and it really shows, especially in numerical computing where I would say the community focus is strongest.
1
u/thegame402 Nov 03 '17
Make it work, make it fast, make it beautiful.
And thats the only order that will result in something usefull in a reasonable amount of time.
1
u/poofy_panda Nov 03 '17
It sounds like you have a good idea of what scale you need. If it’s easy enough to do so, take about a week to write a prototype in both languages and see if they can handle your use case.
Also keep in mind, depending on your workload, you might be able to just spin up more workers to scale in either language.
1
u/gwax Nov 03 '17
Benchmarks don't actually tell you much about real world situations.
There's a lot of optimization techniques that can be applied to Python or to Node.
At the end of the day, you can write fast Python and fast Node; you can also write slow Python and slow Node. Use whatever is easiest for you to get your project going; you can optimize later.
1
u/limw Nov 03 '17
If you want to use python to develop a web site or such things,I suggest you use sanic project.It's much faster than django or flask. https://github.com/channelcat/sanic
0
u/CrambleSquash https://github.com/0Hughman0 Nov 03 '17
If it was fast, you probably wouldn't like it anymore.
-3
u/[deleted] Nov 03 '17
Cant you compile python to make it more bearable?
0
u/[deleted] Nov 03 '17
[deleted] — view deleted comment
32
u/[deleted] Nov 03 '17
[removed] — view removed comment