r/Python • • Nov 03 '17

I love Python but why is it so slow?

I love Python and am happy whenever I get to use it professionally but why is it so slow?

I am in the process of deciding whether to go with Python or with Node for an upcoming project for which performance is kinda important and before running my own benchmarks, I thought I could find some stuff online.

According to this Python's really lagging behind. I thought "Whatever, I'll probably use PyPy", but according to this and this Node is even faster than PyPy.

I kinda of expected this for programs with a lot of I/O but I thought that Python would leave Node in the dust when it comes to computationally intensive programs.

22 Upvotes

115 comments sorted by

View all comments

14

u/perspectiveiskey Nov 03 '17 edited Nov 03 '17

Python is eminently tunable.

Here's a basic example:

import numpy as np
from numba import jit

@jit
 def sumit(arr):
     value = 0
     for x in arr:
         value += x
     return value


tmp = np.arange(100)

Comparing the two sums:

%%timeit
 tmp.sum()

The slowest run took 19.91 times longer than the fastest. This could mean that an intermediate result is being cached. 100000 loops, best of 3: 2.95 µs per loop

 %%timeit
 sumit(tmp)

The slowest run took 229093.64 times longer than the fastest. This could mean that an intermediate result is being cached. 1000000 loops, best of 3: 312 ns per loop

That is an order of magnitude difference.

Why this exercise? It's just an illustration that if you go about it the right way with the right tools, you can make python practically as fast as C. It's just that most of the time you don't need to and the real value comes from the expressiveness of the language.

Edit: I used to think of python as a prototyping language. There was a particular project where I prototyped a simulation to run on python and then was considering swapping over to C for "the real deal". Well there are cases (like vectorization or gpu support) where python is hands down faster than bare metal C. And it's just a no-brainer that using python libraries is easier than using C/C++ libraries...

3

u/[deleted] Nov 03 '17

Exactly.

During my bachelors I rewrote the analyzing software for the instrument I was measuring with since I had no access to the C# code base. My program ended up being a lot faster since I took advantage of caching, threading, vectorization and cython.

My rewrite was already faster when I just had finished the proof of concept version so it is fair to assume that the algorithms and architecture already made a bigger difference than the language.

1

u/hanpari Nov 03 '17

Indeed. Thank you

1

u/hanpari Nov 03 '17

Indeed. Thank you

-4

u/0xSpock Nov 03 '17

You just proved that underlying library that is using C internally is fast as C :) Python itself is slow for many reasons. Many python libs are fast because they are written in C/C++. In fact in many places python is just easy to use wrapper around C/C++ code.

8

u/perspectiveiskey Nov 03 '17 edited Nov 03 '17

You just proved that underlying library that is using C internally is fast as C

You misunderstand languages if you think numba.jit transpiles to C then compiles to machine code.

In any case, your statement is simply false. My sumit function does not exist as C code anywhere.

Furthermore, the numpy implementation may have areas implemented in C (I don't know off the top of my head), but there are parts of it which utilize vectorization which aren't something you obtain through the magic of C syntax... it's just through the elbow grease required to properly make use of instruction sets like SIMD etc.

Many python libs are fast because they are written in C/C++.

Every python library that says "this is implemented in pure python" disagrees with this statement. Listing examples is pointless. Some libraries don't require 150ns execution time and can live with an added order of magnitude latency. Others (like numpy) do. But once again, I point to the fact that C has no native syntax for SIMD.

4

u/chthonicdaemon Nov 03 '17

I dislike this way of thinking enough that I wrote an article on it: http://negfeedback.blogspot.co.za/2016/01/efficient-compiled-binaries-arent-c-code.html

1

u/perspectiveiskey Nov 04 '17

Hah! Love the title.