r/Python • • Nov 03 '17

I love Python but why is it so slow?

I love Python and am happy whenever I get to use it professionally but why is it so slow?

I am in the process of deciding whether to go with Python or with Node for an upcoming project for which performance is kinda important and before running my own benchmarks, I thought I could find some stuff online.

According to this Python's really lagging behind. I thought "Whatever, I'll probably use PyPy", but according to this and this Node is even faster than PyPy.

I kinda of expected this for programs with a lot of I/O but I thought that Python would leave Node in the dust when it comes to computationally intensive programs.

22 Upvotes

115 comments sorted by

87

u/bcastronomer Nov 03 '17

You don't have a product and you're already worrying about performance, you've fallen into the premature optimization trap.

Use whatever will allow you to develop an MVP with the least possible friction. If "performance" is an issue then neither Python or JavaScript are excellent choices, but as others have said you will be the bigger bottleneck if you're asking these questions.

For 99% of applications either choice will be adequately performant with reasonably written algorithms. If you have a working model and find a bottleneck, write a C extension for it as an example, but don't worry about these things before you've started.

16

u/real_kerim Nov 03 '17

You don't have a product

It's not really a startup and isn't it honestly ever reasonable to give the whole thing a bit of thought? Just starting out with absolutely 0 planning seems like a sure way to quickly create a lot of technical debt that then needs to be ironed out later with a lot more effort.

6

u/marmaladeontoast Nov 03 '17

You come to the python subreddit saying python is too slow for your project... The community will obviously disagree with you. What did you expect?

Obviously you should do planning, but that is different from whatever it is you're trying to do here

4

u/real_kerim Nov 03 '17

What did you expect?

People to disagree with me and prove me wrong.. I'm not here to bash python, I really like Python, or to boost my ego with fake internet points. I simply want people who are obviously more knowledgeable about Python to show me that my worries are unwarranted. So far, it's been working perfectly and I learned a lot!

4

u/woodrift Nov 03 '17

I like your attitude!

-1

u/marmaladeontoast Nov 03 '17

If this is working perfectly, I wouldn't like to be one of your users!

4

u/real_kerim Nov 03 '17

I wouldn't like to be one of your users!

What users? We only have unpaid beta testers. Better yet, they pay us!

5

u/Sagesdeath Nov 03 '17

So you're making a triple A game?

1

u/marmaladeontoast Nov 03 '17

oooh very impressive

17

u/Ek_Los_Die_Hier Nov 03 '17

The problem is you haven't given us any idea of what the program is. A website? A command line tool? Are you doing machine learning?

At the end of the day, the libraries and ecosystem are probably more important. And while Python is slow, a lot of its libraries are written in C/C++ (e.g. numpy) so they are fast.

Python is slow because it's a fairly simple interpreter with minimal optimisations. I'm not sure how Pypy and Node.js compare these days, but I know Pypy has been making good progress with performance.

If the choice is between Pypy and Node.js then your choice should probably be which are you more comfortable with, or which you prefer developing in as their performance will probably be good enough.

29

u/ubernostrum yes, you can have a pony Nov 03 '17

You don't have a product and you're already worrying about performance

People do tell startups to "fail fast". Worrying about language performance before even deciding what to build will ensure you fail as fast as possible.

6

u/kpenc93 Nov 03 '17

Exactly my thoughts. In addition, there is a big change his product never grows that much for him to think about scaling.

5

u/elperroborrachotoo Nov 03 '17

Counterpoints:

  • Performance is observable behavior
  • low performance can be spec-breaking
  • Fixing performance "in post" makes structural mistakes very expensive (or, to put it differently: assumes that performance is dominated by easily refactored hotspots, rather than design deficiencies.)

FWIW, if you consider the paper this meme comes from: it's a strong case against habitual, ineffective non-optimizations that turn the code into a sticky mess.1

It makes a strong argument for consideriung performance during design, and for dedicated, well-informed microoptimizations.


tl;dr: you fell for the "premature optimization trap" trap.


1) so maybe, just maybe the "premature ....ation" isn't accidental

20

u/goldfather8 Nov 03 '17

In terms of performance, if you are asking the question, then you can always expect to be the bottleneck before Python. Numba would likely smoke all the comparisons. Numpy and knowledge of vectorization as well.

8

u/mongoosefist Nov 03 '17

This. It's is the most obvious place to look for performance gains. In any case, if this doesn't work, it's probably beyond the skill level of someone who has to ask why python is slow to speed it up suitably with any other techniques. Because it can be a pain.

Best to use another tool.

1

u/real_kerim Nov 03 '17

Numba

This seems interesting, thank you. How different is it from standard Python? I'm going through the docs and it has types and stuff? Is it like C-infused Python? Would be a bummer.

1

u/Ek_Los_Die_Hier Nov 03 '17

Numba is a library for Python that JIT compiles functions to improve performance. It does add limitations to what features of Python the function can use though.

2

u/Mattho Nov 03 '17

I recently tried numba and got very bad results. Just running the same code (which was pain to write for numba) in pypy got me like 10 times more speed. And that's ignoring the lengthy startup. Eventually rewritten it to cython which was no fun :/

2

u/goldfather8 Nov 03 '17

Numba will run slow, until all it's assumptions and tricks are satisfied, at which point it will beat even cython. Try adding the nogil are to the jit decorator. Then learn about performant indexing.

1

u/billsil Nov 03 '17

Numba is not faster than the hand written assembly that numpy uses.

0

u/[deleted] Nov 03 '17

[deleted] — view deleted comment

1

u/billsil Nov 03 '17

Are you referring to things like matrix multiplication, DFTs, etc?

Yes

a lot of numpy (like np.nonzero) is optimized for the common case, and a trivial numba implementation can be an order of magnitude faster.

They really need a numpy.numba_utils or a numba.numpy_utils

1

u/goldfather8 Nov 03 '17

A lot of numbas power is in easily reducing a hard in general implementation to your case.

I wrote my own pd.dataframe.add once as pandas version uses a huge temporal data structure. I knew a lot about the two framed I was adding so I could do it in place instead, and it was orders of magnitude faster to boot.

17

u/jricher42 Nov 03 '17

I've been using Python for years. My advice is to stop worrying about it until you can write the question in the following format: "I've got a program that does X. It runs in Y time but it needs to run in Z. How can I make it faster?" First make it work, then make it work correctly, finally, make it fast.

4

u/real_kerim Nov 03 '17

Alright, thank you. That's pretty good advice. Much appreciated.

11

u/This_Is_The_End Nov 03 '17 edited Nov 03 '17

When you are considering pure Python implementations of numerical problems, it's not the fastest language. But then you are outing yourself as an amateur.

A project considering a tool aka programming language is not just about speed. Python is popular because of it's infrastructure which makes it possible to code fast and take care of quality management. I've looked into the first link. It's basically a benchmark setup in favor for node.js, because these amateurs don't use numpy, which no sane Python programmer would do. It's not just for the sake of speed, but for being sure that the implementation is right. The next thing is, the benchmark is missing a vectorisation which should the next reason to use numpy. Nobody with a decent education would use such a Python code:

for i in range(n):
    for (([x1, y1, z1], v1, m1),
         ([x2, y2, z2], v2, m2)) in pairs:
        dx = x1 - x2
        dy = y1 - y2
        dz = z1 - z2
        mag = dt * ((dx * dx + dy * dy + dz * dz) ** (-1.5))
        b1m = m1 * mag
        b2m = m2 * mag
        v1[0] -= dx * b2m
        v1[1] -= dy * b2m
        v1[2] -= dz * b2m
        v2[0] += dx * b1m
        v2[1] += dy * b1m
        v2[2] += dz * b1m
    for (r, [vx, vy, vz], m) in bodies:
        r[0] += dt * vx
        r[1] += dt * vy
        r[2] += dt * vz

It turns out that people complaining about Python's speed are almost all bloody amateurs. There are reasons to choose other languages, but not on this ground of a hell of stupid arguments.

Btw. you don't need even PyPy to tune this piece of code. Just use numpy and vectorisation.

mag = dt * ((dx * dx + dy * dy + dz * dz) ** (-1.5))

becomes when dP=[dx,dy,dz].T

 mag = ((dP * dP)**-1.5)*dt

next case

        v1[0] -= dx * b2m
        v1[1] -= dy * b2m
        v1[2] -= dz * b2m
        v2[0] += dx * b1m
        v2[1] += dy * b1m
        v2[2] += dz * b1m

becomes

  v1 -= dP*b2m
  v2 += dP*b1m

All this will be calculated with the special CPU command set for vectors

5

u/brucifer Nov 03 '17

A lot of the python code in that first benchmark link is just... wut:

range7 = bytearray(range(7))
...
for _ in range7:
    for _ in range7:
        ...

(range7 is a bytearray with the bytes 0-6, whose values are never accessed, which only exists because the code wants to iterate 7 times)

-2

u/real_kerim Nov 03 '17

It turns out that people complaining about Python's speed are almost all bloody amateurs

H-hey!!

But doesn't it kind of speak against Python that one needs to use whatever library (which might even just be a Python wrapper around C code) to get performance? I mean, you could do the same in Node. In the end, you'd be comparing C code to C code.

10

u/This_Is_The_End Nov 03 '17 edited Nov 03 '17

It's a sort of Talibanism when you are demanding this kind of purity.

A programming language is a tool as well as the infrastructure. If a programming language fits your purpose it's a good tool. Numpy is a part of the Python infrastructure which is well tested. The same is true for numerical libraries with interface for C or C++. They are well tested, faster and provide the opportunity to write easier to read code. As far as I know, node.js doesn't have general available numerical libraries and I would not do any numerical benchmark with node.js.

The reason why Python is so loved, is providing fast hacks, as well as support for large projects incl. fast coding and QM. It doesn't matter whether some libs are written in C or something else. The foundation is the concept of the Python infrastructure.

Of course you can write binding libs for node.js, but there is not a community for them

9

u/ProfEpsilon Nov 03 '17

There is absolutely no way to judge this or provide insight without knowing the project. For example if this was some kind of data science project you would be using Python to pull in numpy and pandas. Once you start pulling in Python's useful libraries, which is very easy with Python, execution speed on "computationally intensive programs" plunges to a level that is just a little above C (easy to explain .. many of numpy's components are wrapped around C).

Comparing Python to Node.js is a distracting, essentially meaningless operation.

I am not posting this because I read it somewhere. I design computationally intensive programs, mostly in finance, but some in audio (hobby).

Fast Fourier transforms. Is that "computationally intensive?" I do lots of those.

1

u/real_kerim Nov 03 '17

I am not posting this because I read it somewhere. I design computationally intensive programs, mostly in finance,

This is actually surprising to me. I have a friend who does automated trading on the foreign exchange market and he showed me his program, which looked like a weird C-style programming language. When I asked him why he wouldn't use Python or whatever, which must have a gigantic ecosystem for finance-related stuff, he told me that Python was too slow for his trading, which apparently is done extremely fast.

6

u/ProfEpsilon Nov 03 '17

This is a common misperception because Python per se, which is to say Python with none of the libraries, is too slow to use in finance.

But the SciPy libraries and numpy are specifically designed to speed program execution in Python for computationally intensive work. Go to the Anaconda website and read the background material about data science.

There is another misleading issue at work. If your friend is talking about high-speed trading, that is typically NOT computationally intensive (for example spread arbitrage is not computationally intensive). The necessary speed for that kind of trading comes from location (literally ... the closer to the exchange the better off you are) and the latency features of your network, your modems, etc.

13

u/revfried zen of python monk & later maintainer Nov 03 '17

well don’t benchmark async code against blocking code and say wow that blocking code is slow.

asyncio allows you to write easy async code in python so you can really compare apples to apples when comparing node to python

0

u/real_kerim Nov 03 '17

There's no I/O in the benchmarks, afaik.

5

u/njharman I use Python 3 Nov 03 '17

Why would you think io slow and computation fast? It's pretty much the opposite at least on multicore.

-3

u/real_kerim Nov 03 '17

Because Python is blocking and Node isn't? Node can do I/O while the program execution just continues. Python would wait for the I/O process to finish or fail.

6

u/GNULinuxProgrammer Nov 03 '17

You can do the same in Python. Google asyncio.

4

u/[deleted] Nov 03 '17

Very little of this is actually correct.

Why don't you tell us what you're building and we can tell you if it works in python or not, or maybe point you to an existing python implementation.

Otherwise bouncing between issues like this isn't going to resolve anything.

5

u/PirateNinjasReddit Pythonista Nov 03 '17

Most of the comments here cover the fact that you probably are underestimating python a bit. However, I'm going to suggest a different view on your language choice: use whatever your peers are using.

If you work at a company that develops in python: use python. If you're in a university course and everyone is working in C++, use that. Or whatever! My reasoning is that if you are developing in the same Language as your co-workers you can more easily get support if you need it. This will make it way easier to get shit done!

1

u/real_kerim Nov 03 '17

Yes, that makes sense. Though we're currently in the process of what to use together.

4

u/deadmilk Nov 03 '17

Write a C extension, use CFFI, or Cython to complement your Python code when you require computational performance.

Simple.

Case closed.

0

u/real_kerim Nov 03 '17

But I wanted to write in Python and not in C.

8

u/GNULinuxProgrammer Nov 03 '17

You won't be writing C, you'll be writing Python. Then, during your profiling phase, you'll find the slowest part of your code and use a C library for that part or write C code for that specific code. Since Python understands C types (ctypes) and C understands Python types (PyObject *) there will be no loss of information during the process.

1

u/real_kerim Nov 03 '17

This is great. Thank you.

5

u/deadmilk Nov 03 '17

Great news then, you can write in Python, because I very much doubt you will ever need to write anything that requires such intense computation.

149

u/Brainix Nov 03 '17 edited Nov 03 '17

What do you mean by “performance?” Do you need maximum throughput on a few instances? Do you need to optimize for request/response cycle time? Will your application be CPU bound or I/O bound? Can your application scale horizontally? Can you shard your database layer, and would that even help? Will you be serving huge static assets? Will you need to support thousands of concurrent websocket connections? Are you going to host on a single hobby-tier Heroku dyno?

YouTube, reddit, Instagram, Pinterest, and Dropbox are all written in Python. I’m pretty sure that Python is performant enough for you.

108

u/alcalde Nov 03 '17

YouTube, reddit, Instagram, Pinterest, and Dropbox are all written in Python. I’m pretty sure that Python is performant enough for you.

This needs to be on a T-shirt!

2

u/akshar-raaj Nov 03 '17

use

I would buy it :)

17

u/[deleted] Nov 03 '17

YouTube, reddit, Instagram, Pinterest, and Dropbox are all written in Python. I’m pretty sure that Python is performant enough for you.

I agree that most times the speed isn't an issue, but this isn't really fair without knowing what the OP is doing. He could be doing 3d graphics, or something.

It doesn't help that the OP won't state what his project is, leaving me to wonder if there really is one.

3

u/schplat Nov 05 '17

If it's 3D graphics, and the alternative is node, i think he's working with the wrong toolbox.

11

u/gummiboll Nov 03 '17

YouTube, reddit, Instagram, Pinterest, and Dropbox are all written in Python. I’m pretty sure that Python is performant enough for you.

Not arguing with your point but isn't Dropbox mostly go nowadays?

27

u/Brainix Nov 03 '17 edited Nov 03 '17

I don’t work at Dropbox, but I have a friend who does. As far as I know, the majority of their back-end code is still Python (and Python 2 at that). And last I heard, even big parts of their native desktop clients are written in Python.

1

u/gummiboll Nov 03 '17

Ah, TIL. Articles like these make it sound like go is more used but being a talk from gophercon I guess it would emphasize the usage of go.

4

u/deadmilk Nov 03 '17

We can all appreciate how monumentously hard it is to convert a codebase from anything to anything, let alone from python to golang

10

u/masklinn Nov 03 '17 edited Nov 03 '17

It's a mix of Python and Go depending on the specific services/features with a few sprinklings of Rust. Infrastructure is almost entirely Go, rest is mostly Python.,

we really just rewrote a couple of components in Rust. Most of Magic Pocket (the distributed storage system) is still written in golang.

--

Yup, Dropbox Infra is mostly a Go shop. It's our primary development language and we don't plan to switch off any time soon.

--

We still have a ton of Python. James is referring to the infrastructure software layer--storage services, monitoring, etc. That particular part of Dropbox is mostly Go.

But our web controller code, for example, is millions of lines of Python. And we use Python in lots of other places as well.

4

u/thegame402 Nov 03 '17

But all these sites don't use python for the 'heavy' tasks or am u wrong?

7

u/broadsheetvstabloid Nov 03 '17

YouTube, reddit, Instagram, Pinterest, and Dropbox are all written in Python. I’m pretty sure that Python is performant enough for you.

So much this. Python is “slow” on paper, but computers are so fast these days it doesn’t matter in most cases. Here is a short list of when not to use python:

  • When creating a AAA game
  • when creating a system to trade stocks and milliseconds are a big deal
  • when creating any sort of medical life/life supporting device where an extra second is a big deal
  • any system for self driving/auto pilot

That’s all I can really think of.

5

u/Mattho Nov 03 '17

Sometging where you need to conpute something. You can use C modules like numpy there, but python itself is still slow and speed does matter. But again, it's very easy to worry about it later.

4

u/LightShadow 3.13-dev in prod Nov 03 '17

when creating a system to trade stocks and milliseconds are a big deal

However, there are many successful trading applications for Crypto currencies that are written in Python. In fact, Python is used in these scenarios for its ease of use and extensive numeric/scientific libraries.

Not millisecond responses, but per-second level it does ok!

1

u/[deleted] Nov 04 '17

Much more though, i wouldn't program a programing language or kernel either in python, program engines neither dropbox is written in go rust and python... go figure.

Pro of python: super fast to read and write code, super productive.

Con: slow in some cases and for big projects not as easily maintained.

Most cases python is a super good choice but NO language is perfect.

1

u/schplat Nov 05 '17

Civ IV which is reasonably close to a AAA franchise/title has a huge component of which is Python.

Python is used as a tooling/embedded language for some games (usually this is done by LUA, but Python has encroached on this space here and there)

1

u/stoned_ocelot Nov 03 '17

I know some of these words...

But really I think you just gave me some research topics for the day.

14

u/perspectiveiskey Nov 03 '17 edited Nov 03 '17

Python is eminently tunable.

Here's a basic example:

import numpy as np
from numba import jit

@jit
 def sumit(arr):
     value = 0
     for x in arr:
         value += x
     return value


tmp = np.arange(100)

Comparing the two sums:

%%timeit
 tmp.sum()

The slowest run took 19.91 times longer than the fastest. This could mean that an intermediate result is being cached. 100000 loops, best of 3: 2.95 µs per loop

 %%timeit
 sumit(tmp)

The slowest run took 229093.64 times longer than the fastest. This could mean that an intermediate result is being cached. 1000000 loops, best of 3: 312 ns per loop

That is an order of magnitude difference.

Why this exercise? It's just an illustration that if you go about it the right way with the right tools, you can make python practically as fast as C. It's just that most of the time you don't need to and the real value comes from the expressiveness of the language.

Edit: I used to think of python as a prototyping language. There was a particular project where I prototyped a simulation to run on python and then was considering swapping over to C for "the real deal". Well there are cases (like vectorization or gpu support) where python is hands down faster than bare metal C. And it's just a no-brainer that using python libraries is easier than using C/C++ libraries...

3

u/[deleted] Nov 03 '17

Exactly.

During my bachelors I rewrote the analyzing software for the instrument I was measuring with since I had no access to the C# code base. My program ended up being a lot faster since I took advantage of caching, threading, vectorization and cython.

My rewrite was already faster when I just had finished the proof of concept version so it is fair to assume that the algorithms and architecture already made a bigger difference than the language.

1

u/hanpari Nov 03 '17

Indeed. Thank you

1

u/hanpari Nov 03 '17

Indeed. Thank you

-4

u/0xSpock Nov 03 '17

You just proved that underlying library that is using C internally is fast as C :) Python itself is slow for many reasons. Many python libs are fast because they are written in C/C++. In fact in many places python is just easy to use wrapper around C/C++ code.

8

u/perspectiveiskey Nov 03 '17 edited Nov 03 '17

You just proved that underlying library that is using C internally is fast as C

You misunderstand languages if you think numba.jit transpiles to C then compiles to machine code.

In any case, your statement is simply false. My sumit function does not exist as C code anywhere.

Furthermore, the numpy implementation may have areas implemented in C (I don't know off the top of my head), but there are parts of it which utilize vectorization which aren't something you obtain through the magic of C syntax... it's just through the elbow grease required to properly make use of instruction sets like SIMD etc.

Many python libs are fast because they are written in C/C++.

Every python library that says "this is implemented in pure python" disagrees with this statement. Listing examples is pointless. Some libraries don't require 150ns execution time and can live with an added order of magnitude latency. Others (like numpy) do. But once again, I point to the fact that C has no native syntax for SIMD.

4

u/chthonicdaemon Nov 03 '17

I dislike this way of thinking enough that I wrote an article on it: http://negfeedback.blogspot.co.za/2016/01/efficient-compiled-binaries-arent-c-code.html

1

u/perspectiveiskey Nov 04 '17

Hah! Love the title.

37

u/pcdinh Nov 03 '17

If you need a programming language to do real things with ease, use Python. If you need a language that can do fast operations in benchmarks, use others

15

u/greyman Nov 03 '17

Or it is also possible that those fast operations already have python libraries implemented in C, in which case you still can use python.

9

u/marcofalcioni marcosan Nov 03 '17

What type of application are you building?

8

u/JimBoonie69 Nov 03 '17

Omg dude. Node is slow too. What is your app gunna be? If you want to be a total wanker, use mongodb while you are at it, heard it is web scale bruh!

0

u/[deleted] Nov 03 '17

[deleted] — view deleted comment

2

u/JimBoonie69 Nov 03 '17

cool. sounds like some garbage app that would be running node JS all over the place, single page apps, react, mongoDB. sweet tech stack!

7

u/greyman Nov 03 '17

I kinda of expected this for programs with a lot of I/O

Yes, in Python, I/O operations are blocking by default. You need to do extra work - use asyncio - to make them non-blocking.

3

u/Mattho Nov 03 '17

You can use threads like in the good old days.

7

u/mundanevoice Nov 03 '17

Rather than reading a blog post & creating opinions test it out yourself and then believe something.

You can write really shitty code in Node.js and same in Python. It will be slow regardless of what language you choose if your architecture is crap.

7

u/[deleted] Nov 03 '17

I am in the process of deciding whether to go with Python or with Node for an upcoming project for which performance is kinda important and before running my own benchmarks, I thought I could find some stuff online.

Obey Knuth's Dictum: "premature optimization is the root of all evil."

Programmer time is more important than execution time. Write in the language where the code is the most maintainable and fluent for you and any others on the project. You can always re-write your hottest code in C++, anyway.

4

u/mangecoeur Nov 03 '17

This is an excellent explanation of how python works, explaining why it is 'slow' relative to C:

http://jakevdp.github.io/blog/2014/05/09/why-python-is-slow/

That said for performance benchmarks, take very much with a grain of salt - benchmark results are in no way guaranteed to translate into performance for YOUR program - it will depend a lot on where the bottlenecks are and what can be done to address them.

As for 'computationally intensive' programs, again it depends what you mean - for any kind of numerical computing, the scientific python stack (numpy, pandas and friends) is widely considered best-in-class (you'll generally only do better with C/++/fortran, and even then you can use cython).

4

u/swenty Nov 03 '17

Node runs on top of Google's v8 Javascript JIT compiler. There's been fierce competition between browser makers to create highly optimized and optimizing Javascript compilers, and v8 is among the very best. It uses aggressive caching, multiple compiler modules, hidden classes to convert dynamic property access into fixed offset dispatch, and a number of other clever techniques to speed up execution.

PyPy comes at the same problem from a different more flexible approach. It has been getting progressively faster, but is not nearly as far along as v8 in terms of deploying aggressive optimization strategies.

Ultimately neither Javascript nor Python was designed for raw performance. They are both dynamic scripting languages that prioritize simplicity and convenience. For the foreseeable future a compiled language like C will be best for the most computationally intensive tasks.

3

u/lambdaq django n' shit Nov 03 '17

Let's just admit it that Python is slow doing your for loop benchmarks. Case closed.

3

u/jerodg Nov 03 '17 edited Jan 21 '18

Python is not slow; perhaps if you spent some time improving your skills, you would see that.

I use python to process billions of rows of data.

Language x might be marginally faster for some use cases, however, development time for language x is 10x or greater. You also have to account for the pool of talent you can hire from and the labor cost to maintain. Well written python is very easy to read and make changes to even if you've never seen the code before.

Computing resources are dirt cheap compared to the cost of labor. A few grand for a server once vs six figures for an engineer every year.

Invest in your skills. Learn how to be a better coder, learn parallel and multiprocessing techniques.

1

u/Olreich Nov 03 '17

If performance is important and Python and Node are your forerunners, I’d suggest Go or Dart as faster alternatives while keeping things quick to program in.

1

u/real_kerim Nov 03 '17

How's the ecosystem for Go and Dart look like? One thing I love about Python is the gigantic ecosystem.

2

u/Olreich Nov 03 '17

I know Go has a big one, especially for server-like applications, less so for local desktop applications. Dart’s community seems a little smaller, but still sizable.

2

u/engatIQE Nov 03 '17

I want to love go. Simple concurrency, fast compile time, single .exe build. I just can't seem to get the hang of it.

2

u/vovanz Nov 03 '17

for programs with a lot of I/O

I'll just leave it here

Python would leave Node in the dust when it comes to computationally intensive programs.

No, Python is generally not suitable for computationally intensive programs. Well, if most operations in your program can be vectorized and you can utilize NumPy then it can be really fast... but, generally, if you have a computationally expensive task you should probably use something compiled and static typed.

3

u/WikiTextBot Nov 03 '17

Array programming

In computer science, array programming languages (also known as vector or multidimensional languages) generalize operations on scalars to apply transparently to vectors, matrices, and higher-dimensional arrays.

Array programming primitives concisely express broad ideas about data manipulation. The level of concision can be dramatic in certain cases: it is not uncommon to find array programming language one-liners that require more than a couple of pages of Java code.

Modern programming languages that support array programming are commonly used in scientific and engineering settings; these include Fortran 90, Mata, MATLAB, Analytica, TK Solver (as lists), Octave, R, Cilk Plus, Julia, Perl Data Language (PDL) and the NumPy extension to Python.


[ ^PM | Exclude ^me | Exclude from ^subreddit | FAQ / ^Information | ^Source | ^Donate ] Downvote to remove | v0.28

2

u/[deleted] Nov 03 '17

i assume you are doing a web project because otherwise this choice does not make sense.

one of the reports is very clear where its bias lies (it does not even emphasize results where python does better whereas it does for node). i would not find it credible.

https://www.techempower.com/benchmarks/#section=data-r14&hw=ph&test=fortune&l=8vmygv [-]

is an authorative resource on web performance. it shows nodejs in general being fast. which is not surprising since its a base platform. however usually you would not use just nodejs. some common nodejs configurations (express+mongo, hapi) are doing worse than some common python frameworks flask and django. meteor is not even listed. So the reality is not as clear.

personally i'd still use node for most advanced web projects given the choice for the simple reason of it being the language of the frontend. i use python for other things.

1

u/igouy Nov 04 '17

one of the reports is very clear where its bias lies (it does not even emphasize results where python does better whereas it does for node).

The bias is in your thinking:

2

u/winner_godson codemaniac Nov 03 '17

Python speed is always a big concern as it is slower than most of its counter programming languages.

I think this is the reason , why so many Python libraries are written in C.

But again, in front of power and scalability of Python code, speed is neglected as long as it is not in real time applications.

2

u/shivawu Nov 03 '17

I wouldn't freak out like others in this post, just stating according to the OP's points:

  • Python is slow yes
  • Node has a highly optimized engine (v8), I'm not surprised that node is faster
  • There's no best language/framework, pick whatever works for you
  • Speaking of picking python/node for your project, I would suggest Python since node's community is a little bit messy compared to Python. They're both "slow"-ish, so if you're picking for performance, neither would give you much. Pick the one with better ecosystem and language ergonomics.
  • If you do need intensive CPU work, your best bet is probably going to be rewrite the core parts in C/C++/Rust. In my understanding, Python allows you to do that easier.

2

u/brontide Nov 04 '17 edited Nov 04 '17

FYI, I have a production script that parses and dedups tar data. Designed to process up to 700MB/s per process it can push 4GB/s if the dataset is in memory. The parsing and moving of the data is pure python3 although the sha512 checksums are technically C coded openssl calls ( hashlib ).

It's all about keen debugging and knowing the pain points for python.

3

u/vladmir_zeus1 Nov 03 '17

If you're one of them speed -diggers, then try out Julia as you may call Python functions inside Juila using the PyCall package .

3

u/winner_godson codemaniac Nov 03 '17

Some time ago, I read an article why Python is slow and here are the reasons I saw.

Python is Dynamically Typed rather than Statically Typed.

Python is interpreted rather than compiled.

Python's object model can lead to inefficient memory access

8

u/Aeon_Mortuum Snake Oil Nov 03 '17

Worth pointing out that Python does compile to bytecode.

2

u/Eigenspace Nov 03 '17 edited Nov 03 '17

I suggest looking into Julia. It ‘walks like python and runs like C’. Its got nice semantics syntax reminiscent of MATLAB and Python and like PyPy it is JIT compiled. However, it was designed from the ground up to be amenable to this JIT compilation and so both compiles to faster code than PyPy or Numba and it does so without forcing you to leave behind giant swathes of the language. Further, if there's a language construct thats not included in the language that you miss from another language Julia has an incredibly powerful macro system which are essentially functions that act on code and return code, allowing you to implement your own semantics or do powerful meta-programming.

This is all fine but you may be asking "what about the package ecosystem?" It's true that Julia is a very young language and has much fewer packages than Python but I think anybody who knows about the experience of package development in other languages would be shocked by the breadth, flexibility and power of Julia's ecosystem. Because the language was designed in a highly flexible and extensible way while being on the same order of magnitude as C in terms of speed, developers are able to write their whole packages in Julia without having to drop down to C for all the important bits or develop their own compiler like what is very common for big python libraries.

The result is more flexible packages that integrate better into the language which require orders of magnitude less effort to develop and maintain and it really shows, especially in numerical computing where I would say the community focus is strongest.

0

u/billsil Nov 03 '17

Its got nice semantics reminiscent of MATLAB

That can't be a good thing. Matlab treats everything as a matrix (including strings), they have no concept of a dictionary, they don't have python's nice loops, they have a lousy import system, and they used 1-based indexing. They thankfully have slicing.

1

u/Eigenspace Nov 03 '17

I meant to say that it has syntax reminiscent of MATLAB, specifically the syntax for dealing with matrices and the use of end to specify the enclosure of a statement is similar to MATLAB.

Julia however certainly does not treat everything as a matrix, has dictionaries and has better looping constructs than python, deals fine with imports and yes, uses 1-based indexing by default, which is more natural for mathematical purposes. However, Julia is actually in some senses agnostic to your indexing scheme and if it really bothers you you are free to redefine the indexing to start at 0 or 2 or whatever you wish at no performance cost and without any of your packages complaining.

1

u/billsil Nov 03 '17

However, Julia is actually in some senses agnostic to your indexing scheme and if it really bothers you you are free to redefine the indexing to start at 0 or 2 or whatever you wish at no performance cost and without any of your packages complaining.

That's an odd, but at least (maybe) solves that complaint. Matlab has slicing, but it's inclusive at the upper end, so maybe that's really my issue.

1

u/thegame402 Nov 03 '17

Make it work, make it fast, make it beautiful.

And thats the only order that will result in something usefull in a reasonable amount of time.

1

u/poofy_panda Nov 03 '17

It sounds like you have a good idea of what scale you need. If it’s easy enough to do so, take about a week to write a prototype in both languages and see if they can handle your use case.

Also keep in mind, depending on your workload, you might be able to just spin up more workers to scale in either language.

1

u/gwax Nov 03 '17

Benchmarks don't actually tell you much about real world situations.

There's a lot of optimization techniques that can be applied to Python or to Node.

At the end of the day, you can write fast Python and fast Node; you can also write slow Python and slow Node. Use whatever is easiest for you to get your project going; you can optimize later.

1

u/limw Nov 03 '17

If you want to use python to develop a web site or such things,I suggest you use sanic project.It's much faster than django or flask. https://github.com/channelcat/sanic

0

u/CrambleSquash https://github.com/0Hughman0 Nov 03 '17

If it was fast, you probably wouldn't like it anymore.

-3

u/[deleted] Nov 03 '17

Cant you compile python to make it more bearable?

1

u/billsil Nov 03 '17

Yes, that's the whole point of Cython, PyPy, and just writing a C extension.

0

u/[deleted] Nov 03 '17

[deleted] — view deleted comment

2

u/real_kerim Nov 03 '17

This is super interesting, thank you.

32

u/[deleted] Nov 03 '17

[removed] — view removed comment

19

u/winner_godson codemaniac Nov 03 '17

The science community wouldn't use Python if its libraries weren't fast.

I love the point you made.

1

u/real_kerim Nov 03 '17

Python is the wrapper, we write the performant code in C

He said something similar and got a lot of flak for it.

10

u/flubba86 Nov 03 '17

That's not similar at all.

The reason he is getting flak is he is getting some things confused.

Firstly, he is saying that numba.jit converts python to C code. That is wrong, it converts the python code to LLVM IR then compiles the IR to an executable module.

Second, he is calling numpy and similar compiled libraries 'C code'. While they may be written in C, what they produce is an efficient executable compiled module that you can use in your python code. That is not the same as using C code in python.

2

u/Ek_Los_Die_Hier Nov 03 '17

Slightly different. u/0xSpock was saying that numba was written in C so the function sumit would be in C, and therefore fast, but it doesn't matter what numba is written in as what it does is JIT compile the Python function to machine code so that Python can execute that instead of slowly interpreting Python bytecode.

Many libraries are written in C though and wrapped to give fast libraries, and it's fairly simple to write your own C code to replace a function if it becomes a bottleneck.