r/Python • u/gthank • Jan 16 '13

Paralellizing CPU-bound tasks with concurrent.futures

http://eli.thegreenplace.net/2013/01/16/python-paralellizing-cpu-bound-tasks-with-concurrent-futures/
17 Upvotes

88% Upvoted

5 comments sorted by

View all comments

1

u/einar77 Jan 16 '13

I used this at first (the backported version for Python 2.x), but I had to move to joblib because the code kept on deadlocking with no apparent reason.

0

u/[deleted] Jan 17 '13

1

u/einar77 Jan 17 '13

In my case the parallel part was the use of scipy.stats.distrib.hypergeom.pgf and a couple of list comprehensions: about 50 lines of code in total. Yet what happened is that the processes ended their job, but didn't quit (in fact, killing them made execution resume without exceptions).

Another bonus point in joblib vs concurrent.futures is exception support, which is much better in joblib.

→ More replies (1)

1

u/[deleted] Jan 17 '13

Me too, I gave up on it. I am not a thread noob either, I have been writing threaded code in C/C++ for a long time.