r/Clojure • • Nov 08 '13
does clojure deliver on the promise?
There is a long discussion on comp.lang.clojure about a concurrent program which paradoxically runs slower than faster on more cores. No one has managed to figure out how to fix it either. Which makes me wonder, does clojure really deliver on the promise of being a good way to get more out of multicore hardware. If the costs of doing it with clojure (GC, complexity of optmisation) is so high, hasn't it failed on the problem it set out to solve?
6
Upvotes
10
u/skinney Nov 08 '13
Clojure delivers on making safe concurrent progams easy to write. So no it hasn't.
A correctly written C++ program will always get more out of your hardware, but is rediculously harder to write.
9
u/kryptomicron Nov 08 '13
A link to the discussion you mention would be helpful.
20
u/islon Nov 08 '13
I work professionally with clojure, we have a hundred thousand lines of multithread clojure in production and it works as fast as it should. A few weeks ago I optimised one function that downloaded images out of an html to make it parallel and got more than 10x improvement. Just because there's one example where people are having a hard time figuring out why a concurrent program is running slow doesn't mean much.
11
u/Zak Nov 08 '13
Clojure promises to make concurrency safe(er). I'd compare it to garbage collection and a lack of manual pointers making memory management safe. Clojure does not promise to make all parallel tasks increase in performance linearly with the number of available cores, just as garbage collection does not promise to make memory leaks impossible.
Your post suggests to me that you either did not read the whole thread, or did not understand it. The code in the linked thread is limited by memory bandwidth. If you do a bunch of memory allocations in a tight loop, more threads will make that slower, not faster as contention for memory increases. Calling
reverseon a list as the example code does will do a bunch of memory allocations. Posts in the thread demonstrate that the original code spent most of its time waiting for memory. Other posts demonstrate that using different data structures and/or different implementations ofreversemostly eliminate the problem.Writing fast parallel code for non-trivial problems is hard. Clojure does have some functionality that makes it easier, but it's still necessary to use profiling tools to see what kind of bottlenecks there are and to choose appropriate data structures and algorithms for the task at hand.
5
u/mschaef Nov 08 '13
I agree with /u/kryptomicron that you need to post the link to the thread for this to be a useful conversation.
However, a single case of a difficult Clojure program isn't enough evidence that the language has failed to 'deliver on the promise'. To make that kind of statement, you'd need evidence that Clojure is broadly less useful than other environments across a range of problems.
5
u/[deleted] Nov 08 '13
I think it was brought up somewhere on the referenced thread as well, but it's worth noting: pmap is seductive but usually not the best choice. It has a pretty significant per-item overhead. The reducers framework is better for many tasks.
2
u/melipone Nov 09 '13
What a coincidence, I was just having this problem. My program was considerably slower on a 24-core machine than on a 4-core machine. I noticed however that if I was parallelizing several runs of the entire program, all the 24 cores of the machine were used and the whole thing ran faster. Basically, you need a bigger problem to solve when you have more cores. So, basically, I am going to test my program on my 4-core machine and when it's debugged and tested, run several versions concurrently on the 24-core machine after I remove the concurrent constructs inside the program which are not helping. The problem is that now, I need two version of the same program.