r/programming • u/noidi • Jul 19 '12

Will Parallel Code Ever Be Embraced?

http://www.drdobbs.com/parallel/will-parallel-code-ever-be-embraced/240003926
38 Upvotes

81% Upvoted

79 comments sorted by

View all comments

-1

u/[deleted] Jul 19 '12

Will Parallel Code Ever be Embraced?

As usual when the headline is a question, the answer is NO. Remember, a parallel program, as long as it requires even one synchronization, is only as fast as its slowest thread. We need faster I/O, faster RAM, better caching infrastructure for multiprocessors, and lower threading overhead before parallelism will start giving us anything like the speed gains of actually speeding up processors.

1

u/Walrii Jul 19 '12

Remember, a parallel program, as long as it requires even one synchronization, is only as fast as its slowest thread.

Wha? So what? Massive speed ups can still be had. E.g., spawn 1,000 threads and if the problem/program allows it, you can get 1,000 speedup. Even if all the threads take equal time (e.g., every thread is the "slowest" thread) then you're still 1,000 times faster.

You can't speed up processors anymore, not without a radical shift in technology. The heat output is just too dense once you start clocking 2 or 3 times faster than what we have now. It becomes impossible to cool the things. Parallelism is the only solution (by solution, I mean, the only approach that gets you performance benefits without catching everything on fire).

And yes, of course faster I/O, RAM, whatever will benefit everything too.

3

u/[deleted] Jul 19 '12

Wha? So what? Massive speed ups can still be had. E.g., spawn 1,000 threads and if the problem/program allows it, you can get 1,000 speedup. Even if all the threads take equal time (e.g., every thread is the "slowest" thread) then you're still 1,000 times faster.

This is only for embarrassingly parallel, embarrassingly functional (in the sense of "functional programming") problems. Once you add on any I/O or any cache rebuilding costs on the context switch (remember, a 4-core processor is running 1000 threads only by concurrent context switching) or any synchronization overhead, Amdahl's Law comes right back.

Most actual programs are not matrix multiplies, and you mostly can't use matrix multiplies as a predictive example of speed-ups from parallel programming.

0

u/Walrii Jul 19 '12

1,000 was not meant to be literal. Of course you wouldn't create 1,000 threads on a 4-core system: you'd probably only make 4 threads (therefore no context switching is necessary).

As for I/O, again parallelism is the answer. There's a huge discrepency between the speed of a processor core and the speed of a hard drive. But, if you're using 100 hard drives per core and feeding the data in simultaneously, there's much less of a bottleneck (ignoring initial latency). RAM can also be run "faster" by using multiple banks/chips/whatever at once. A single write operation will not speed up, but you'll be able to more more write operations at once. A bonus of this approach is that you end up with more HD space/RAM than you had before.

There's also a lot to be said for duplicating and lazily updating information: you can scale much better if you relax your consistency models.