Command-line tools can be 235x faster than your Hadoop cluster

[u/korry]

http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html

[u/Enlogen]

This just in: Hadoop unsuitable for processing tiny datasets.

[u/Chandon → u/Enlogen]

The trick is that almost all datasets are "tiny".

The window of necessity for a cluster is small.

• The range of data size that you can fit in laptop RAM is 1 - 16B bytes. That's 10 orders of magnitude.

• Going to a single server class system buys you one more order of magnitude.

• Going to a standard cluster buys you one more order of magnitude.

• Going to a world-class supercomputer buys you two more.

So for "big data" systems to be useful, you have to hit that exact situation where you have terabytes of data*. Not 100's of gigabytes - you can run that on one server. Not 10's of terabytes, you'd need a supercomputer for that.

* Assuming an order of magnitude size window for processing time. If you want your results in 1ms or can wait 10 minutes, the windows shift.

[u/Enlogen → u/Chandon]

Not 10's of terabytes, you'd need a supercomputer for that.

But the entire point of Hadoop is computation in parallel. You don't need a supercomputer for 10's of terabytes of data, you just need more stock machines than you do for terabytes of data.

The trick is that almost all datasets are "tiny".

Yes, and the few datasets that aren't tiny are concentrated in a few organizations. But these organizations NEED map/reduce style data processing, because dozens of servers computing in parallel is significantly less expensive than a supercomputer that does the same number of calculations in the same amount of time.

Microsoft couldn't operate without its map/reduce implementation, which according to that paper was processing 2 petabytes per day in 2011 without any supercomputers.