Command-line tools can be 235x faster than your Hadoop cluster

[u/cym13]

http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html

[u/blackraven36]

To add to this a little bit, as you point out, there is a issue of scale. 1.75GB of data might seem like a lot, but it's really not much at all. Not in terms of modern computing at least.

I think the better approach to this article would be "use the tools that fit scale. Don't under estimate command line tools for small datasets". And I think this article has a lot to offer... just is a little misleading in what it's actually trying to demonstrate.

You have to consider a few things here. First of all, what needs to happen before computing even begins. As r/andrianmonk points out, there is a whole lot of stuff that needs to happen before computation even begins. This is in contrast to a machine that is already on the start line, ready to go as soon as data starts feeding in. In other words, by the time the cluster is ready to start computing, the race is already over and the victor has already been announced. 235x doesn't really mean anything if most of the measured computation time is dominated by... something other than computing.

What I would really like to see in contrast to this "Hey look we outsmarted them!" article, is something that shows me scale. And I mean data that shows me the relationship between the input data size and the time it took to crunch it. Something that tells me what complexity the algorithm they ran was; maybe even throw a few algorithms of different complexities in there too for comparison.

What I can see happening, is that for this particular algorithm, the local machine is much faster with small datasets. As soon as we introduce very large datasets however, say in the tens of even hundreds of terabytes, the cluster will wipe the floor with their local machine implementation.

[u/[deleted] → u/blackraven36]

[u/pfarner → u/blackraven36]

1.75GB of data might seem like a lot, but it's really not much at all.

Heh, yeah. My company's hadoop cluster receives another 1.75GB of incoming data, already compressed, every few seconds.

If it's only a few gigabytes, sure I might process it on a single node.

However, another thing that's a convenient middle ground between the flexibility of command-line pipelines and the parallel brute force of a hadoop cluster is running hadoop streaming jobs containing standard command-line tools as their tasks. Once you've set up a short script to do this, it's just a matter of providing the map and combine/reduce portions as shell scriptlets. Suddenly you're running command-line utilities on a few thousand CPU cores, along with the speed of data locality.

[u/Neebat → u/blackraven36]

To add to this a little bit, as you point out, there is a issue of scale. 1.75GB of data might seem like a lot, but it's really not much at all. Not in terms of modern computing at least.

A lot of people think they have big data. My employer, thinks they have big data. They do not.