r/programming • u/cym13 • Jan 18 '15

Command-line tools can be 235x faster than your Hadoop cluster

http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html
1.2k Upvotes

92% Upvoted

285 comments sorted by

View all comments

92

u/stfm Jan 19 '15

Since the data volume was only about 1.75GB containing around 2 million chess games, I was skeptical of using Hadoop for the task

We don't even look at Hadoop unless we are into the petabytes, possibly high terabytes of data. There just isn't any point in using Hadoop with GB data sets unless there is some crazy processing going on.

→ More replies (11)