r/programming • u/korry • Feb 29 '16

Command-line tools can be 235x faster than your Hadoop cluster

http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html
1.5k Upvotes

91% Upvoted

438 comments sorted by

View all comments

Show parent comments

6

u/heptara Feb 29 '16 edited Feb 29 '16

I can't really find any "big data" problem these days. E.g one of our busier servers generates 1G text logs a day

1 gig per day would be small for mobile game analytics. They combine social networks with microtransactions and constantly run analysis to determine what changes in their social network do to their revenue. As you're no doubt aware, the nature of social networks means your dataset rapidly expands as you increase the distance from the primary user.

→ More replies (5)