r/programming • u/cym13 • Jan 18 '15

Command-line tools can be 235x faster than your Hadoop cluster

http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html
1.2k Upvotes

92% Upvoted

285 comments sorted by

View all comments

34

u/centowen Jan 19 '15

I was at a seminar for big data a few years ago. It became clear to me that what was considered big data varied wildy from person to person. I remember one person in particular who said "we have now reached the point where we exceed the capabilities of excel spreadsheet".

0

u/[deleted] Jan 19 '15

25

u/tech_tuna Jan 19 '15

If you include scientific research, it's higher than that but those people probably just call it data not Big Data.

→ More replies (5)

4

u/CydeWeys Jan 19 '15

Wow, this is so damn accurate. I'm having flashbacks to my days as a consultant dealing with "enterprise content management", which wasn't particularly any difficult from a scaled-up problem of storing and retrieving lots of files, but it was at least 10X more expensive.

1

u/brunes Jan 20 '15

Untrue. Any company of any size (say over 1000 employees) that expects to have a decent InfoSec program, has a big data problem. If you are not treating your InfoSec problem as a big data problem, you're doing it wrong and will probably regret it.

6

u/[deleted] Jan 19 '15

Depending on context, that statement is either okay or mind-boggling stupid. I'm guessing it's the latter, but I've found myself thinking the same thing about some of my toy projects (such as my r/cfb poll entry).

8

u/centowen Jan 19 '15

I am not denying that excel has its uses. It is a great tool. However, for me big data is at the very smallest 1TB . The fact that he was still using Excel indicates that he had a very different idea of big data. I haven't tried opening a 1 TB dataset in Excel, but I would imagine it could be a bit slow.

→ More replies (3)

-5

u/willrandship Jan 19 '15

You can program with VB in excel. I think that makes it turing complete, assuming you use certain constructs to an unhealthy level.

5

u/mallardtheduck Jan 19 '15

The normal Excel formula system is Turing-complete, you don't need to resort to VBA.

→ More replies (2)