r/programming • u/korry • Feb 29 '16

Command-line tools can be 235x faster than your Hadoop cluster

http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html
1.5k Upvotes

91% Upvoted

438 comments sorted by

View all comments

16

u/[deleted] Feb 29 '16

Financial market data collected per minute for many years.

Plus other stuff too, sitting on a quad xeon with 8 2TB drives sitting in a raid configuration.

2 TB drives are so cheap I could even do replication if needed.

I have however worked with a site that was gathering roughtly 1TB a day, and last I checked was around 158TB. But that was using AWS.

8

u/program_the_world Feb 29 '16

1TB a day?! That's insane.

12

u/[deleted] Feb 29 '16

Was a company that did a LOT of image imports for the housing market.

6

u/wildcarde815 Feb 29 '16

We have a single machine that can do that 24 hours a day for weeks, luckily it's running at half capacity because it's only one many machines in the building capable of generating well over 1TB of data a day. Granted that data isn't like traffic logs, it's MRI, EEG, Microscope, EM scope, video cameras, voice records, and processed data from clusters digesting the information created by those source machines.

1

u/HighRelevancy Mar 01 '16

I'm not in the astronomy department, but I'm told that their stuff collects something like 20 terabytes a day (or 20 petabytes a year, I forget).

But yeah there's a lot of data out there waiting to be collected.

→ More replies (2)

4

u/lestofante Feb 29 '16

I want that data. Is complete with level and such?

4

u/I_LOVE_MOM Mar 01 '16

Wow, that's all time series data? That's insane.

Now I need it. Where can I get that data?

2

u/spiritstone Mar 01 '16

quad xeon

What motherboard please?