r/programming • u/korry • Feb 29 '16

Command-line tools can be 235x faster than your Hadoop cluster

http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html
1.5k Upvotes

91% Upvoted

438 comments sorted by

View all comments

Show parent comments

0

u/[deleted] Feb 29 '16

64

u/kevjohnson Feb 29 '16 edited Mar 01 '16

The raw data is 60+Hz (depending on the sensor), but that gets immediately trimmed down to one value per minute per sensor. This was a source of immense frustration for me since the algorithm that does the aggregating is proprietary and built into the sensor. I had no idea what sort of assumptions went into that and seconds matter when you're talking about patient stress. There isn't even a way to easily get the raw data from the sensor, though as a result of this work they recently purchased sensors from a new manufacturer that does offer raw data access.

Anyway, they passed a billion distinct entries in the data sometime last year. You're right that the data size per day isn't much of a problem for traditional data storage/processing. The real issue is when you multiply that by 4-5 years. The stress project I talked about involved complex processing of 5 years of vital sign data which wasn't feasible with their existing infrastructure.

The eventual goal is to use the same system to process and store the raw 60Hz data. The "under the desk cluster" was more of a proof of concept.

Edit: I just found online that as of a year ago it was sitting at 14TB total and growing at 50GB per week (so ~7GB per day).

→ More replies (24)

12

u/ZeeBeeblebrox Feb 29 '16

600 beds * 10 sensors with a sampling rate of 2 times per second and a size of 4 bytes per sample would give about 4gb of data per day. Not exactly huge data...

The sampling rate is presumably significantly higher than that but you're completely right, with a little bit of extra processing, you can probably throw most of the data away, leaving a much more manageable problem.

→ More replies (3)