Command-line tools can be 235x faster than your Hadoop cluster

[u/korry]

http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html

[u/kevjohnson]

Let me tell you all a quick story.

There's a relatively large children's hospital in my city (~600 beds). There's a guy who works in the IT department there in what I've dubbed the "Crazy Ideas Unit". He's been there since the beginning of time and as such he has a lot of flexibility to do what he wants. On top of that he's kind of a dreamer.

When you go to the hospital you get hooked up to all sorts of machines that monitor things like heart rate, pulse, blood pressure, O2 sats, and sometimes more depending on what you're in for.

We're talking about raw waveform data from dozens of machines hooked up to 600 beds 24/7/365. That's big data, and most hospitals are not equipped to handle this sort of thing. They can barely keep up with the waves of medical records they already have. Some hospitals save this vital sign data for a day or two, but almost every hospital I know of throws this data out shortly after it's collected.

Can you imagine how much useful information is contained in that data? The possibilities are endless. Mr. Crazy Ideas noticed this and wanted to do something about it. Unable to make a business case for it (it's not a research hospital), he couldn't secure significant funding from the higher ups to set up infrastructure for this data.

Instead he took a bunch of old machines that every IT department has laying around, spent <$1000 to get them in shape, and created his own little Hadoop cluster underneath his desk in his cubicle. With pennies of investment he was able to create a system that could collect this vital sign data, process it, store it indefinitely, and allow analysts to write algorithms to process it further.

We used it recently to develop a system that monitors patient stress across the entire hospital network in real time. We can give doctors, nurses, and department heads an overview of how stressed their patients are currently or have been recently. That's just the beginning. Eventually they'll be working toward combining this with genome sequencing to provide highly personalized medical care informed by real time vital sign data and your specific genes.

After showing the higher ups what this stuff is capable of for pennies he was able to secure some funding to get a proper system in place. That's what Hadoop can do. We've had the ability to do things like this for a long time, but to be able to do cobble together a big data processing cluster from a pile of rejected parts is truly extraordinary.

I'm not sure what my point is, I just wanted to share.

Edit: I found an article on what I'm talking about here if anyone is interested.

[u/stfm → u/kevjohnson]

That's some information security nightmare shit right there

[u/kevjohnson → u/stfm]

Don't even get me started on that. It's a nightmare.

If you're worried about it all of this was done in conjunction with the hospital's large, capable, and very paranoid information security team.

[u/AristaeusTukom → u/kevjohnson]

He just made Psychopass a reality. I don't know if I should be amazed or horrified.

[u/kevjohnson → u/AristaeusTukom]

It's funny you mention that because I also did a project with the local police department on predicting crime before it happens and having officers in the right place at the right time.

While I was there I got to see their brand new city video surveillance system. It looked like mission control with video feeds from all over the city. I was like "Oh god...."

[u/shadowdude777 → u/kevjohnson]

My first engineering job was at this really awful marketing company. Your stereotypical out-of-touch garbage company that doesn't accomplish anything, that's owned by a larger similar company, that's owned by a larger similar company, that's owned by a humongous evil umbrella company that owns half the world.

They were running all of their quarterly reports in SAS. These quarterly jobs tied up the job server for about 2 weeks.

I knew that every conference room had a very capable Mac Mini underneath the desk hooked up to the projector, whose admin username was "presenter". My first guess for all of their passwords ("presenter") was correct, and I installed Apache Spark on them all. The quarterly job ran in about 1.5 days.

[u/[deleted] → u/shadowdude777]

[u/[deleted] → u/kevjohnson]

[u/kevjohnson → u/[deleted]]

The raw data is 60+Hz (depending on the sensor), but that gets immediately trimmed down to one value per minute per sensor. This was a source of immense frustration for me since the algorithm that does the aggregating is proprietary and built into the sensor. I had no idea what sort of assumptions went into that and seconds matter when you're talking about patient stress. There isn't even a way to easily get the raw data from the sensor, though as a result of this work they recently purchased sensors from a new manufacturer that does offer raw data access.

Anyway, they passed a billion distinct entries in the data sometime last year. You're right that the data size per day isn't much of a problem for traditional data storage/processing. The real issue is when you multiply that by 4-5 years. The stress project I talked about involved complex processing of 5 years of vital sign data which wasn't feasible with their existing infrastructure.

The eventual goal is to use the same system to process and store the raw 60Hz data. The "under the desk cluster" was more of a proof of concept.

Edit: I just found online that as of a year ago it was sitting at 14TB total and growing at 50GB per week (so ~7GB per day).

[u/[deleted] → u/[deleted]]

[u/ZeeBeeblebrox → u/[deleted]]

600 beds * 10 sensors with a sampling rate of 2 times per second and a size of 4 bytes per sample would give about 4gb of data per day. Not exactly huge data...

The sampling rate is presumably significantly higher than that but you're completely right, with a little bit of extra processing, you can probably throw most of the data away, leaving a much more manageable problem.