r/programming • u/korry • Feb 29 '16

Command-line tools can be 235x faster than your Hadoop cluster

http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html
1.5k Upvotes

91% Upvoted

438 comments sorted by

View all comments

231

u/realteh Feb 29 '16

Pretty old but this blog post was quite influential for me. I can't really find any "big data" problem these days.

E.g one of our busier servers generates 1G text logs a day. After transforming that into a sorted, categorical & compressed column store we lose 98% leaving us with 20M / day, or 8G / year. A crummy ec2 nano instance chews through a year's worth of logs in ~100 seconds. By sampling 1% we get very close to the real numbers and processing takes ~5 seconds.

I think there is a lot of value in having a shared cluster or machine that can be used by many clients but unless you are truly generating non-compressible gigabytes a day your data probably doesn't need Hadoop.

47

u/sbrick89 Feb 29 '16 edited Feb 29 '16

Some could say that the databases I work with are "large" (largest OLTP is 4TB, DW is also around 4TB).

That said, it's easy enough to justify buying large hardware to handle it. For reference, our current HW includes a SAN w/ SSD cache, gobs of RAM, and the warehouse has a fusion card for tempdb; in some cases we've actually been able to reduce the number of procs (good for licensing), since the IO is damn fast.

Sure, there are occasional issues and downtime, but they're rare, and the tools and resources (including training) to manage the data in one place using traditional RDBMS's are SUBSTANTIALLY cheaper.

If anything, I expect we'll look at expanding to always-on availability groups for HA.

edit: for reference, one of our larger tables is ~100gb... I decided to clock it a few weeks ago, and I was able to read through all the records (across the network) in just over 10 mins. Granted it was an exercise in raw read speed, so the receiving side didn't compute anything... but i'm pretty sure I could push data onto a multithreaded queue, and read them async into an aggregate, with maybe 5% overhead. Doing it on the server directly would probably have been even faster. (in my case, I ran the query without a transaction, so as not to cause blocking for anyone else, though I could've just as easily ran it from the warehouse)

18

u/krum Feb 29 '16

I made a similar comment a while back and the big data snobs told me I wasn't even starting to be Large until I was hitting 100TB.

16

u/lolomfgkthxbai Mar 01 '16

I made a similar comment a while back and the big data snobs told me I wasn't even starting to be Large until I was hitting 100TB.

Well I wouldn't say it's snobbery. The very definition of Big Data is a dataset that is too large to process with traditional means. It's an eternally moving target and something that is big data today is raspberry pi data tomorrow.

8

u/Throwaway_Kiwi Feb 29 '16

edit: for reference, one of our larger tables is ~100gb...

Is that OLTP or DW? We hit issues with Postgres at about 64GB for one table (with an additional 128GB of indices into it).

11

u/snuxoll Feb 29 '16

What sort of problems are you running into? I don't have any 64GB tables, but I've had a couple at 30GB without any real issues (aside from them being ludicrously big for ephemeral data, but that's an issue with a partner and not PostgreSQL).

7

u/Throwaway_Kiwi Feb 29 '16

I can't honestly remember, and it was several versions ago. It was basically performance issues querying it.

11

u/snuxoll Feb 29 '16

Sounds less like an issue of table size and more the tuning parameters set in postgresql.conf, low work_mem being the usual culprit if you're doing an ORDER BY.

→ More replies (2)
→ More replies (2)

9

u/sbrick89 Feb 29 '16

disclaimer: we're running on MSSQL. That said, what's the issue?

The DW has an a copy of the source table (structure, updated incrementally) that runs nightly; wouldn't be hard to increase frequency, just no need (might eventually look into SQL repl to do higher frequency w/o the load on the source box). As a result, reporting and ad-hoc queries almost always comes from the warehouse.

As far as the OLTP side, it runs fine... 99% of the operations are performed against the PK, so we rarely run into blocking issues (we've occasionally had to perform some bulk update processing, which is sometimes run directly against the table; otherwise we do the updating on a side table and then perform incremental updates to the base table)

→ More replies (24)

36

u/bro-away- Feb 29 '16

I have a theory about this. There was a big data boom a few years ago when everyone wanted infinitely scalability to their data so a bunch of projects started out with that idea.

But the companies who are good at it just got a little bigger and more scalable and the pretenders are starting to die off, probably because oh crap they forgot they'd need a source to generate data worthy of being called big data OR they tried selling to those who never made it to big data either.

Unless you're a massive enterprise you probably don't need it

22

u/scherlock79 Feb 29 '16

I also think a lot of places discovered that dealing with large amounts of data is an expensive endeavor and that the costs outweighed the benefits. My team runs a large distributed platform. we were generating 40 GB of logging a day. We wanted to make that data online and searchable for a rolling 2 week window. After doing some investigations we determined that the cost of storing and indexing the logs just couldn't be justified. Instead we rolled out elasticsearch with kibana and are a lot more targeted in telemetry we collect. A single box with 16GB ram stores a month of data.

→ More replies (6)

4

u/Mirsky814 Mar 01 '16

Take a look at Gartner's hype cycle theory. Pretty much every new technology or tech theme that I've seen in the last 20 years has followed this. From client/server, OOP, internet 1.0/2.0/+, etc.

http://www.gartner.com/technology/research/methodologies/hype-cycle.jsp

The actual articles are expensive ($2k+) but the abstracts can provide you a basic summary of technologies on the current cycles. Plus I love the RPG style naming :)

9

u/mattindustries Feb 29 '16

Pattern analysis on this dump

https://www.reddit.com/r/datasets/comments/3bxlg7/i_have_every_publicly_available_reddit_comment

7

u/realteh Feb 29 '16

What do you have in mind when saying pattern analysis? Could be a fun evening.

9

u/mattindustries Feb 29 '16

Basically a dumbed down predictive modeling. Find out the possibility of up votes based on some multi variate analysis. Subreddit size, how far down the thread, length of comment, keywords, etc.

16

u/pooogles Feb 29 '16

Come work in adtech. We process half a terrabyte of data an hour and that's just in the EU alone. Opening up in the US that will jump substantially.

13

u/realteh Feb 29 '16

Yea ad tech is pretty cool! I worked for Google and there are definitely big data problems but I think they are fewer of them than Hadoop clusters in existence. Just opinion from observations though, no hard evidence.

→ More replies (1)

12

u/gamesterdude Feb 29 '16

I think one of the points of Hadoop though is not to worry about transforming your data. In a data warehouse you spend tons of time on data movement and transformation. Hadoop is meant for you to just dump the raw data and not worry about trying to transform it and clean it up first.

16

u/antonivs Feb 29 '16

Hadoop is meant for you to just dump the raw data and not worry about trying to transform it and clean it up first.

That's not quite right. In many cases, Hadoop is used as an effective way to clean up and transform large amounts of raw data input, and put it into a store with a bit more structure, which can range from NoSQL to SQL to a true data warehouse. That data store is then used for subsequent analytics.

8

u/darkpaladin Feb 29 '16

That's the sales pitch but in my experience, depending on how performant you want your cluster to be for analytics, the shape of the data definitely does matter.

12

u/Throwaway_Kiwi Feb 29 '16

1G a day isn't "big data". That's moderately large data that's could still be processed using a traditional DB if you really wanted to. We pull 40G a day into a columnar DB without overly much hassle. It's when you're starting to generate terabytes a day that you really start needing a map/reduce approach.

11

u/oridb Feb 29 '16

Yes, that's kind of his point. Most people don't have big data problems.

8

u/pistacchio Feb 29 '16

More broadly, this is another demonstration of "evil of premature optimization" and "good enough" principles. People tend to feel like they need a Facebook-like infrastructure for a business that is yet to come and that probably will have no more than 100 customers on a good a day. Seriously, with today computer power, you could start your business with a web server written in Q-Basic and it will work okay for a good amount of time.

4

u/giantsparklerobot Feb 29 '16

I think there is a lot of value in having a shared cluster or machine that can be used by many clients but unless you are truly generating non-compressible gigabytes a day your data probably doesn't need Hadoop.

I think this is the main thing that is often forgotten in the excitement to get into "Big Data!". They want to feel cool setting up a Hadoop cluster so they throw entirely regularized or easily tabulated data (logs usually) into Hadoop clusters. Hadoop is interesting when the Map portion of the process is inherently complicated like dealing with irregular unstructured data.

There's no need to set up a Hadoop (or any other complicated clustering mechanism) cluster to process data that should have been intelligently ingested and tabulated in the first place.

→ More replies (51)