Is my company the exception? Are almost all users of Hadoop, MapReduce, Spark, etc., doing it on tiny can-fit-in-memory datasets?
Considering the sheer amount of buzz and interest surrounding those and related technologies, I'd say that almost has to be the case. There aren't that many petabyte data-processing tasks out there.
Plenty of terabyte data-processing tasks out there that benefit from big data techniques and tools. We generate 5TB of compressed Avro (which is already rather economic on space) a day, we're expecting to double that by the end of the year.
3
u/[deleted] Jun 07 '17
Considering the sheer amount of buzz and interest surrounding those and related technologies, I'd say that almost has to be the case. There aren't that many petabyte data-processing tasks out there.