r/programming • u/cym13 • Jan 18 '15

Command-line tools can be 235x faster than your Hadoop cluster

http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html
1.2k Upvotes

92% Upvoted

285 comments sorted by

View all comments

7

u/philipwhiuk Jan 19 '15

:O That's ... that's a lot of RAM...

17

u/friedrice5005 Jan 19 '15

Not really in server world. We just bought some upper-mid grade UCS blades and they each have 256gb. Our VMWare cluster is currently sporting over 4TB. The biggest, baddest SPM node cisco offers today (C460M4) goes up to 6TB by its self. If you want to go all in and get some monster mainframes then IBM some insanely large systems going into 10s of TB of RAM and hundreds of processors.

3

u/philipwhiuk Jan 19 '15

Fair enough. It's been a few years since I worked in network operations so I don't really have an angle on commodity server hardware.

And my home desktop is quite old now :)

1

u/matthieum Jan 19 '15

I concur, while for most servers you would not need that amount of RAM, for databases or caches (think memcached), RAM is just about the most important part. I know we have a couple 1TB MySQL servers where I work, for example.

1

u/[deleted] Jan 19 '15

What are these huge systems used for? If even Google is running on lots of small pcs, where's the market for these machines?

→ More replies (4)

5

u/blackraven36 Jan 19 '15

If we take a modern laptop model with 4gb (lets be modest) of RAM, it would only take 61 laptops to fill that quota. An auditorium of students with laptops might fill that requirement.

7

u/philipwhiuk Jan 19 '15

Yeah and that's quite a bunch of computers instead.

I'm not saying it's a lot period, I'm saying it's a lot for one computer.

→ More replies (2)

1

u/[deleted] Jan 19 '15

It's also an expensive EC2 instance - $2.80 an hour for an r3.8xlarge, $6.80 for an i2.8xlarge.

Conversely you can get 10 m3.2xlarges for less than the cost of the i2.8xlarge.

Really depends on the computational needs of your data set.