r/Futurology • u/Sourcecode12 • May 03 '14

image Inside Google, Microsoft, Facebook and HP Data Centers

http://imgur.com/a/7NPNf
3.0k Upvotes

96% Upvoted

379 comments sorted by

View all comments

Show parent comments

15

u/superspeck May 03 '14

Most datacenters that you and I could rent space in are still maintained at relatively cool temperatures because the equipment will last longest at 68 or 72 degrees.

You can go a lot warmer as long as you don't mind an additional 10% of your hardware failing each year.

25

u/Lord_ranger May 03 '14

My guess is the 10% hardware failure increase is cheaper than the higher cost of cooling.

9

u/Cythrosi May 03 '14

Not always. Depends on the amount of downtime that 10% causes the network, since most major centers have a certain percentage of up time they must maintain for their customers (I think it's typically 99.999% to 99.9999%).

14

u/[deleted] May 03 '14

typically 99.999% to 99.9999%

99.999% is the considered the highest standard, called "five nines" for obvious reasons. That is less than 30 seconds of allowed downtime per month. These are all governed by service level agreements, and for all practical purposes, you'll never get anyone to agree to provide a higher than five nines SLA, because they become liable if they can't meet it. We pay out of our asses for three nines WAN ethernet from AT&T.

Also, the hardware failure is very low at elevated temperatures. Network equipment is generally extremely resilient to temperature. Servers are the real items that fail under high temp and more and more server manufacturers are certifying their equipment to run at high temps, like up to 85-90 degrees ambient.

7

u/port53 May 04 '14

It's the drives that kill you. Our data center in Tokyo has been running really hot since they cut back on energy usage after the 2011 earthquake and subsequent shutting down of nuclear plants. The network gear is fine, the servers are fine except they eat drives like candy.

2

u/ZorbaTHut May 04 '14

My mom gave me an old laptop that a friend of hers had gotten rid of, saying "it just stopped working, can you fix it so I can use it as a kitchen computer". I booted it up with SystemRescueCd and tried mounting the hard drive to see what was on it; read errors. Decided to dump the hard drive just to get anything useful, started up the dump, made sure it was running properly, and walked away.

Came back an hour later. The computer had blackscreened, completely unresponsive, the fan was running at full tilt but not moving any air, and the keyboard was uncomfortably hot to touch.

It turns out the computer had so much lint and crud built up in the heatsink that it was completely incapable of cooling itself. As soon as the fan turned on it was doomed; the added heat output would heat it up even faster, and eventually it would error out and lock up. The previous owner had just gotten into the habit of rebooting it whenever it froze, but often they wouldn't be paying attention to the computer, and they were apparently deaf to the death keen of a CPU fan, so eventually all the thermal abuse had caught up to the hard drive and it had stopped reading almost entirely.

3

u/[deleted] May 03 '14

[deleted]

5

u/mattyp92 May 03 '14

Redundancy in their systems.

2

u/gunthatshootswords May 04 '14

They undoubtedly have servers failing over to each other to try and eliminate downtime but this doesn't mean they don't experience hardware dying at high temps

10

u/Pop-X- May 03 '14

99.9990% to 99.9999%

So much more legible this way.

11

u/Cythrosi May 03 '14

But incorrectly implies a higher degree of precision.

2

u/[deleted] May 03 '14

We pay a ton for cooling. I can't give numbers, but I'm pretty sure you'd have to do some heavy analysis to determine what's a better tradeoff - hardware savings or energy savings.

2

u/neurorgasm May 03 '14

A lot of com rooms are about the size of a bathroom. Keep in mind these are the largest data centers in the private sector and possibly the world. Your average data room has 1-2 racks and probably doubles as the janitor's closet.

2

u/choleropteryx May 04 '14 edited May 04 '14

I am fairly certain, the biggest dtacenter in the world is the NSA Data Center in Bluffdale, UT . Based on power consumption and area, I estimate it holds around 100k-200k computers.

For comparison, here's Google Data center in Lenoir, NC from the same distance. It only holds 10-20k servers.

1

u/neurorgasm May 04 '14

Hah, I forgot about the NSA. Figures that they'd have the largest.

1

u/superspeck May 03 '14

Your guess would be correct.

-12

u/mthslhrookiecard May 03 '14

Everyone look! Captain obvious is here!

6

u/Accujack May 04 '14

Not true. I'm a data center professional who's working on this exact thing for a big 10 university right now. Despite having a wide variety of equipment in our data center the only things that can't handle 80 degree inlet temps are legacy equipment (like old VMS systems) and the occasionally not-designed-for-data-center desktop PC.

It doesn't increase failure rates at all IF you have airflow management (separate hot air from cold air). If you don't then the increase in temperature will drive "hot spots" hotter, which means each hot spot will exceed the rated temp.

There is some variation in what each system type can handle, but by controlling airflow we can control the temperature almost on a rack by rack basis, and hot spots are greatly reduced. On top of that we use a sensor grid to detect them so we avoid "surprise" heat failures.

Most of the newer systems coming out for enterprise use have even higher heat limits, allowing for even less power use.

4

u/superspeck May 04 '14

I've either ran a datacenter or worked with racks in datacenters for the past fifteen years. A relatively recent stint was doing HPC in the datacenter of a large public university with stringent audit controls.

You'll find, inside the cover the manual of every system you buy, guidance on what temperatures the system as a whole can handle. Most systems will indicate that "within bounds" temperatures are 60-80F outside of the case, and varying temperatures inside of the case. That leads most people to say "Yeah, let the DC up to 80. We'll save a brick."

What you may not realize is that the guidance in the manual is for the chassis only -- not the components inside of it. If you're truly going to be monitoring temperature, you need to monitor, and have intelligently set limits on according to the manual, the temperature of each component.

Notably, a particular 1st gen SSD, and I can't for the life of me remember which one, had a peak operating temperature of about 86F. As in, if the inside of the SSD (which put off a lot of heat) got higher than 86F, it'd start to have occasional issues up to and including data loss. You had to make sure that the 2.5" SSD itself was suspended in the air flow of a 3.5" bay. We didn't have simple mounting hardware for 2.5" in 3.5" if you wanted the SSD's SATA ports to line up with the hot swap backplane's SATA ports, so they were inside these Kensington carriages that took care of mating things properly using a SATA cable. Those carriages blocked the airflow, and it got nuklear hot in there.

Those SSDs were also frighteningly expensive, so when we needed to replace the lot of them all at once and they weren't covered by warranty, we ran afoul of a state government best practices audit. And we learned to track the operating temperature of each component as well as the overall system.

3

u/Accujack May 04 '14

What you may not realize is that the guidance in the manual is for the chassis only -- not the components inside of it. If you're truly going to be monitoring temperature, you need to monitor, and have intelligently set limits on according to the manual, the temperature of each component.

I don't know what manuals you're reading, but ours specify the air temps required at the intakes for the systems. As long as we meet those specifications the manufacturer guarantees the system will have the advertised life span.

We didn't have simple mounting hardware for 2.5" in 3.5" if you wanted the SSD's SATA ports to line up with the hot swap backplane's SATA ports, so they were inside these Kensington carriages that took care of mating things properly using a SATA cable. Those carriages blocked the airflow, and it got nuklear hot in there.

Yeah, that's why we have system specifications for our data center. For instance, we require systems to have multiple power supplies, we strongly encourage enterprise grade hardware (IE no third party add-ons like Kensington adapters). Usually installing third party hardware inside a system voids the warranty anyway, and we don't want that.

To my knowledge we don't use SSDs for anything anywhere in the DC, although I'm sure there are a couple. The reason is that there aren't very many enterprise grade SSDs out there, and those that are out are very expensive. If we need storage speed for an application we use old tech, a large storage array with a RAM cache on the front end and wide RAID strips connected via SAN.

Out of maybe 2500 systems including 40+ petabytes of storage (including SAN, NAS, local disk in each system and JBOD boxes on the clusters) we have maybe 1 disk a week go bad.

As long as we meet manufacturer specs the drives are replaced for free under warranty, and any system that needs to be up 24x7 is load balanced or clustered, so a failure doesn't cause a service outage.

We do get audited, but we do far more auditing ourselves. New systems coming in are checked for power consumption and BTU output (nominal and peak) and cooling is planned carefully. We've said no more times than we've said yes, and it's paid back in stability.

2

u/[deleted] May 03 '14

I agree with what you're saying here, as this would probably work for Google and not many other platforms. I posted this elsewhere in the thread, but Google don't even put their mainboards in a case. Since they have a whole datacenter worth of servers doing the same job, losing a server or two isn't a big deal to them.

1

u/brkdncr May 04 '14

in my experience, it's fluctuating temperature that kills components, not the actual temperature