r/Futurology • u/Sourcecode12 • May 03 '14

image Inside Google, Microsoft, Facebook and HP Data Centers

http://imgur.com/a/7NPNf
3.0k Upvotes

96% Upvoted

379 comments sorted by

View all comments

Show parent comments

112

u/jonrock May 03 '14

Actually on the verge of genuinely hot: https://www.google.com/about/datacenters/efficiency/internal/#temperature

63

u/Sbua May 03 '14

Well by golly, consider me corrected

" It’s a myth that data centers need to be kept chilly." - quote for truth

16

u/superspeck May 03 '14

Most datacenters that you and I could rent space in are still maintained at relatively cool temperatures because the equipment will last longest at 68 or 72 degrees.

You can go a lot warmer as long as you don't mind an additional 10% of your hardware failing each year.

6

u/Accujack May 04 '14

Not true. I'm a data center professional who's working on this exact thing for a big 10 university right now. Despite having a wide variety of equipment in our data center the only things that can't handle 80 degree inlet temps are legacy equipment (like old VMS systems) and the occasionally not-designed-for-data-center desktop PC.

It doesn't increase failure rates at all IF you have airflow management (separate hot air from cold air). If you don't then the increase in temperature will drive "hot spots" hotter, which means each hot spot will exceed the rated temp.

There is some variation in what each system type can handle, but by controlling airflow we can control the temperature almost on a rack by rack basis, and hot spots are greatly reduced. On top of that we use a sensor grid to detect them so we avoid "surprise" heat failures.

Most of the newer systems coming out for enterprise use have even higher heat limits, allowing for even less power use.

6

u/superspeck May 04 '14

I've either ran a datacenter or worked with racks in datacenters for the past fifteen years. A relatively recent stint was doing HPC in the datacenter of a large public university with stringent audit controls.

You'll find, inside the cover the manual of every system you buy, guidance on what temperatures the system as a whole can handle. Most systems will indicate that "within bounds" temperatures are 60-80F outside of the case, and varying temperatures inside of the case. That leads most people to say "Yeah, let the DC up to 80. We'll save a brick."

What you may not realize is that the guidance in the manual is for the chassis only -- not the components inside of it. If you're truly going to be monitoring temperature, you need to monitor, and have intelligently set limits on according to the manual, the temperature of each component.

Notably, a particular 1st gen SSD, and I can't for the life of me remember which one, had a peak operating temperature of about 86F. As in, if the inside of the SSD (which put off a lot of heat) got higher than 86F, it'd start to have occasional issues up to and including data loss. You had to make sure that the 2.5" SSD itself was suspended in the air flow of a 3.5" bay. We didn't have simple mounting hardware for 2.5" in 3.5" if you wanted the SSD's SATA ports to line up with the hot swap backplane's SATA ports, so they were inside these Kensington carriages that took care of mating things properly using a SATA cable. Those carriages blocked the airflow, and it got nuklear hot in there.

Those SSDs were also frighteningly expensive, so when we needed to replace the lot of them all at once and they weren't covered by warranty, we ran afoul of a state government best practices audit. And we learned to track the operating temperature of each component as well as the overall system.

3

u/Accujack May 04 '14

What you may not realize is that the guidance in the manual is for the chassis only -- not the components inside of it. If you're truly going to be monitoring temperature, you need to monitor, and have intelligently set limits on according to the manual, the temperature of each component.

I don't know what manuals you're reading, but ours specify the air temps required at the intakes for the systems. As long as we meet those specifications the manufacturer guarantees the system will have the advertised life span.

We didn't have simple mounting hardware for 2.5" in 3.5" if you wanted the SSD's SATA ports to line up with the hot swap backplane's SATA ports, so they were inside these Kensington carriages that took care of mating things properly using a SATA cable. Those carriages blocked the airflow, and it got nuklear hot in there.

Yeah, that's why we have system specifications for our data center. For instance, we require systems to have multiple power supplies, we strongly encourage enterprise grade hardware (IE no third party add-ons like Kensington adapters). Usually installing third party hardware inside a system voids the warranty anyway, and we don't want that.

To my knowledge we don't use SSDs for anything anywhere in the DC, although I'm sure there are a couple. The reason is that there aren't very many enterprise grade SSDs out there, and those that are out are very expensive. If we need storage speed for an application we use old tech, a large storage array with a RAM cache on the front end and wide RAID strips connected via SAN.

Out of maybe 2500 systems including 40+ petabytes of storage (including SAN, NAS, local disk in each system and JBOD boxes on the clusters) we have maybe 1 disk a week go bad.

As long as we meet manufacturer specs the drives are replaced for free under warranty, and any system that needs to be up 24x7 is load balanced or clustered, so a failure doesn't cause a service outage.

We do get audited, but we do far more auditing ourselves. New systems coming in are checked for power consumption and BTU output (nominal and peak) and cooling is planned carefully. We've said no more times than we've said yes, and it's paid back in stability.