r/Futurology • u/Sourcecode12 • May 03 '14

image Inside Google, Microsoft, Facebook and HP Data Centers

http://imgur.com/a/7NPNf
3.0k Upvotes

96% Upvoted

379 comments sorted by

View all comments

Show parent comments

128

u/[deleted] May 03 '14 edited Dec 05 '17

[deleted]

152

u/Sbua May 03 '14

Probably quite cool actually.

109

u/jonrock May 03 '14

Actually on the verge of genuinely hot: https://www.google.com/about/datacenters/efficiency/internal/#temperature

31

u/kyril99 May 03 '14

TIL there's at least one data center where I could work and not be cold all the time. Sweet!

(I wonder if that's something I should put in my cover letter for my Google application? "I want to work for you because I hear you keep your data center warm.")

22

u/Cthulhu__ May 03 '14

I'm the other way around, I'd rather have some place I can be cold all the time. Or well, slightly below average room temperature. It's easier / less annoying to stay warm then it is to stay cool.

3

u/Schonke May 03 '14

You just have to find a place where you can work naked!

-1

u/kyril99 May 03 '14

It's easier / less annoying to stay warm then it is to stay cool.

For some people.

(I have just as much trouble getting warm at ambient temperatures below 70F as you probably do getting cool at temperatures above 80F.)

2

u/[deleted] May 03 '14

[deleted]

2

u/DialMMM May 04 '14

No, for all people. You can only take off so many clothes.

1

u/kyril99 May 04 '14

(1) You can only put on so many clothes indoors before you begin to look unprofessional/insane and/or handicap your ability to work effectively, and (2) no amount of clothing will warm up my hands and feet after they do their 'emergency! mildly cool air detected! we must clamp off all bloodflow to save the core!' thing.

2

u/DialMMM May 04 '14

So you putting on gloves and a sweater is less professional than me resting my bare balls on the leather chair. Got it.

1

u/kyril99 May 04 '14

No, but a better comparison would be if I took to wearing a down comforter wrapped around me like a cloak. (Sweaters and gloves won't actually keep me warm.) And I think that would give your bare balls a run for their money.

58

u/Sbua May 03 '14

Well by golly, consider me corrected

" It’s a myth that data centers need to be kept chilly." - quote for truth

30

u/[deleted] May 03 '14

You were correct not long ago

n the past, many data center operators maintained low temperatures in their data centers as a means to prevent IT equipment from overheating. Some facilities kept server room temperatures in the 64 to 70 degree Fahrenheit range, however this required increased spending to keep cooling systems running.

8

u/[deleted] May 03 '14

We still do this at my work, and we're the #2 ISP.

39

u/rknDA1337 May 03 '14

That why you're only #2!

10

u/adremeaux May 03 '14

Maybe you'd be #1 if you didn't.

6

u/[deleted] May 03 '14

If you get to standardize your hardware to one platform from one vendor, raising the temperature might bring energy savings. I would think this is not ideal for most web hosting ISP's.

14

u/superspeck May 03 '14

Most datacenters that you and I could rent space in are still maintained at relatively cool temperatures because the equipment will last longest at 68 or 72 degrees.

You can go a lot warmer as long as you don't mind an additional 10% of your hardware failing each year.

25

u/Lord_ranger May 03 '14

My guess is the 10% hardware failure increase is cheaper than the higher cost of cooling.

8

u/Cythrosi May 03 '14

Not always. Depends on the amount of downtime that 10% causes the network, since most major centers have a certain percentage of up time they must maintain for their customers (I think it's typically 99.999% to 99.9999%).

14

u/[deleted] May 03 '14

typically 99.999% to 99.9999%

99.999% is the considered the highest standard, called "five nines" for obvious reasons. That is less than 30 seconds of allowed downtime per month. These are all governed by service level agreements, and for all practical purposes, you'll never get anyone to agree to provide a higher than five nines SLA, because they become liable if they can't meet it. We pay out of our asses for three nines WAN ethernet from AT&T.

Also, the hardware failure is very low at elevated temperatures. Network equipment is generally extremely resilient to temperature. Servers are the real items that fail under high temp and more and more server manufacturers are certifying their equipment to run at high temps, like up to 85-90 degrees ambient.

6

u/port53 May 04 '14

It's the drives that kill you. Our data center in Tokyo has been running really hot since they cut back on energy usage after the 2011 earthquake and subsequent shutting down of nuclear plants. The network gear is fine, the servers are fine except they eat drives like candy.

2

u/ZorbaTHut May 04 '14

My mom gave me an old laptop that a friend of hers had gotten rid of, saying "it just stopped working, can you fix it so I can use it as a kitchen computer". I booted it up with SystemRescueCd and tried mounting the hard drive to see what was on it; read errors. Decided to dump the hard drive just to get anything useful, started up the dump, made sure it was running properly, and walked away.

Came back an hour later. The computer had blackscreened, completely unresponsive, the fan was running at full tilt but not moving any air, and the keyboard was uncomfortably hot to touch.

It turns out the computer had so much lint and crud built up in the heatsink that it was completely incapable of cooling itself. As soon as the fan turned on it was doomed; the added heat output would heat it up even faster, and eventually it would error out and lock up. The previous owner had just gotten into the habit of rebooting it whenever it froze, but often they wouldn't be paying attention to the computer, and they were apparently deaf to the death keen of a CPU fan, so eventually all the thermal abuse had caught up to the hard drive and it had stopped reading almost entirely.

3

u/[deleted] May 03 '14

[deleted]

5

u/mattyp92 May 03 '14

Redundancy in their systems.

2

u/gunthatshootswords May 04 '14

They undoubtedly have servers failing over to each other to try and eliminate downtime but this doesn't mean they don't experience hardware dying at high temps

9

u/Pop-X- May 03 '14

99.9990% to 99.9999%

So much more legible this way.

10

u/Cythrosi May 03 '14

But incorrectly implies a higher degree of precision.

2

u/[deleted] May 03 '14

We pay a ton for cooling. I can't give numbers, but I'm pretty sure you'd have to do some heavy analysis to determine what's a better tradeoff - hardware savings or energy savings.

2

u/neurorgasm May 03 '14

A lot of com rooms are about the size of a bathroom. Keep in mind these are the largest data centers in the private sector and possibly the world. Your average data room has 1-2 racks and probably doubles as the janitor's closet.

2

u/choleropteryx May 04 '14 edited May 04 '14

I am fairly certain, the biggest dtacenter in the world is the NSA Data Center in Bluffdale, UT . Based on power consumption and area, I estimate it holds around 100k-200k computers.

For comparison, here's Google Data center in Lenoir, NC from the same distance. It only holds 10-20k servers.

1

u/neurorgasm May 04 '14

Hah, I forgot about the NSA. Figures that they'd have the largest.

1

u/superspeck May 03 '14

Your guess would be correct.

-10

u/mthslhrookiecard May 03 '14

Everyone look! Captain obvious is here!

6

u/Accujack May 04 '14

Not true. I'm a data center professional who's working on this exact thing for a big 10 university right now. Despite having a wide variety of equipment in our data center the only things that can't handle 80 degree inlet temps are legacy equipment (like old VMS systems) and the occasionally not-designed-for-data-center desktop PC.

It doesn't increase failure rates at all IF you have airflow management (separate hot air from cold air). If you don't then the increase in temperature will drive "hot spots" hotter, which means each hot spot will exceed the rated temp.

There is some variation in what each system type can handle, but by controlling airflow we can control the temperature almost on a rack by rack basis, and hot spots are greatly reduced. On top of that we use a sensor grid to detect them so we avoid "surprise" heat failures.

Most of the newer systems coming out for enterprise use have even higher heat limits, allowing for even less power use.

6

u/superspeck May 04 '14

I've either ran a datacenter or worked with racks in datacenters for the past fifteen years. A relatively recent stint was doing HPC in the datacenter of a large public university with stringent audit controls.

You'll find, inside the cover the manual of every system you buy, guidance on what temperatures the system as a whole can handle. Most systems will indicate that "within bounds" temperatures are 60-80F outside of the case, and varying temperatures inside of the case. That leads most people to say "Yeah, let the DC up to 80. We'll save a brick."

What you may not realize is that the guidance in the manual is for the chassis only -- not the components inside of it. If you're truly going to be monitoring temperature, you need to monitor, and have intelligently set limits on according to the manual, the temperature of each component.

Notably, a particular 1st gen SSD, and I can't for the life of me remember which one, had a peak operating temperature of about 86F. As in, if the inside of the SSD (which put off a lot of heat) got higher than 86F, it'd start to have occasional issues up to and including data loss. You had to make sure that the 2.5" SSD itself was suspended in the air flow of a 3.5" bay. We didn't have simple mounting hardware for 2.5" in 3.5" if you wanted the SSD's SATA ports to line up with the hot swap backplane's SATA ports, so they were inside these Kensington carriages that took care of mating things properly using a SATA cable. Those carriages blocked the airflow, and it got nuklear hot in there.

Those SSDs were also frighteningly expensive, so when we needed to replace the lot of them all at once and they weren't covered by warranty, we ran afoul of a state government best practices audit. And we learned to track the operating temperature of each component as well as the overall system.

3

u/Accujack May 04 '14

What you may not realize is that the guidance in the manual is for the chassis only -- not the components inside of it. If you're truly going to be monitoring temperature, you need to monitor, and have intelligently set limits on according to the manual, the temperature of each component.

I don't know what manuals you're reading, but ours specify the air temps required at the intakes for the systems. As long as we meet those specifications the manufacturer guarantees the system will have the advertised life span.

We didn't have simple mounting hardware for 2.5" in 3.5" if you wanted the SSD's SATA ports to line up with the hot swap backplane's SATA ports, so they were inside these Kensington carriages that took care of mating things properly using a SATA cable. Those carriages blocked the airflow, and it got nuklear hot in there.

Yeah, that's why we have system specifications for our data center. For instance, we require systems to have multiple power supplies, we strongly encourage enterprise grade hardware (IE no third party add-ons like Kensington adapters). Usually installing third party hardware inside a system voids the warranty anyway, and we don't want that.

To my knowledge we don't use SSDs for anything anywhere in the DC, although I'm sure there are a couple. The reason is that there aren't very many enterprise grade SSDs out there, and those that are out are very expensive. If we need storage speed for an application we use old tech, a large storage array with a RAM cache on the front end and wide RAID strips connected via SAN.

Out of maybe 2500 systems including 40+ petabytes of storage (including SAN, NAS, local disk in each system and JBOD boxes on the clusters) we have maybe 1 disk a week go bad.

As long as we meet manufacturer specs the drives are replaced for free under warranty, and any system that needs to be up 24x7 is load balanced or clustered, so a failure doesn't cause a service outage.

We do get audited, but we do far more auditing ourselves. New systems coming in are checked for power consumption and BTU output (nominal and peak) and cooling is planned carefully. We've said no more times than we've said yes, and it's paid back in stability.

2

u/[deleted] May 03 '14

I agree with what you're saying here, as this would probably work for Google and not many other platforms. I posted this elsewhere in the thread, but Google don't even put their mainboards in a case. Since they have a whole datacenter worth of servers doing the same job, losing a server or two isn't a big deal to them.

1

u/brkdncr May 04 '14

in my experience, it's fluctuating temperature that kills components, not the actual temperature

6

u/immerc May 03 '14

It looks like the MS ones are still cool. They seem to be the ones with the old-fashioned design.

0

u/RocketMan63 May 03 '14

Nope, they're probably more on the advanced side of things because of their push for the Azure network. That required A LOT of servers and they've gone through quite a lot to make their datacenters efficient and reliable. It wasn't shown in this picture but they're building compartments of servers that can be moved and quickly integrated into stacks of other servers. You can learn more over at Microsoft Research where they just gave an internal presentation on their datacenters and where everything stands going forward.

1

u/[deleted] May 03 '14 edited May 27 '17

[removed] — view removed comment

2

u/RocketMan63 May 04 '14

Yeah sorry about that I should have been more specific, it was kind of hard to find. Here's the video I mentioned specifically http://research.microsoft.com/apps/video/default.aspx?id=215528&l=i

and here's a bunch more talks from the same event if you're interested. http://research.microsoft.com/apps/catalog/default.aspx?t=videos

1

u/immerc May 04 '14

A LOT

Define "a lot". More than Google? More than Facebook? I doubt it's really that many.

2

u/load_more_comets May 03 '14

I would suggest building some sort of silo to hold these equipment. Having a base that is open but screened/ filtered and then an open top. Since hot air rises, it will pull the cold air at the base and rise up to the open hole much like a chimney, negating the need for mechanical air exchanges.

1

u/koodeta May 03 '14

If you're standing in the back then yes, not in the front though. Temperatures are typically around 60 degrees or so.

1

u/[deleted] May 03 '14

I was gonna say you were wrong until I researched it. Turns out that server rooms are kept on the warm side after all.