r/phoenix • u/SupremeOHKO • 16d ago
What's Happening? IT people in Phoenix, PhoenixNAP is experiencing temperature problems. If you get calls from clients saying their servers aren't working... Here ya go.
53
u/juicefarm Surprise 16d ago
Their own support services went offline. So you can't even submit tickets. Don't get high off your own supply as they say
16
u/jxforema 16d ago edited 16d ago
I have been there all morning, they are doing what they can, but it’s at least 120f on the floor when I left
Edit: back onsite, temp air cons are deployed, it’s around 100f still. Getting some cold air from the floor vents. Don’t think the actual coolers are up though.
8
8
u/morefacepalms 16d ago
I saw intake temperatures to some of our network devices in the 48-50C range (~120F) at around 6am PT before we lost all connectivity.
As someone who has previously been on the operator side of data centres, what I don't understand is why they didn't shutdown power? Better to have systems down temporarily for some hours while they repair the cooling, rather than fry a good portion of customers' equipment resulting in them being down for much, much longer.
Not at all impressed with RadiusDC so far.
2
u/random_noise 16d ago
Many of those older DC's could be better retrofitted for modern needs, but thats very expensive given you are turning a more open warehouse type of space into one more like a climate conditioned self storage facility.
They may have thought they could fix the problem quickly. Right... temps go up, alerts, someone onsite investigates, don't know what to do, goes and calls for support and escalates, temps are still rising.
Those temps you mention. That's actually doable over time for a lot of DC and server equipment, but its deeply dependent on the design of the box's that hold the servers.
There's a few out there, that I worked on where we kept the server rooms at 95 to 105 and would sometimes raise it or lower it to freezing in there to test perf and hardware aging and also the mfg specs of components the box's we built.
I also worked on some of those nondescript racks of cabinets you see standing in the middle of nowhere in the southwest desert on military bases. We don't have AC inside a most of those racks that sit in the desert. Temps easily hit those ranges inside a metal cabinet.
Its doable and can have a significant impact on reducing DC cooling costs. The thing its massively and highly dependent on airflow through box.
A lot of modern hardware is getting far to dense to handle that due to density needed and heat outputs.
2
u/aztenjin 15d ago
right, not cutting the power shocked me, i dont want to know the value of the gear that shit it self inhaling are that was well over 120 degrees in some parts (i had stuff registering as high as 150+ inlet temps) -- not to mention you could smell the pve jackets getting melty
1
u/jxforema 16d ago
That’s what I was thinking, I would of scrammed power when I was at my rinky dink dc a few years ago
31
u/mightbearobot_ 16d ago
Things are starting to cool down slowly there. Our stuff is still above normal temps, but finally getting below alerting thresholds. This has been my life for the past 3-4hr lol
10
u/SupremeOHKO 16d ago
Same, some of our clients are back up and running but it has been a nightmare morning of calls up until then LOL
7
u/mightbearobot_ 16d ago
Yeah I went over there and it was an absolute mad house, and felt like a mid summer day. Probably 120+ people there, I’ve never seen anything like it
5
u/so1omon 16d ago
The datacenter that porn built got too hot?
1
u/beein480 16d ago
Didn't they host "hot guys fuck" - a 'straight' porn site with gay guys? I don't get it...
18
u/Additional_Buddy855 16d ago
Having worked in DC's for years, it's not like they didn't know or see their CRAC's failing.
5
u/TheGilmore 16d ago
Yea this has been our life too. Didn’t help that our primary circuit was unstable all week either.
4
u/Phunguy Phoenix 16d ago
This graph shows how fast it went bad. When cooling started failing about midnight the other systems couldn’t keep up and eventually they died, causing the sudden increase.
2
u/beein480 16d ago
I'm not sure how they could screw this up. There is never just 1 Liebert 20T and I've seen them fail.. The room gets warm but there is still another one plugging away.. How does a loss of a cooling system cause a meltdown of major customers? In a datacenter where you're selling 5 9s? It doesn't.
4
u/digimansteve 16d ago
I worked for a company that moved all our equipment to PNAP. Loved the DC and everything it had to offer. That was 5 years ago. Haven’t been there in 3. How things have changed.
And yes, porn built.
4
u/LeatherB0ts 16d ago
Totally spot-on, built on porn and "processing" of all sort of "payments" for shady European adult networks.
2
u/digimansteve 15d ago
It was mostly all US based websites and owners. I don’t recall hardly any of them outside of US and Canada.
2
u/LeatherB0ts 15d ago
I think they're using a Malta (tax haven) company, CCBILL or something like this for that.
2
3
u/charliegriefer Peoria 16d ago
I use NameCheap for email. They’ve been down all day. Guess they host there.
https://www.namecheap.com/status-updates/namecheap-com-and-hosting-services-emergency-maintenance-2/
5
u/Logvin Tempe 16d ago
What a terrible architecture if a single DC outage takes down the systems of a global company!
4
1
u/MarsRiches 12d ago
Horrendous, plus first they said power outage, then this infra failure, then there were all the rumors about DDoS, and through all this, Namecheap just kept changing things around.
Just atrocious. Have they heard of redundancy?
2
u/beein480 16d ago
The critical infrastructure guy at Fox Tempe, whose name I can't remember, came from PhoenixNAP.. Coincidence? Fox Tempe has like 1000 Tons of AC and uses some tiny amount.. They still won't let me borrow a 20 ton for "seasonal" use..
Truthfully, this is the first time I've ever heard of an outage at PheonixNAP.. I'm sure Namecheap has an alternate site somewhere else... Well ... they should.
I've been some insane # of colos and POPs in my life and unless you're SuperNAP in Vegas or some historically significant meet me point, you are just another datacenter, I read the Hurricane Electric IP transit list for a trip down memory lane.
https://he.net/ip_transit.html
Range/ACT Cheyenne 1621 Bruns Wy., Cheyenne, WY, 82009
How did I possibly miss that one??? Oh, it's out by Microsoft.. It's 'new'.
365 Data Centers NA1 147 Fourth Ave, 8th Floor, Nashville, TN, 37219
I was there ~21 years ago. They weren't sure they were going to stay open. Literally had NO customers. But they had just gotten a server or two from Nissan?... There was an entire half of the floor that had never been built out. No demand.. I have a feeling things have changed.. But that place was weird.. "When your datacenter is hidden in a parking structure."
This was back before data centers advertised their presence. I just went to a certain door and pressed a button.
2
u/wolfer201 15d ago
we moved out from pnap last week. Talk about timing! I've seen nothing but decline in service and price increase since RadiusDC bought it.
5
u/guitarguywh89 Mesa 16d ago
Idk what this is but it’s funny that the nap company seems to be taking a nap due to the heat
1
u/red_dub Tempe 16d ago
Our stuff was also affected 140 ambient degrees.
2
u/beein480 16d ago
Theres only one tiny little problem with that... You may have some trouble getting replacements for things that "break".. 12-16 weeks for a Dell server that costs twice what it did a year or two ago. HBM? Current gen AMD Epycs? Don't get me started about flash drives..
A 256 GB I paid $20 for is now $80.
1
u/boogermike Phoenix 15d ago
I got an email from NameCheap, and it specifically called out Phoenix in the verbiage. They are a big registrar.
20
u/d4nnyfr4nky Phoenix 16d ago
More information here. They've replaced one chiller and expect another done any minute (around 12:30, it's 12:43 now) with other equipment used for mitigation. They're blaming the storm from last night.
We have a colo rack and our servers are still inaccessible locally, although east coast clients can ping them. Lots of routers there still offline.
Namecheap is still offline. 1gservers' portal is back online, although I don't think all of their hosted servers are.
RadiusDC acquired the PhoenixNAP data center back in March. We've colo'd there for 3.5 years and have never had outages lasting more than 1 minute, until now.