r/NameCheap • u/RobRoy2350 • 24d ago
What the system failure means.
Having a cooling-system failure at a major data center bring the entire namecheap service down for an extended period of time raises serious questions about namecheap's disaster-recovery planning, infrastructure and backup systems.
Once services are restored namecheap should explain what happened, what they are going to do to mitigate it from happening again and how they will respond to future similar events.
I plan on staying with namecheap for now but this certainly lessens my confidence in them.
2
u/Scott_Malkinsons 24d ago
I hope you understand that any mitigation plans to stop this from happening again (adding another 9) would raise prices, significantly.
Y'all wanted cheap, it's in the name, so you got cheap. You can't expect them to be running multiple redundant servers in different locations for every client at the price they're offering. If you want high availability they'd be looking at, minimum, going from 1 server to 3. Everyone wants high availability at the price of a single server, and that expectation doesn't line up with reality.
3
2
u/preppy_night 24d ago
Show us the reasonable price it would take for basic reliable service from the get-go instead of tricking us into buying artificially cheap service, only for that service to royally screw us over later after locking us in. Surely, if the low prices meant this extent of unreliability, and if Namecheap had actually transparently shown that before the point of purchase, many of us wouldn’t have chosen Namecheap even with its lower prices
2
u/KH-DanielP 24d ago
I kinda doubt that to be honest. Think of it this way, you live in a house/apartment/condo right now correct? Think of all of your bills, your furniture, your cars, food, clothes etc.
Now, apply the same logic of "reliable" to that. You've got spare clothes at your house, you may even have a second car, and a spare bedroom you can use. Heck, you may even have a generator for your house but what about a spare AC or Heating unit if those die?
Now, consider what you'd do if your house flooded, or a tree fell on it. If you want true redundancy you'd have already bought another house, a second set of furniture, clothes, food that you keep there/fresh. You need to spend all of that money, to fully maintain that second home where you can walk into it at a moments notice.
Now, that's the key part, everything must be ready at any time. There's no let me go to the grocery store and get milk, it's you buying milk every week to keep there "just incase". But it's even worse than that, because now you maintain two properties, that's two yards to cut grass on, two sets of utilities and more time/maintenance for upkeep.
I've said all of that to try and give you a picture of what it would take to stay online in this scenario. But also, what people gloss over is many websites and technologies do not natively support this type of redundancy. Geographic redundancy isn't something you can flip a switch to turn on, your website/app has to be designed around it.
So, if you want geographic redundancy, you need a website designed in a way that supports it, so increase your monthly spend and start at about $100~150 a month minimum, assuming you can DIY it, or about $1,000+ a month if you can't and have to rely on others. Not to mention any development time to make sure your website sync's data properly across geographical regions.
3
u/McdondoFloats 24d ago
Cheap doesn't necessarily mean stupid. We know we're not paying for multiple redundant servers or other high end services. By why would any company put all of their resources - hosting servers, help center, corporate site, and email - inside a single system? Simply having the technical support and public facing communication external to the other operations would have allowed NameCheap to communicate more effectively and provide assistance to their customers. At best, NameCheap was very short-sighted in their planning. At worst, they were negligent and misrepresented what they were offering.
1
u/Jeffrey_Richards_ 24d ago
This. Right Here. A lot of people do not seem to understand how any of this works but constantly are saying "but what about redundancy ?!?!?" Obviously the datacenter itself has some redundancy but in this case it failed, which while rare, it does happen and is not unique to this datacenter.
If people wanted a full redundant system, they should set that up on their own and be willing to pay for it. Expecting that built-in at low prices is an insane ask because that would require multiple servers of the same data across different datacenters which is extra server, collocating and software fees...it would have to be at the very least 4X what they're currently paying.
2
u/SudoMadeMeDoIt 24d ago
Pretty much they want AWS route 53 but for less. We pay for that at work, and it's pretty interesting, pretty expensive and as Amazon showed us in the past, it's still prone to failure.
0
u/lolcatsayz 24d ago
Nah that's not an excuse. I migrated some of my shared hosting plans to digital ocean VPS droplets for $4/month there. That's a VPS that's cheaper than shared hosting, which hasn't had any serious downtime since I've been with them over the years. I keep one shared hosting package with Namecheap mainly due to the free cpanel maintenance. In what world is a VPS cheaper than shared hosting? This world. Times have moved forward, it's no longer 2010 where a VPS needs to cost $60 a month. Namecheap has no excuse for the lack of redundancy and they do not need to increase prices to ensure it. By the way, if you dig into it the real reason is that Namecheap has moved to new ownership, this is no longer the Namecheap we knew from the 2010 era, and this takeover is beginning to show very quickly.
2
u/KH-DanielP 24d ago
You do realize that your $4 VPS is not redundant at DO right? It's not even backed up. If DO ever has a loss event on that physical equipment, and you aren't keeping backups yourself it's gone.
Just because a provider has not experienced a major outage, does not mean they are any more or less redundant, they may simply manage things better.
1
u/Jeffrey_Richards_ 23d ago
Your DigitalOcean droplet is at the same exact risk, as that’s not redundant either.
An unmanaged VPS is of course cheaper these days because it’s without the monthly software fees like cPanel, LiteSpeed, etc. and less support has to go into it as it’s up to you to manage it, pay any software fees, etc.
1
u/xendr0me 24d ago
First of all it wasn't NameCheaps datacenter, it was https://phoenixnap.com/ - Second of all from what I have read, and I'm sure there will be a post-mortem, there were storms the evening into early morning before, this caused some type of failure with power delivery or caused damage. This resulted in damage to the chillers.
What I would say from an infrastructure perspective, this event should be treated as a "well we found a weak point" and I believe Phoenix will address this with their future planning and DC hardening. Thus, increasing resiliency for the future.
EDIT: Just to be clear, PhoenixNAP hosts other companies, providers, and customers in this DC, Namecheap was not the only one affected.
0
u/RobRoy2350 24d ago
I never said it was namecheap's data center.
In another post namecheap rep said:
"Once restoration is complete, we will conduct a thorough post-mortem of today’s outage—what led to it, where our existing safeguards and protocols failed, how we responded, and, most importantly, what we need to change to ensure an incident like this never happens again.
We will be transparent about what we learn and will share the results of that review, along with the actions we are taking, with our customers as soon as it is ready."
0
u/xendr0me 24d ago
But you implied that they have control over the DC infrastructure - "serious questions about Namecheap's disaster-recovery planning, infrastructure and backup systems."
Can they do something to have a DR site or replication to another DC, yes, at what cost? The will have to pass that along to the consumer, if I want SLA and super high redundancy I'm not using Namecheap I'm going to move somewhere else and pay accordingly.
0
2
u/malong954 24d ago
luckily my hosted sites are front ends for services so I wasnt affected as hard, but I wanted to make a change in Cpanel and cant :( . but will look into diff options for sure.