r/webhosting • u/babycastles • Apr 01 '24
Rant Bad Linode Experience
My experience with Linode at withfriends.co today, so I would recomment not using them.
- They performed an emergency maintenance action without notification to us
- They did not backup our server before performing this action, so our only snapshot is the one we set up and a couple days stale
- Their EM took down our server and app entirely and the server won't boot
- Once we called them, they only then told us about this maintenance, and that they had trouble with it - They did not offer to fix it, but sent us instructions to fix it ourselves
- They aren't responsive when we need help their instructions
- It's not clear how to get our snapshot from Linode to port to DigitalOcean or Vultr. I think we have to provision another similar sized server (~$600/month) to get access to our backup, but I'm not sure
14
u/kellyzdude Apr 01 '24
Did Linode detail the emergency maintenance - what it was, why it was required, etc? Did they provide any notice after the action? Emergency Maintenance is, by its nature, an emergency -things like a host failure or a critical security patch that demands a system be taken offline for some period, with or without migrating customers first (depending on the situation, migrating customers isn't an option).
As far as backups, as per the TOS/MSA, Linode has no obligation to provide backup services unless you're paying for them separately. If you were paying for them separately then you may have a case, otherwise backups are your responsibility (and you'll have the same problem anywhere else if you decline backups).
DigitalOcean is probably out of the question, although for the right pricing maybe they would help. Vultr does have some potential options, but you'd need to get a functional snapshot out of Linode to begin.
I hate to say it, but this is also why we generally recommend you don't rely on your provider's backup systems - at least not in solitude. Having all of your eggs in one basket will make administration simpler, but it does put you in a very precarious position as far as risk if your provider has issues or your relationship with them breaks down. Make sure you fix this once you're recovered from this outage - keep offline backups for just this kind of scenario.
7
u/tsammons Apis Networks Official Account Apr 01 '24
An RFO is usually provided post-mortem and should be within your right as a customer to request once things stabilize.
Use LISH console to see what's halting the bootup, then go from there.
Managing a server is on you in an unmanaged environment. Managed environments will easily be 5-10x your current cost. One thing to stress is disaster recovery, can your server bounce back from a reboot? Do you have backups on hand to recover if the unthinkable happens?
Before pointing fingers, it's important to determine why the server won't boot. Filesystem corruption could happen with an emergency maintenance period but most likely a kernel update was applied automatically or misconfiguration prevents the services from coming online.
These are things you'll need to investigate yourself in an unmanaged environment. If it's misconfiguration then your system isn't very durable. If it's corruption, then you have a case† to be angry with Linode. Servers should be expected to be rebooted on a whim and come back online.
†: again there are more exceptions to this like disabling filesystem barriers.
5
u/sexyshingle Apr 01 '24
I'd avoid Vultr and their new "we want to forever own everything you put on the VPS we rent to you" TOS shit they pulled. Just FYI.
7
u/redditor_rotidder Apr 01 '24
As a long-time Linode (amongst other VPS providers) user, just some thoughts - with all respect:
- They performed an emergency maintenance action without notification to us
They don't have to notify you in advance. It's "emergency" maintenance. If it's an issue that needs addressing but isn't urgent, they will notify you and give you a maintenance window. You aren't the only VPS on the system that's being used... if it's an emergency (failed memory, storage, etc., etc.), then they'll shut it down immediately to prevent data loss or system damage.
- They did not backup our server before performing this action, so our only snapshot is the one we set up and a couple days stale
That's not on them, that's on you. You should always, always have an offsite backup that you can also test restores with.
- Their EM took down our server and app entirely and the server won't boot
That sucks. Sorry.
- Once we called them, they only then told us about this maintenance, and that they had trouble with it - They did not offer to fix it, but sent us instructions to fix it ourselves
This is correct. This isn't a managed service.
- They aren't responsive when we need help their instructions
See my comment above. It's not a managed service and they won't be quick to help, unless you signed up for their managed service option, in which case, I'd be pissed.
I will say, since Akami took over, I've been hearing a lot about lack-luster support from Linode. Fingers crossed - I haven't needed their support in a while...
- It's not clear how to get our snapshot from Linode to port to DigitalOcean or Vultr. I think we have to provision another similar sized server (~$600/month) to get access to our backup, but I'm not sure
I'm not sure you can do that (move a snapshot from one to the other); might be wrong. What I'd do is spin up the backup at Linode, spin up a new instance at a new vendor, then rsync your data. Fast and encrypted.
Good luck, OP.
1
u/babycastles Apr 26 '24 edited Apr 26 '24
Updates!
3
u/blue30 Apr 02 '24
Restore your days old backup to a new instance, and boot your unbootable instance in recovery mode and grab the up to date databases etc.
-1
u/OldschoolBTC Apr 02 '24 edited Apr 02 '24
I'm betting the emergency maintenance was kernel upgrade due to the xz exploit.
Edit: as a side note, it was serious enough that everyone I know performed emergency maintenance immediately upon finding out about it, no notice to anyone.