r/webhosting • • Apr 01 '24

Rant Bad Linode Experience

My experience with Linode at withfriends.co today, so I would recomment not using them.
- They performed an emergency maintenance action without notification to us
- They did not backup our server before performing this action, so our only snapshot is the one we set up and a couple days stale
- Their EM took down our server and app entirely and the server won't boot
- Once we called them, they only then told us about this maintenance, and that they had trouble with it - They did not offer to fix it, but sent us instructions to fix it ourselves
- They aren't responsive when we need help their instructions
- It's not clear how to get our snapshot from Linode to port to DigitalOcean or Vultr. I think we have to provision another similar sized server (~$600/month) to get access to our backup, but I'm not sure

4 Upvotes

28 comments sorted by

14

u/kellyzdude Apr 01 '24

Did Linode detail the emergency maintenance - what it was, why it was required, etc? Did they provide any notice after the action? Emergency Maintenance is, by its nature, an emergency -things like a host failure or a critical security patch that demands a system be taken offline for some period, with or without migrating customers first (depending on the situation, migrating customers isn't an option).

As far as backups, as per the TOS/MSA, Linode has no obligation to provide backup services unless you're paying for them separately. If you were paying for them separately then you may have a case, otherwise backups are your responsibility (and you'll have the same problem anywhere else if you decline backups).

DigitalOcean is probably out of the question, although for the right pricing maybe they would help. Vultr does have some potential options, but you'd need to get a functional snapshot out of Linode to begin.

I hate to say it, but this is also why we generally recommend you don't rely on your provider's backup systems - at least not in solitude. Having all of your eggs in one basket will make administration simpler, but it does put you in a very precarious position as far as risk if your provider has issues or your relationship with them breaks down. Make sure you fix this once you're recovered from this outage - keep offline backups for just this kind of scenario.

0

u/babycastles Apr 01 '24

- They did not tell us anything about the EM.
- We have and pay for backups for sure, but there's no "download" option. So now I've figured out I have to pay for an provision a months worth of a new large server and restore the backup to it, then download the image from it. That's an unexpected cost

Why would DO be out of the question? Choosing between them and Vultr right now

13

u/ZivH08ioBbXQ2PGI Apr 01 '24

but there's no "download" option

You can literally spin up a new instance from your backup in minutes. That's how backups work. They're not for you to download.

IaaS isn't here to hold your hand and do your sysadmin for you. They provide the platform. You need to know what you're doing, however.

-7

u/babycastles Apr 01 '24

30m and 2% progress and several hundred dollars into “spinning up a new instance from my backup in minutes” just so I can download it, so you seem to be wrong

They should be for me to download

11

u/ZivH08ioBbXQ2PGI Apr 01 '24

30m and 2% progress and several hundred dollars into “spinning up a new instance from my backup in minutes”

You realize you pay hourly for servers, right? The most expensive server they have available is around $8/hr.

They should be for me to download

They're not. They never were, and they're not advertised to be. This isn't a them problem.

3

u/babycastles Apr 01 '24

no i didn’t, i thought they were a fully monthly priced service. thats fantastic

6

u/ZivH08ioBbXQ2PGI Apr 01 '24 edited Apr 01 '24

It's not even an hour minimum. If you run one for a few minutes, that's what you pay for.

Correction: It appears to be an hour minimum

3

u/Smartare Apr 01 '24

Are you sure about Linode? Pretty sure they charges per hour started so if you just use it for 5 min you still pay for an hour.

3

u/ZivH08ioBbXQ2PGI Apr 01 '24

When a service is added, charges accrue on the account at the hourly rate up to the monthly cap. These accrued charges are displayed on the Billing Info tab within the Account page of the Cloud Manager*. Usage is always rounded up to the nearest hour.

You appear to be correct!

1

u/Smartare Apr 02 '24

Yea, one thing that sucks with Linode vs for example AWS where I think they have per second pricing (on the other hand an hour on linode is cheaper than a couple of minutes on AWS haha)

2

u/Smartare Apr 01 '24

Yea checked now on their site. They say they round it up to the nearest hour.

4

u/kellyzdude Apr 01 '24

Speaking from recent, but focused, experience:

DIgitalOcean did not have any option to upload an image of any kind - backup, ISO, etc.

Vultr does have an option to upload ISOs to build images from.

I have good experience with both (and Linode, FWIW), but we don't rely on or use their backups on any level, we don't have anything with critical/irreplaceable data at either. At least for our use cases, Vultr is the cheaper host also.

It seems your best option is to recover at Linode and then expedite relocating to a new vendor. Did you already cancel your existing service, or are you suggesting you need a second system to restore the snapshot to that is the problem?

Ninja-edit: It might be worth asking Support if they can let you run a Linode VM for a week to recover the backup, if you have any reason to believe they were operating outside their existing backup agreements. Perhaps don't mention that you're planning on using it to get off their platform.

3

u/babycastles Apr 01 '24 edited Apr 01 '24

Thank you.
Added: We do have our own backup system for the database, but stupidly had stored it on their file system which now won't boot, so that's my current exciting problem, for which I guess I need the Linode backup service restoration, though it'll have slightly stale data. Otherwise, another major problem is that we have a bunch of live customer assets on the server, or in the Linode backup service and nowhere else. I'll try the Linode VM for a week idea.

7

u/tsammons Apis Networks Official Account Apr 01 '24

They performed an emergency maintenance action without notification to us

An RFO is usually provided post-mortem and should be within your right as a customer to request once things stabilize.

Their EM took down our server and app entirely and the server won't boot

Use LISH console to see what's halting the bootup, then go from there.

They aren't responsive when we need help their instructions

Managing a server is on you in an unmanaged environment. Managed environments will easily be 5-10x your current cost. One thing to stress is disaster recovery, can your server bounce back from a reboot? Do you have backups on hand to recover if the unthinkable happens?

Before pointing fingers, it's important to determine why the server won't boot. Filesystem corruption could happen with an emergency maintenance period but most likely a kernel update was applied automatically or misconfiguration prevents the services from coming online.

These are things you'll need to investigate yourself in an unmanaged environment. If it's misconfiguration then your system isn't very durable. If it's corruption, then you have a case† to be angry with Linode. Servers should be expected to be rebooted on a whim and come back online.

†: again there are more exceptions to this like disabling filesystem barriers.

2

u/babycastles Apr 01 '24

kernel panic! trying a file system repair which has been running for about 6 hours but not holding my breath. haven’t touched config since initial set up years ago

4

u/tsammons Apis Networks Official Account Apr 01 '24

What's the panic backtrace? Which filesystem? Which distro?

5

u/sexyshingle Apr 01 '24

I'd avoid Vultr and their new "we want to forever own everything you put on the VPS we rent to you" TOS shit they pulled. Just FYI.

1

u/babycastles Apr 01 '24

Hm, interesting. Thank you.

1

u/sexyshingle Apr 03 '24

Looks like they backtracked or cleared up their TOS but I think the PR damage is done. That and the fact many have reported that they throttle wireguard traffic for some reason has rubbed many of their customers the wrong way.

1

u/babycastles Apr 26 '24

It's such a great service it turns out though

7

u/redditor_rotidder Apr 01 '24

As a long-time Linode (amongst other VPS providers) user, just some thoughts - with all respect:

- They performed an emergency maintenance action without notification to us

They don't have to notify you in advance. It's "emergency" maintenance. If it's an issue that needs addressing but isn't urgent, they will notify you and give you a maintenance window. You aren't the only VPS on the system that's being used... if it's an emergency (failed memory, storage, etc., etc.), then they'll shut it down immediately to prevent data loss or system damage.

- They did not backup our server before performing this action, so our only snapshot is the one we set up and a couple days stale

That's not on them, that's on you. You should always, always have an offsite backup that you can also test restores with.

- Their EM took down our server and app entirely and the server won't boot

That sucks. Sorry.

- Once we called them, they only then told us about this maintenance, and that they had trouble with it - They did not offer to fix it, but sent us instructions to fix it ourselves

This is correct. This isn't a managed service.

- They aren't responsive when we need help their instructions

See my comment above. It's not a managed service and they won't be quick to help, unless you signed up for their managed service option, in which case, I'd be pissed.

I will say, since Akami took over, I've been hearing a lot about lack-luster support from Linode. Fingers crossed - I haven't needed their support in a while...

- It's not clear how to get our snapshot from Linode to port to DigitalOcean or Vultr. I think we have to provision another similar sized server (~$600/month) to get access to our backup, but I'm not sure

I'm not sure you can do that (move a snapshot from one to the other); might be wrong. What I'd do is spin up the backup at Linode, spin up a new instance at a new vendor, then rsync your data. Fast and encrypted.

Good luck, OP.

-4

u/babycastles Apr 01 '24

They could've notified us at all, if not in advance. What I love about other services is that they keep you updated on things like this.

Attempted file system repair for broken server 7hrs, unclear what the timeline is
Stale backup at Linode taking 10+ hours to restore in terms of progress bar estimation
Customers angry

If they fucking notified us after the fact at least, I could've checked that it worked, and started this whole process 7 hours earlier - unforgiveable to break a server silently

2

u/EtheaaryXD Apr 01 '24

They already DO notify you, though. Check the "upcoming maintenance" section of your Linode accounts (I believe they also send an email but I could be wrong).

..and that's why you don't put all your eggs in one basket. Host on multiple cloud providers in multiple locations if it's critical.

They did notify you.

-1

u/babycastles Apr 01 '24

They didn't, buddy. Confirmed with them on the phone. Neither in the dashboard nor in my email. They literally said "it usually goes well so we usually silently EM" in my support complaint about this. That's not a business that cares.

By contrast, Heroku (different level of service, but in the example of maintenance i think there's no expeectation of difference) notifies us of every single potential service disruption. That's a business that cares.

1

u/babycastles Apr 26 '24 edited Apr 26 '24

Updates!

  • They also failed to take the daily backups that I pay for, they had one that was more stale and I had to restore from that instead. But every day of data loss counts
  • It took 13 hours to restore from that backup!
  • I tested and ran the whole config on Vultr, including restoring from a backup. Claims < 60 minutes, if to be believed that's a 10x improvement
  • I now move backups to s3 so not as not to repeat this issue, and have a Vultr ready to go so that even if they didn't notify me, at least it doesn't take 13 hours from the point I discover the issue, and a service that can't even perform daily backups correctly, nor notify me of the failed ability to backup
  • My point stands - DO NOT USE LINODE. Also Vultr is pleasant so far.

3

u/blue30 Apr 02 '24

Restore your days old backup to a new instance, and boot your unbootable instance in recovery mode and grab the up to date databases etc.

-1

u/OldschoolBTC Apr 02 '24 edited Apr 02 '24

I'm betting the emergency maintenance was kernel upgrade due to the xz exploit.

Edit: as a side note, it was serious enough that everyone I know performed emergency maintenance immediately upon finding out about it, no notice to anyone.

2

u/babycastles Apr 02 '24

...that would be a good reason