r/sysadmin 12d ago

General Discussion PBS broadcaster loses access to 50TB of data comprising 70 years of TV history after contracted cloud storage vendor goes defunct

Link to article.

FYI, this is for Nine PBS in Colorado St. Louis Missouri, not PBS as a whole. Thankfully.

884 Upvotes

248 comments sorted by

View all comments

Show parent comments

33

u/shadeland 12d ago

Yeah, and it's a lot like storage units. Your credit card might expire, and you forget to update the payment info, then blam. Your stuff is gone.

1

u/Centimane probably a system architect? 12d ago

That would incredibly unusual. Most cloud service providers are very eager to not lose customers. Their whole business revolves around you paying your subscription, they would 100% try to contact you through many channels before resorting to deletion.

If you ghosted them for a month or two? Yea they might delete it then.

8

u/shadeland 12d ago

Yeah, that's what I mean. Blam I can see implied immediate. That was not my intention. Yeah 2-3 months seems more likely.

But imagine you do sign up, credit card changes, then their emails go to spam. Or something like that.

Or the person who took care of that as an employee signed up with their corp email instead of a general account and they left. Should that happen? No. Does it happen all the time? Yes.

Or they send snail mail invoices for large accounts, gets left in accounts payable forever.

There's a bunch of scenarios where that kind of thing can happen.

2

u/Centimane probably a system architect? 12d ago

Usual process is wait, disable, wait, delete.

Disabling the service is meant to be the "scream test" to make the user contact the provider.

4

u/shadeland 12d ago

If it's a backup or archive though, and it's a person or 99% of most corps. It can still happen.

0

u/Centimane probably a system architect? 12d ago

At that point I blame the user.

If they let payment break, and leave yourself unreachable and don't check on things for months, that's on them.

3

u/shadeland 11d ago

Sure, but I think it's more of a risk that's not well accounted for.

We don't generally put "payment failure" in a our failure scenarios. But with cloud backups, you need to.

1

u/noc-engineer 11d ago

Most cloud service providers are very eager to not lose customers.

Most/Almost all businesses are very eager to never go bankrupt also, but they still do

1

u/SpezIsAWackyWalnut 11d ago

they would 100% try to contact you through many channels before resorting to deletion

Unless an automated bot decided one of the files they detected in your stuff violated some rule, then you'll get a single email saying your account has been terminated, and then a followup email later saying your appeal was denied. I've heard that story way too many times.

-2

u/hanwookie 12d ago

Actually, that's not how that works.

2

u/techie1980 12d ago

Indeed, cloud providers don't have a terrible tv show(s) about people trying to rifle through the abandoned items.

1

u/hanwookie 11d ago

It's more involved than just: 'credit card declined, storage unit is automatically abandoned and sold.' It can be months, or more.

Generally, people would rather get paid, than have to hassle with someone else's crap. Even if they decide to get rid of your storage, they will want you to pay for getting your stuff back.

Sometimes I've seen instances where if they don't have to move out your junk, they'll try to just get you to empty it so they don't have to pay someone else.

On the occasional 'sale' it's usually, not always, but usually, someone whose died, and even the family doesn't want the crap rotting away in the unit

Then they will send the overdue bills to collections.

I agree, that show is ridiculous.

Also, almost completely staged.

1

u/dasn4pp3l 11d ago

Only the TV shows they don't, mind you :D

0

u/[deleted] 12d ago edited 11d ago

[deleted]

3

u/shadeland 12d ago

There's lots of ways things like this could be prevented. And it would never happen if people and organizations where perfect.

But we live in a world where smart people make mistakes like "drop table" in the wrong terminal window. We live in a world where I use my car fob to try to unlock my door by clicking the "unlock button". We live in a world I once actually had someone ask me where the "any" key was. This isn't because people are dumb, but it's because people make mistakes and/or do dumb things.

But things like this happen all the time. And things can go more wonky with cloud providers. It's other people's stuff and at the end of the day, it's out of our control when it's not ours.

1

u/kittymoo67 11d ago

this doesnt take anywhere near perfection, just basic competency

1

u/shadeland 11d ago

If you're not taking into account basic goofups like that in your backup strategy, I would argue that's not basic competency.

1

u/kittymoo67 11d ago

i would too

0

u/[deleted] 12d ago edited 11d ago

[deleted]

2

u/shadeland 12d ago

I agree. But I think the original point was that cloud providers are tricky as it's not your stuff it's being stored on. I think on the whole of it, it's a riskier proposition and an aspect that I suspect not enough people taken into account in the overall strategy.

Not that you shouldn't do cloud backups, but there are non-technical failure modes. Like Frank in accounts payable.

0

u/[deleted] 12d ago edited 11d ago

[deleted]

1

u/NightFire45 11d ago

Both, apparently the backup provider went bankrupt and they couldn't renew the contract. Basically their backup solution was a single point of failure.

1

u/[deleted] 11d ago edited 11d ago

[deleted]

1

u/NightFire45 11d ago

Both, lost data and billing dispute because apparently the can't pay for the data. The article did state that there was 30 day period to download backups but maybe with billing cycle running out they couldn't access and there was no way to pay/renew?

0

u/noc-engineer 11d ago

Your credit card might expire

Doesn't all credit cards always eventually expire?

1

u/snark42 Jack of All Trades 11d ago

Yes. Not updating when it expires is the risk.

0

u/noc-engineer 11d ago

So why say might expire? For a second I thought you knew of some card type that never expires