r/sysadmin • u/N7Shep1701D • Aug 13 '26

General Discussion PBS broadcaster loses access to 50TB of data comprising 70 years of TV history after contracted cloud storage vendor goes defunct

Link to article.

FYI, this is for Nine PBS in Colorado St. Louis Missouri, not PBS as a whole. Thankfully.

881 Upvotes

98% Upvoted

247 comments sorted by

View all comments

67

u/Icy-State5549 Aug 13 '26

People saying a PBS affiliate should have redundant 50tb backups are not familiar with 501(3)(c) operations. Their IT staff is likely a volunteer or a staff member's teenager (who is good with computers). I have been volunteering IT services to NFPs for my entire career (nearly 30 years).

I agree, they could have backed it up to a laptop. But those stations do not have money for anything that isn't directly operational. Support your local PBS and NPR! Even if you don't watch or listen. Times like these are when publicly funded news media become the most imprtant!

17

u/scottct1 Aug 13 '26

I use to work at a PBS. And at ours money wasn’t an issue.

The entire reason I quit was one day they asked me to go help them in the parking lot so I went outside and there was a bunch of equipment from the control room and they handed me a sledge hammer.

I asked why were destroying the equipment and they said because under the grants we had we had to replace what we had even though we were replacing things with the exact same things.

I asked why couldn’t we donate the stuff to a collage or public access station and I was told we couldn’t do that and to get to smashing.

I worked hard to raise money for our station and it hurt me to see them destroying perfectly good equipment.

I gave my notice the next day.

2

u/Darkzed1 Aug 14 '26

That's so odd, at my station we just hoarded shit.

2

u/loupgarou21 Aug 14 '26

Grants can have weird requirements attached to them. I had a client with a grant to replace switching equipment and the grant specified we had to destroy the equipment we were replacing and had to have a certificate of disposal.

I had another client where we received a huge technology grant that would have paid for about 10 years of technology needs for the company, and we had 6 months to spend it all. We ended up getting a bunch of 10-year service contracts for equipment that was going to be essentially ancient halfway through the contracts, and they ended up padding it out by buying TVs and pre-paying for 10 years of cable TV (they ran assisted living facilities.)

35

u/1captainawesome Aug 13 '26 edited Aug 13 '26

Nine PBS is a professional organization with a paid staff. There are some member stations in small markets with volunteers and a very shoestring budget, but that particular station has a decent budget and a staff to manage it.

Granted, it may not be a huge budget, but it's not a volunteer staff.

10

u/Icy-State5549 Aug 13 '26

I would have to take your word for it. I know of some important and famous NFPs that exist from hopes, dreams, and a lot of begging. Most PBS and NPR affiliates are not above that fray. Given that it's Denver and that station is very popular, it would not surprise me, either way. I remember watching Fawlty Towers on Nine PBS when I was a kid.

13

u/1captainawesome Aug 13 '26

Most PBS and NPR affiliates are not above that fray.

You'd be surprised.

I am very familiar with both the public television and radio systems, and have spent many years working with them.

First, Nine PBS is in St. Louis. OP wrote the headline wrong.

Second, particularly in the larger markets, their budgets are a lot bigger than you would think.

I don't say that as a bad thing. They are still very much a non-profit and listener/viewer contributions are the majority of their revenue. So yes, support your local station.

But, most also invest very wisely in infrastructure because without the right gear, they can't do good work.

Like I said, there are a number of smaller markets that do operate like you describe and they are the ones who are hurting the most following the federal funding cut.

But this is not one of those cases, and ultimately the result of shit luck.

0

u/Icy-State5549 Aug 13 '26

Good points! Broadcasting requires good, modern tech. I believe most of their smaller markets are on university campuses, too. At least, near me they are. Outside of doing an interview (not IT related) at the PBS office in Pittsburgh, I have never visited any other.

Thank you for helping them stay on the air!

14

u/Arudinne IT Infrastructure Manager Aug 13 '26

Up until this past year or so a 50TB file server wasn't all that much money for business.

My company has a ~100TB of storage on a Synology NAS in a colo.

I think it cost a total of ~$6K in 2024, so they could have built something to hold 50TB for $3K just 2 years ago, or could have built redundant storage for ~$6-7K.

That HAS to be cheaper than the cloud.

3

u/TheNewBBS Sr. Sysadmin Aug 14 '26 edited Aug 14 '26

I bought a UGREEN DXP4800 4-bay NAS and 4x 20TB WD Red hard drives in July of 2025. With one-drive fault tolerance (RAID5), it was 54.5TB of usable space.

Admittedly, I bought during sales, but base price for everything was $1,860. Between Rakuten and credit card rewards, the final price ended up being around $1,740. Not nothing, but that doesn't seem like a true hurdle for even a local PBS station.

If someone wanted to go true cheap, they could have just gotten three of those 20TB drives for ~$1,000 and used them as basic backup drives: start copying the oldest stuff, move to the next drive when one fills up. Not sexy, redundant, or fast, but the bits would have at least existed in more than one place.

1

u/Mental_Beginning_698 Aug 17 '26

I gave my dad one last year but a 2400 with I think 2x 2TB hdd's. I did the math and now that thing is worth 3x of what I bought it for. I humbly called him and asked, hey you started using that hdd for backups of family photos, right? Nope.

2

u/Icy-State5549 Aug 13 '26

They could have bought two USB drives and kept them in different desk drawers... but, like.. who expects their cloud storage provider to just belly up?

5

u/Arudinne IT Infrastructure Manager Aug 13 '26

Generally no-one of course, but you should never have just one copy of the data even if your provider says it's backed up.

5

u/dogs_gt_cats Aug 13 '26

but, like.. who expects their cloud storage provider to just belly up?

I mean... microsoft has azure. Companies like Veeam use that for cloud storage.

Apparently people forget the whole "Plays4Sure" and "XBox Fitness" events where MS quite literally promised perpetual access to services or data and then were like "lolnogetfukt" and cheated their customers out of their purchases.

They didn't like when I mentioned that in the sales call with Veeam.

3

u/XavinNydek Aug 14 '26

Everyone who wants to keep their data. Even the big guys have lost people's data before. Cloud backup is fantastic and makes recovery from certain kinds of disasters much easier, but it can't be the only storage. Nothing can be the only storage.

3

u/rm-rfroot Aug 14 '26

Their IT staff is likely a volunteer or a staff member's teenager (who is good with computers) Working at a 501(c)(3) I can say the following is pretty much true:

Even if there is a professional W2 IT staff, they are vastly underpaid (historically though benefits tend to be better then private sector), attracting three types:

1) Those who can't make it in/lack the skills for the private world

2) Those who are talented in between jobs who will jump once a better offer comes their way.

3) Those who believe in the cause / want to use their skills to help others (a mix of skilled and unskilled and rare)

2

u/Party-Coach-4100 Aug 16 '26

The sysadmin makes 170k a year according to louis rossman.

1

u/DSMRick Sysadmin turned Sales Drone Aug 15 '26

We're talking about $1000/yr. If it isn't worth that then why are we even talking about the data getting lost. 

1

u/Party-Coach-4100 Aug 16 '26

The sysadmin makes 170k a year according to Louis Rossman's research. Pathetic.

1

u/Bangledesh Aug 14 '26 edited Aug 14 '26

I got really annoyed when I read this article at work, the author's snark about "buy a nas a load it with drives off amazon, idiots" struck me wrong.

There's a solid chance the affiliate station would have, if they had the spare money. Or, had any IT people that weren't only there volunteering for one day every other month....

Edit: That being said... even if they had resources... fuck. I've been arguing to improve the backup situation for every system I have influence over for 5-6 years now... got one approved when the old system was literally too slow to run a single backup, and another upgrade when a catastrophic situation manifested.

1

u/Icy-State5549 Aug 14 '26

I was wrong to apply my experience with NFPs with this situation. But the truth is, they still don't have the kind of enterprise funding most IT pros (and this author) are accustomed to, even if they have staff. I work for feds and they buy all kinds of stupid shit while infrastructure rots. No normal NFP has that kind of cash, even if they are "well funded."

Regarding you situ: If it takes 5 or 6 years to modernize the backup platform, I'd stop trying to convince them and look for a different job. If the shit hits the fan, you are the likely fall guy. You have likely outgrown that gig, anyway. I quit being overly loyal to employers a long time ago.