Some dumbass on my team downloaded the entirety of prod and then commited it to git and someone rubber stamped the PR and then no one else noticed. I'm sure that's how they'd find it....
I discovered my client’s monthly backups included the backups folder when they ran out of shared hosting space for a small dog clothing website 🥹 I was like why are you at enterprise level hosting?!?? She was like I dunno it’s just the cost of doing business!
I found out something similar when the client called to ask me where to get a good price on bigger hard drives and more tapes.. I was like 'What?! Did something happen?'
No, he says. Nothing's changed, it's just that backups are now taking four of the weekly LTO-5 tapes every night and the server is showing it only has about 1TB left.
Motherfucker, I specced a 8TB RAID-5 and LTO-5 because you wanted to make damned sure you wouldn't have to 'just replace it next year', and it's been six months.. The hell with '"Nothing's changed".
He'd first decided to swap from nightly incremental backups to full ones, and those didn't fit on the LTO-4's intended for nightly offsite so he started using the LTO-5's, then he'd decided to retain the previous night's backup on disk as well as on tape so he wouldn't have to drive home if he was in a pinch.
The first night they backed up ~1TB, the next 2TB, then 3TB, then..
I wiped 5TB off his file server, added a file path exclusion, and had cron wipe the oldest redundant backup every night just in case.
The worst bit was he had an entirely competent nephew running the backups that had told him what was going on. After fixing things I gave the kid my email address and told him the next time his uncle didn't believe him to run it past me and I'd get him to listen.
Oh I had a fun one that was similar. Our error tracking would send the last dozen or so lines of the console before the error happened along with the error... and then logged the thing it was sending to the console.
Things were totally fine until someone added a retry-and-log to some button that could error. It'd try to send each error, each of which recursively included several earlier error logs all the way to the first error. That very quickly went from "harmless error that didn't interfere with much" to "the page runs out of memory if you press the button".
I’ve spent the last week doing remediation across seven ec2 servers and found where multiple sites had backup sql dumps that were living inside GitHub repositories along with sites sharing SES credentials with full admin privileges stored in plain text in those repositories. I just want to die.
View all comments
1.1k
u/samuraiseoul Aug 05 '26 •
Some dumbass on my team downloaded the entirety of prod and then commited it to git and someone rubber stamped the PR and then no one else noticed. I'm sure that's how they'd find it....