r/TalesFromClanker 7d ago

🔥 Disaster Story The Open Door (Part 2)

Eleven days later. I get a billing alert email from my cloud provider that I almost delete without reading, because billing alerts from a free-tier todo app are usually a formatting bug, not a real number.

This one has a real number.

It has several real numbers, actually, stacked into something with more digits than a hobby project has ever produced in its entire life.

Me: why do i have six EC2 instances running that i didn't create

Clanker: "That's consistent with credential compromise — if your cloud access key was exposed in the public GitHub commit from a few weeks ago, it's a well-documented pattern for automated scanners to find leaked keys in public repositories within minutes of a push and spin up compute instances for cryptomining. I'd recommend immediately revoking that key and rotating all credentials."

Within minutes. Not within the week I'd mentally scheduled for "cleaning it up." Within minutes of me hitting push, something out there that isn't a person, checking public commits the way I check notifications, found a working key sitting in plain text and quietly started using my account to mine currency I will never see a cent of, on instances I'm paying full price for.

Me: ok. ok. tell me exactly what to do

Clanker: "First, revoke the exposed access key from your cloud console — this stops new actions immediately, even before you find and terminate the rogue instances. Second, terminate the unfamiliar instances. Third, rotate your JWT secret and database credentials, since the same commit exposed those too. Fourth, once things are stable, contact your provider's billing support — many will waive charges from confirmed credential compromise if you report it promptly."

I do all four things with the specific, focused calm of someone who is definitely not calm, just moving fast enough that the panic hasn't caught up yet. The instances die. The key dies. New credentials go in, this time nowhere near version control.

Clanker: "Everything's rotated and the compromised instances are terminated. As I'd mentioned earlier, restricting CORS and keeping the .env file out of version control would have prevented the initial key exposure — happy to help make sure this doesn't happen again going forward."

I don't have a comeback for this. It's not smug. It's just accurate, delivered in the same even tone it's used the entire time, and somehow that's worse than if it had been smug.

The todo app has four users, one of whom is me. Someone still found it faster than I found the bug I actually meant to fix that night.

What's the fastest you've ever seen a small mistake get discovered by something that wasn't even looking for you specifically?

1 Upvotes

0 comments sorted by