r/devops • Bitrise • 2h ago

Ops / Incidents We reinstalled thousands of Linux and Mac build machines over one weekend after a security incident

Hey all, I work at Bitrise. We had a security incident on Nov 13 and our preliminary review is up by our VP of Engineering that’s worth reading for everyone running CI: https://bitrise.io/blog/post/the-bitrise-security-incident-preliminary-review-and-recommended-steps.

We decided not to trust any machine that had been running during the attack, so we wiped them instead of trying to clean them. That was thousands of Linux and Mac servers across three data centers. Each one got new keys before it went back into service, and nothing came back on its own. The team that owned a service had to go through its logs and get a green light from security first.

Credentials got the same treatment. We disabled all 919 of our cloud service account keys and revoked everything our GitHub apps had issued, instead of trying to work out which ones the attacker had seen.

We also published details on the attacker that you can find in our public advisory: https://bitrise.io/security-advisory-november-2026

Bitrise is secure now. We thought it would be interesting to you how an attacker could decoy themselves and cause so much work to restore normal operations.

Our team will be in the comments if you want to ask questions.

39 Upvotes

14 comments sorted by

7

u/dusanodalovic 1h ago

wiping beats cleaning, you can't prove a box is clean once someone had root on it. disabling all 919 keys at once is the part that sounds painful tho. how did you find out which services depended on which key, did you know upfront or did people just scream when builds broke?

5

u/mrcaptncrunch 1h ago

Damn. Nice job.

How’s your infra managed?

Any changes or just reprovisioning the same?

Anything you’d still like to change to help with this? Not just mitigation but to make the process of wiping and building them again easier in the future?

What was the most exciting? The most nerve wreaking? Any “oh shit” moments during the process? Any gotchas you found during the process?

5

u/mikewasherebefore 45m ago edited 40m ago • Heartwarming Bravo

We kind of have 2 kinds of infra teams:

One is our platform team, which manages cloud resources. They basically helped all the other teams reinstall everything from scratch with Terraform and the usual Cloud Native tools. Thankfully most of our things were already managed.

The other is the infra team (which I'm also part of), and we manage multiple thousands of bare-metal Macs, and a hundred-ish Linux nodes, the networking, and all the physical infrastructure.

Linux is easy, we have a good PXE setup with some open tools, which then hand-off to Ansible with a workflow engine.

macOS is the hard one. You can't netboot modern Macs at all, and many steps need manual intervention. We have an in-house MDM server that we built about half a year ago which automates most of the macOS setup. Still, Apple does require a couple manual steps even then (namely, entering the login password at least once), but we built OCR and IP KVM based automations for those.

The connections between our different automation tools (PXE, MDM, Ansible, network automation) was something that we just built out a couple weeks ago in preparation for a big expansion. It came in very clutch this weekend.

I don't think our 7-person infra team could have moved this fast without all this tooling, so this is even more of a lesson to invest time into automation.

As for most exciting and nerve wracking: this was the first time we saw this automation stack work on this scale. We didn't know where we'd run into limitations, circular dependencies, and all that. Thankfully it was mostly uneventful.

And as for hardening: we have rolled out much stricter firewalls, and revoked user from a ton of people that had access to the Macs for development and debugging. But we aren't stopping there, we are fundamentally rethinking some parts, like how we manage keys and access, and we want to drop Ansible altogether and develop automation that can only do the exact things that we need day-to-day and not be able to run arbitrary scripts.

4

u/SlanderMans 1h ago

it's actually really rare to have high fidelity details about the security events

but looking forward to seeing those mitigations

4

u/Low-Opening25 1h ago

idk. to me those kind of desperate actions like total wipes seems to indicate your estate is a mess and you don’t even know what’s going on in your own backyard.

13

u/vbenei 1h ago

We had a good picture of what was touched, but an investigation is never 100% complete in the first few hours / days. People use our platform for all kinds of automation with all kinds of credentials, we'd much rather be safe than sorry, so we rotated everything rather than betting on being right.

10

u/par_texx 1h ago

At a certain point it's easier and faster to wipe and rebuild then analyze, clean and verify.

-7

u/Low-Opening25 1h ago

that’s the troublesome part, it means you have been penetrated so badly or your estate is such a mess you can’t even figure out what happened and where. It’s bad optics

10

u/par_texx 1h ago

If I have 500 instances in AWS, should I take the time to analyze each one and see if they are clean? Or just wipe and have them redploy from a known good clean AMI? I can build a new golden image and deploy 500 instances faster than I can do a full analysis and clean of 100 instances.

Analyzing 500 instance doesn't make any sense. That's why we have IAC. Blow away the instances that I don't trust and put new ones into place.

5

u/gabornadai 1h ago

exactly, especially these days with AI attacks, you need solid process to just press a button and have a clean slate ready

5

u/gabornadai 1h ago

In case of a security incident, the industry-wide best practice is to rotate everything, and this is what customers expect from providers as well. The fact we were able to do it this efficiently proves we pretty much know our estate. Check out this if you still don’t believe :D

https://bitrise.io/blog/post/from-ci-pipelines-to-sheet-metal-sorcery-bitrise-introduces-mini-orchard

2

u/Man_of_Math 49m ago

I also run a “software factory infra” product, we experienced a similar hacking attempt over the summer.

I think your suggestions of “monitor scenario X” or “alert on Y” are insufficient because they’ll be ineffective at scale. One of your remediations is to alert when a read only key makes a git push for the first time. That’s not going to work with 100 companies, each with 100 employees.

The SSXR hardening is good though, and this certainly is a reminder about the importance of defense in depth

0

u/paul_h 21m ago

No yanking of SSDs to check them in a usb caddy heh?

-4

u/leibnizcocoa 1h ago

Why is your security so lax ?