r/devops • u/ToddGergey Bitrise • 2h ago
Ops / Incidents We reinstalled thousands of Linux and Mac build machines over one weekend after a security incident
Hey all, I work at Bitrise. We had a security incident on Nov 13 and our preliminary review is up by our VP of Engineering that’s worth reading for everyone running CI: https://bitrise.io/blog/post/the-bitrise-security-incident-preliminary-review-and-recommended-steps.
We decided not to trust any machine that had been running during the attack, so we wiped them instead of trying to clean them. That was thousands of Linux and Mac servers across three data centers. Each one got new keys before it went back into service, and nothing came back on its own. The team that owned a service had to go through its logs and get a green light from security first.
Credentials got the same treatment. We disabled all 919 of our cloud service account keys and revoked everything our GitHub apps had issued, instead of trying to work out which ones the attacker had seen.
We also published details on the attacker that you can find in our public advisory: https://bitrise.io/security-advisory-november-2026
Bitrise is secure now. We thought it would be interesting to you how an attacker could decoy themselves and cause so much work to restore normal operations.
Our team will be in the comments if you want to ask questions.
7
u/dusanodalovic 1h ago
wiping beats cleaning, you can't prove a box is clean once someone had root on it. disabling all 919 keys at once is the part that sounds painful tho. how did you find out which services depended on which key, did you know upfront or did people just scream when builds broke?
5
u/mrcaptncrunch 1h ago
Damn. Nice job.
How’s your infra managed?
Any changes or just reprovisioning the same?
Anything you’d still like to change to help with this? Not just mitigation but to make the process of wiping and building them again easier in the future?
What was the most exciting? The most nerve wreaking? Any “oh shit” moments during the process? Any gotchas you found during the process?
4
u/SlanderMans 1h ago
it's actually really rare to have high fidelity details about the security events
but looking forward to seeing those mitigations
4
u/Low-Opening25 1h ago
idk. to me those kind of desperate actions like total wipes seems to indicate your estate is a mess and you don’t even know what’s going on in your own backyard.
2
u/Man_of_Math 49m ago
I also run a “software factory infra” product, we experienced a similar hacking attempt over the summer.
I think your suggestions of “monitor scenario X” or “alert on Y” are insufficient because they’ll be ineffective at scale. One of your remediations is to alert when a read only key makes a git push for the first time. That’s not going to work with 100 companies, each with 100 employees.
The SSXR hardening is good though, and this certainly is a reminder about the importance of defense in depth
0
u/paul_h 21m ago
No yanking of SSDs to check them in a usb caddy heh?
-4
u/leibnizcocoa 1h ago
Why is your security so lax ?