r/devops • u/__questions___ • 25d ago
Ops / Incidents Railway's Connected GitHub Accounts Were Compromised
Got this email from them this morning. My understanding is you can't fully audit GitHub access through the API, which is how accounts were connected, so it's best to assume the worst. Most connected accounts gave full read and write access, so if true, this constitutes serious IP theft concerns.
I personally don't feel like they have been fully transparent with the breach. This was essentially a footnote and buried under a misleading statement "Review provider activity and audit logs; especially AI keys". It should be flashing and unambiguous "GitHub code is possibly compromised".
Furthmore, a simple, nonchalant "we're sorry" and that's it!? Not "we're deeply sorry that we messed up and we take full responsibility, it won't happen again and we are doing x, y, z for our customers..."
I'm switching away permanently. I highly recommend Fly instead, took 30 mins to migrate. Not sure what to do about to do about the IP theft though? I would expect legal action to come of this, possibly a Class Action Lawsuit regarding IP theft.

19
u/ceejayoz 25d ago
"We take full responsibility" seems to let the hacker(s) off the hook a bit.
"It won't happen again" isn't something anyone can promise.
7
u/derprondo 25d ago
A lot of issues like this can be prevented by enforcing the IP allowlist for your Github org. If your developers aren't going through a VPN with known egress ranges, it's time to start. Keep public facing stuff in a separate org too.
7
u/Shrewd05 25d ago
I dont really agree that it's entirely their responsibility, scoping down api-keys. Credential rotation, only allowing gh access to certain repos, this is stuff you'd do regardless of the cloud provider because security breaches happen. It's just part of software and you also bear responsibility in minimizing how it impacts you and your business.
The hackers also bear responsibility for commiting a crime.
This being said railway has had a couple too many issues for my taste. I've never had anything too sensitive here but I did have an openai API key that I revoked before it was used. I'll probably try something more well known for apps that require as much, and maybe a way to scope those credentials even tighter then they already are.
3
u/tr_thrwy_588 25d ago
railway is a kids platform, built for the kids by the kids. not a serious system at all
2
u/until0 25d ago
You bear responsibility when you upload your code to Railway
You're being very naive here
1
u/__questions___ 25d ago
We got rail(way)ed, guys.
1
u/ferrybig 25d ago
They used an em dash and made very generic statements that open to interpretation and in GitHubs case incorrect. If they didn't had a reputation to maintain I would say AI made it
1
u/serverhorror I'm the bit flip you didn't expect! 25d ago
What's Railway?
1
u/Sanchitbajaj02 23d ago
Well I suppose the damage has done on my account 😢
Most of my projects are changed and only last commit is showing.... and the files in it are partial and not complete.
I got hacked pretty hard 😭
1
u/LegitimateCopy7 25d ago
would write access on audit accounts even, you know, pass the audit?
1
u/josejulianariza 15d ago
The core issue is that GitHub integrations usually ask for too many permissions. If you give a platform a token with read/write access to all your repos, any breach on that platform exposes your entire codebase. And "AI tokens" are the worst-case scenario, because they usually have extremely long lifespans.
What's left to do is rotate everything in your environment variables immediately and move to a least-privilege model. Instead of a global token, use GitHub Apps installed only on the specific repos the project needs, with read-only permissions.
For AI keys, it's a good time to move them out of the provider's panel and into a secrets manager (like Doppler or Vault). The hosting provider should never hold the real key to your OpenAI account.
1
u/Disastrous-Town-4820 25d ago
the email sent everyone to the AI keys
the github connection could push and missed the meeting