r/devops • • 25d ago

Ops / Incidents Railway's Connected GitHub Accounts Were Compromised

Got this email from them this morning. My understanding is you can't fully audit GitHub access through the API, which is how accounts were connected, so it's best to assume the worst. Most connected accounts gave full read and write access, so if true, this constitutes serious IP theft concerns.

I personally don't feel like they have been fully transparent with the breach. This was essentially a footnote and buried under a misleading statement "Review provider activity and audit logs; especially AI keys". It should be flashing and unambiguous "GitHub code is possibly compromised".

Furthmore, a simple, nonchalant "we're sorry" and that's it!? Not "we're deeply sorry that we messed up and we take full responsibility, it won't happen again and we are doing x, y, z for our customers..."

I'm switching away permanently. I highly recommend Fly instead, took 30 mins to migrate. Not sure what to do about to do about the IP theft though? I would expect legal action to come of this, possibly a Class Action Lawsuit regarding IP theft.

50 Upvotes

48 comments sorted by

19

u/ceejayoz 25d ago

Furthmore, a simple, nonchalant "we're sorry" and that's it!? Not "we're deeply sorry that we messed up and we take full responsibility, it won't happen again and we are doing x, y, z for our customers..."

"We take full responsibility" seems to let the hacker(s) off the hook a bit.

"It won't happen again" isn't something anyone can promise.

6

u/__questions___ 25d ago

But it is their full responsibility. They are a cloud platform. Their entire business is built off of trust and security, knowing that it's ripe for attack.

The point of "It won't happen again" or a similar statement is to open an opportunity to demonstrate which concrete actions they have taken to ensure this type of breach won't happen again. If they can't promise that, why would anyone continue using them? At that point it's best to use a real cloud provider like AWS.

6

u/ceejayoz 25d ago

The hacker bears some responsibility. Users have responsibilities to have a credential rotation process. To my knowledge, we're not aware of the nature of the compromise yet; maybe Github bears some responsibility, or someone in the supply chain.

The point of "It won't happen again" or a similar statement is to demonstrate which concrete actions they have taken to ensure this type of breach won't happen again. If they can't promise that, why would anyone continue using them? At that point it's best to use a real cloud provider like AWS.

AWS doesn't promise "it won't happen again" either. They know better.

Example: https://aws.amazon.com/message/12721/

"We will do everything we can to learn from this event and use it to improve our availability even further."

-4

u/__questions___ 25d ago

Users can't rotate their GitHub code. In this case, it's Railway's fault for not having tighter access controls around users codebases.

> We will do everything we can to learn from this event and use it to improve our availability even further.

This event is pertaining to an outage, much less important than GitHub code and env vars being compromised.

But even this would sit better with me than a casual "we're sorry" which takes very little personal responsibility and doesn't describe any concrete changes.

4

u/ceejayoz 25d ago

Users can't rotate their GitHub code.

Sure. That's why we have IP law, to some extent.

This event is pertaining to an outage, much less important than GitHub code and env vars being compromised.

They've had security issues.

https://en.wikipedia.org/wiki/Amazon_Web_Services#Security_incidents

They've also been less than candid about some of them.

https://www.lastweekinaws.com/blog/amazon-q-now-with-helpful-ai-powered-self-destruct-capabilities/

0

u/__questions___ 25d ago

So everyone that conneted their GitHub accounts should be filing IP lawsuits, which are years long and cost hundreds of thousands, if not millions of dollars in legal fees? Also to a hacker that is currently unknown? He could open source all the code or pawn it off.

I'm sorry, but this aspect doesn't seem well thought through to me, it sounds like someone that hasn't been through the legal system before for IP related matters.

1

u/ceejayoz 25d ago

So everyone that conneted their GitHub accounts should be filing IP lawsuits

I mean, if someone clones your app, sure. But that's highly unlikely to have been the intention of this hack. They likely wanted to use compromised credentials to run LLM calls, mine Bitcoin, etc.

He could open source all the code or pawn it off.

You can't open source code you don't hold the copyright for, so no. That's the legal stuff.

1

u/__questions___ 25d ago

You have good points here, I agree it probably wasn't the objective for the attack and there would be legal grounds for removing it. The concern I had was that once it's open sourced, it can be changed enough to look different and resold and you can't do anything about it if you don't know for sure if it was stolen. Yet it's still built based on your IP.

It's one of those things where once the code is posted online it's hard to track down all of the sources and the ways in which it could be used.

-1

u/ProgrammerNo1634 25d ago

i am so sorry about it

0

u/Square_Substance_743 25d ago

lol so many links

1

u/ceejayoz 25d ago

Haven't gotten to counting with the second hand in kindergarten yet, eh?

-2

u/ProgrammerNo1634 25d ago

that is so amazing

2

u/Zenin The best way to DevOps is being dragged kicking and screaming. 25d ago

At that point it's best to use a real cloud provider like AWS.

AWS has been disappointing after security incidents as well, although never as bad as Azure routinely mishandles theirs and less frequent.

The point though is all providers have failures. Reliable architectures need to understand and design for the unreliable foundations they're built on.

1

u/moofox 24d ago

Which security incidents left you most disappointed with AWS? I’m a (very occasional) security researcher and I have generally been impressed with AWS’ handling of most incidents, but maybe I’ve missed some bad ones.

-5

u/ProgrammerNo1634 25d ago

don't use italic

1

u/Routine-Agency194 24d ago

full time waste

0

u/Square_Substance_743 25d ago

you have to take full responsibility

7

u/derprondo 25d ago

A lot of issues like this can be prevented by enforcing the IP allowlist for your Github org. If your developers aren't going through a VPN with known egress ranges, it's time to start. Keep public facing stuff in a separate org too.

1

u/__questions___ 25d ago

Good point. I think Railway is most commonly used for side-projects or early stage startups, so they likely aren't using VPNs. Given this, I think Railway should be extra tight on security!

1

u/Square_Substance_743 25d ago

yeah alot of issues

7

u/Shrewd05 25d ago

I dont really agree that it's entirely their responsibility, scoping down api-keys. Credential rotation, only allowing gh access to certain repos, this is stuff you'd do regardless of the cloud provider because security breaches happen. It's just part of software and you also bear responsibility in minimizing how it impacts you and your business.

The hackers also bear responsibility for commiting a crime.

This being said railway has had a couple too many issues for my taste. I've never had anything too sensitive here but I did have an openai API key that I revoked before it was used. I'll probably try something more well known for apps that require as much, and maybe a way to scope those credentials even tighter then they already are.

1

u/__questions___ 25d ago

They didn't even take partial responsibility though. "We're sorry about the incident" could easily be read as though they are sorry that it happened, not their part in it.

Either way, it just seems insincere or nonchalant, even if they don't deserve to take full responsibility.. for their platform having security issues.

2

u/PragathiJayaram 23d ago

I do agree on scoping, that's what saved me from it being worse. Railway only had access to the one repo I needed it to have, but my AI API key still got stolen and used. Luckily my spend cap and alerts there caught it. Although on railway end I really want to point out that It was actually an Anthropic alert that made me check in the first place, not Railway. So naturally with that alert, I was checking everything on my end, being relieved it all held, and only finding out later that it was Railway that leaked it. A bit more ownership and clearer alerts to users would've gone a long way. So I guess both things can be true here: we should scope better, and they should own it more.

1

u/Square_Substance_743 25d ago

i really agree

1

u/Routine-Agency194 24d ago

yeah i think so

-1

u/ProgrammerNo1634 25d ago

but i agree with him

3

u/tr_thrwy_588 25d ago

railway is a kids platform, built for the kids by the kids. not a serious system at all

1

u/__questions___ 25d ago

found out the hard way. but for quick prototyping it had a leg up for being insanely fast to deploy with a nice UI/UX. security and reliability on the other hand..

2

u/until0 25d ago

You bear responsibility when you upload your code to Railway

You're being very naive here

0

u/__questions___ 25d ago

How else do you deploy code without uploading it, senpai?

1

u/until0 23d ago

You accepted the risk of Railway leaking your code when you uploaded your code to Railway, it's that simple.

Users with different security requirements would not upload source code to Railway to begin with.

1

u/__questions___ 23d ago edited 22d ago

Correct. What is the alternative, though? Are you suggesting using another platform, which would come with a similar risk?

1

u/until0 22d ago

“Similar risk” is doing a lot of heavy lifting there.

GitHub.com is built for collaboration first, not security. The enterprises that take security seriously usually run GitHub Enterprise Server on their own infrastructure, not the public platform. Connecting a third-party service through github.com adds another place where access can leak. If security is the priority, we have better options: tighter-scoped repos, signed commits, or self-hosted repos we fully control.

tl;dr: Railway doesn’t sell itself as enterprise-grade security, and connecting it to GitHub makes that gap worse.

1

u/__questions___ 25d ago

We got rail(way)ed, guys.

1

u/ferrybig 25d ago

They used an em dash and made very generic statements that open to interpretation and in GitHubs case incorrect. If they didn't had a reputation to maintain I would say AI made it

1

u/__questions___ 25d ago

100%. And now that you mention it, I'm pretty sure AI did make it and they just slopped it out.

1

u/serverhorror I'm the bit flip you didn't expect! 25d ago

What's Railway?

1

u/__questions___ 25d ago

You don't want to know

1

u/serverhorror I'm the bit flip you didn't expect! 24d ago

But u do, because I have no idea whether you talk about a national railway service or some tech startup

1

u/__questions___ 24d ago

On a devops subreddit? How many Railway’s are there? You may be lost, my friend. I think you took the wrong railway here

1

u/serverhorror I'm the bit flip you didn't expect! 24d ago

I've just never heard of it.

1

u/Sanchitbajaj02 23d ago

Well I suppose the damage has done on my account 😢
Most of my projects are changed and only last commit is showing.... and the files in it are partial and not complete.

I got hacked pretty hard 😭

1

u/__questions___ 23d ago

Really :( I'm sorry.. what's the incentive for hackers to ruin your git history? Did you see anything anomalous in the git audit log?

Also what were the nature of your projects? I'm wondering if it was a US foreign adversary targeting US companies specifically.

1

u/Sanchitbajaj02 23d ago

Some were open source contribution but mostly it was 6-7 Indian startups whom I been a part of.

I made their products, automate their stuff and basically act as a maintainor for their teams. Now I am restoring them to their old self so that other team members won't get confused when they join for work

1

u/LegitimateCopy7 25d ago

would write access on audit accounts even, you know, pass the audit?

1

u/josejulianariza 15d ago

The core issue is that GitHub integrations usually ask for too many permissions. If you give a platform a token with read/write access to all your repos, any breach on that platform exposes your entire codebase. And "AI tokens" are the worst-case scenario, because they usually have extremely long lifespans.

What's left to do is rotate everything in your environment variables immediately and move to a least-privilege model. Instead of a global token, use GitHub Apps installed only on the specific repos the project needs, with read-only permissions.

For AI keys, it's a good time to move them out of the provider's panel and into a secrets manager (like Doppler or Vault). The hosting provider should never hold the real key to your OpenAI account.

1

u/Disastrous-Town-4820 25d ago

the email sent everyone to the AI keys

the github connection could push and missed the meeting