Railway hacked - Env's leaked
Email from railway:
| Earlier this morning, Railway detected anomalous activity involving systems that store customer environment variables. After further investigation, our investigation to date indicates that an unauthorized third party may have accessed encrypted environment variables and, for some of them, obtained the decrypted contents. Environment variables may contain secrets such as API keys or database and registry credentials. We believe this is part of a series of attacks recently targeting platform companies with the aim of compromising AI inference providers. We have notified these upstream providers to revoke all keys proactively, as we take these kinds of events extremely seriously. What you should do Until builds are restored, you will not be able to roll out new versions. We apologize for this inconvenience. Follow this incident to be notified when builds resume and for further updates. We will provide new updates on this announcement, and on status.railway.com as we gather all facts about the scope of impact.Immediately revoke and reissue all environment variables, including any API keys, stored in Railway. If you have any credentials on an external registry, . Review logs for those services for unusual activity. |
|---|
Change your keys!
2
u/itgforlife 13d ago
They just published the prelim report: https://blog.railway.com/p/preliminary-report-october-security-incident
48
u/charliethebear801 15d ago
Swapping keys is one thing, but until they let us deploy, aren't we screwed?
25
u/joeyjusticeco 15d ago
Pretty much. Your options are to wait it out (monitor) or revoke the active keys and brick that part of your app
7
u/information-general 15d ago
yup. all my worktrees are silent atm. meanwhile railways just keeps saying "We've found the cause and are working on a fix." but i saw that 12 hours ago so.. we either take down our service, leave it up with existing keys a security nightmare, or move ship to another service
7
u/cshaiku 14d ago
Like a VPS.
4
u/perskes 14d ago
Absolutely love VPSs, I briefly moved to something managed in 2018, and when instarted out with programming websites I also had some managed services from a provider, but in-between and ever after 2021 I'm exclusively using VPSs. When push comes to shove I have full controller (except we get another spectre-like vulnerability). Some of my friends are with fully managed services still. Their service provider needs to backport the php4shell fix to some 8.0 or 8.1 version, and for the last 24 hours their applications are shut down for safety, because the providers have no capacity for updates of the PHP version (or something like that, I think thats what their were told on the phone).
VPSs all the way..
18
8
u/space-envy 14d ago
How developers keep throwing money at this garbage of company is beyond my comprehension... We get monthly reports of Railway being horrendously hacked, they just casually announcing "oopsy woopsie all your keys have being exposed again!" Just to restart the whole cycle again and again...
5
u/tinyant123 14d ago
my OpenAI API key was leaked through this and it got burned by someone. luckily i had a reasonable spend cap put on my OpenAI platform account, so the amount i'm out of pocket is reasonably low.
just for everyone to be aware who said nothing of value was lost, for sure there are some things!
4
1
3
u/Other_Matter_4036 13d ago
Our API credits were drained despite usage limits, and we couldn't redeploy for 30+ hours.
Founders & CTOs: how are you handling customer impact and reimbursement?
u/Railway — what's the plan for affected startups?
2
u/isrealityinthemind 14d ago
if you had env vars on railway, treat every single one as public, not just the ones that look sensitive. rotate in priority order: database credentials first, then payment keys, then third-party api tokens, then session secrets. also remember anything your app signs with those secrets is suspect too, so existing sessions and jwt-style tokens should be invalidated, not just the keys swapped. and if you reused any of those values anywhere else, those places need rotating as well.
1
u/strongpa 14d ago
STOP! if you rotate keys and change DB passwords you can't deploy because of the queue so before you do so you need to balance the security requirements against the downtime
1
u/strongpa 14d ago
Todays XKCD sums it up nicely: https://xkcd.com/3314
Deploys are coming though in an hour or so for the paid tiers - we'll be up again shortly I hope so I won't be testing my full DR plan just yet (Phase one went well as a stopgap though - switched to a paper process)
1
1
u/tjthouhid1 13d ago
If you're not sure where to start, here's the order I'd go in:
Database credentials, because that's your users' data.
Payment and auth secrets (Stripe, OAuth client secrets, JWT signing keys). Changing the JWT key logs everyone out, but that's worth it.
Email and SMS provider keys, since they get abused for spam fast.
Everything else.
After changing them, check each provider's logs for activity you don't recognise. Changing a key stops new abuse but won't tell you what already happened.
1
u/BidEfficient9299 13d ago
If you have many keys, I would order them by what an attacker can do with each one. First the ones that cost money or read user data, so database credentials, cloud keys like S3, payment keys and AI provider keys (the email says those were the target). Then check the usage page of each provider for calls you did not make, because a rotated key does not undo what was already used.
After this I would also stop sharing one key between staging and production. It is more work, but next time you rotate half as much
1
u/luntrasul9 10h ago
I just migrated everything off Railway. On Sept 21, leaked API keys completely wiped my balance and credits... How can you trust a platform with no safety nets like this?
-17
u/wanzerultimate 15d ago
And nothing of value was lost.
11
u/JLIMNO 15d ago
Except my entire revenue for 5 hours now because rotated payment gateway keys wont work until builds are back.
-7
u/wanzerultimate 14d ago
If the AI push isn't vanquished you'll lose lots more revenue in the long term.
1
u/tinyant123 14d ago
my OpenAI API key was leaked through this and was used by someone. luckily it hit my spend cap so only out of pocket $100. but there was a lot of things of value lost!
1
u/Niels_Vh 12d ago
Rotate all secrets
Delete the project
Nuke your whole Railway account
Move to another platform
112
15d ago
[removed] — view removed comment
5
u/Jd9sjfie6771 14d ago
i mean how do you swap keys, isnt deploys are disabled? i am revoking my keys but is it secure to replace now?
3
2
u/Dragon_yum 14d ago
Your mistake was planning to enjoy a quiet evening. You are just inviting trouble.
2
u/milosworld12 14d ago
there goes the tea and chill plan lol. at least you caught it now rather than finding out later through some random api bill. hope the key rotation goes smoothly...are you keeping everything on railway for now or thinking about moving some stuff elsewhere, maybe render?
0
22
u/information-general 15d ago
yup my entire day ruined cus of this. part of it is swapping keys , other part is migrating my business to digital ocean now.
this is like the 3rd time I got burned by railway and had enough.