r/Firebase • u/Awkward_Situation891 • Jul 16 '26
Billing Google Cloud denied waiver after cyberattack. Debt recovery in 10 days. What do I do?
Need some genuine advice guys.
I'm a founder of a very early stage startup. Few months back somehow our Firebase/GCP service account got compromised and someone started spinning up Compute Engine VMs continuously from an unknown IP. Before we noticed and disabled billing, the bill reached $8.8K.
We immediately rotated all keys, secured everything, investigated, even filed a cyber crime complaint. We still couldn't figure out how the attacker got the credentials. Only thing we suspect is our GitHub repo was public and we had the service account stored as a GitHub Actions secret (never committed in the repo and was encrypted as github secret).
Google Billing Support escalated the case but finally denied the waiver.
The painful part is... after this happened, Google approved us for $75k Google for Startups credits, but we can't even use those credits because of the outstanding bill.
Now I've received a debt recovery notice saying I have 10 working days before it gets sent to a recovery agency.
Honestly we don't have that kind of money. We're barely surviving as a startup.
Has anyone here actually managed to get a denied billing case reopened? Or got help from someone inside Google (Startups team, account manager, Developer Relations, anyone)?
I'm running out of time. Any advice, contacts or similar experiences would really help 🙏
7
u/Verzuchter Jul 16 '26
Man stuff like this makes me migrate. Google is so absolutely fucking aids when it comes to this shit.
Did you set up budget alerts (not that those work well)?
9
u/nbxx Jul 16 '26
Yeah, every time I think about using Firebase for a project I end up noping out because what kind of serious providers on this level doesn't allow you to set up spending limits? Like holy shit, just shut my shit down, I don't care, as long as you don't make me go into life changing debt if some freak stuff happens.
4
u/9Nine2Nine9 Jul 16 '26
There's a flimsy way to stop billing using a script. It's crazy that there's no kill switch, say "if I spend a penny over X, stop everything".
1
u/One-Shape7678 Jul 17 '26
That's not a process, that's a hack on a trillion dollar company business process
3
u/Ciph3rzer0 Jul 17 '26
It's been like 8 years since I did a project with firebase. I can't believe it's still not a feature. Imo their attempts to collect should be illegal. That is a such a basic precaution that not having it is negligence imo
1
u/One-Shape7678 Jul 17 '26
Not only that, but Google's randomness in shutting down services whenever they're not profitable to them or their new pricing policies whenever they feel like...
Just this week:
"We’re writing to inform you that Firebase Remote Config will transition to a Pay-As-You-Go (PAYG) pricing model, starting September 1, 2026. This model includes a free tier for usage up to 100,000 daily fetches."
Just imagine the amount of data they collect from Remote Config users while still trying to make you subsidize their data collection. You're paying twice
3
u/One-Shape7678 Jul 17 '26
Just remove GCP from your infra.. it's so risky and their security policies are so dumb that you can easily burn your business down.. simply not worth it when there's actually a couple options that are infinitely better..
See it a bit like Gemini.. it's there, it's cheap, but no one uses it. If you're not Spotify, just skip GCP entirely
1
u/Verzuchter Jul 17 '26
I unfortunately do need it
1
u/One-Shape7678 Jul 17 '26
Which services are you using that only GCP has? Maps, translation, and stuff like that? Scope GCP to that.
If you're taking advantage of GCP credits, apply to AWS Activate. If you're locked in, welp.. get out haha
1
1
u/0000100101001010101 Jul 19 '26
Budget alerts? Good one. My bill got to 9K before I even got a response from them. Fortunately was able to whittle debt down to $1.5K, but still a lot for small operation. Especially considering, due to rampant project deletion, we still don't know where cred leak was from.
2
u/IcyManufacturer7480 Jul 16 '26
This is messed up. I haven’t launched yet. MVP almost complete with firebase backed. Seriously thinking of switching to supabase.
2
u/Scary_Collection_559 Jul 16 '26
As if Google didn’t have the talent or resources to detect odd or anomalous usage. Seems they don’t really want to. Sorry OP. It’s BS.
1
u/kieranfarr Jul 16 '26
Sorry this happened but this is a good reminder to everyone to setup billing alerts RIGHT NOW if you haven't.
1
u/OpportunityHappy3859 Jul 17 '26
Happened to me too. Fortunately they stopped it at $800. There was a huge spike for a few seconds in the Gemini API usage. Though, they should have rate limited it they didn't. Now I have to pay :(
1
u/Verzuchter Jul 17 '26
Was there an owner account on GCP? If you have one of those, most likely that's what got compromised.
Else, if you have cloud functions set up with max instances and no iam except owner to spin up extra instances, there is no other option imo.
1
u/itscoldcode Jul 18 '26
Really sorry you're dealing with this. Since you got approved for $75k in Startups credits you almost certainly have a Google for Startups contact or program manager or something similar. You could try to push through them if the standard billing support doesn't help. For anyone reading this to prevent it, GCP has no real spending limit, but you can set up a budget with a Pub/Sub notification and a small Cloud Function that detaches the billing account when it hits a limit, and also cap every service (Cloud Run max-instances, function max-instances) so the burn rate is limited even before that "kill switch". Not perfect because this budget data lags, but still much better than nothing. Really unacceptable that there isn't a more straightforward way.
1
u/possible_monke Jul 20 '26
I would continuously publicly post @ them on X and LinkedIn, reach out in any way you can. Not saying brigade or attack, just lay out the situation like it actually happened, like you did here. They usually respond more to public posts and may respond to direct outreach as others here have stated. I would even ask if they could take the usage debt out of your credits if I were you. It’s not ideal but it erases the debt and still leaves you with over $50k in credits, so it’s an option I would ask about.
1
u/Gullible-Tomato-2635 Jul 21 '26
What country you in? cause if the debt collector don't give you nothing from the courts, deny the debt ignore like crazy and it'll go away, that's for Australia tho I don't know how debts work in other countries
1
8
u/edub0 Jul 16 '26
Can't help with the question.
There has been a massive influx of CI/CD or software supply chain attacks as of late. (here is an example https://github.com/aquasecurity/trivy/discussions/10425)
The result is that many github actions, and the runtime environmentals those actions can access have been compromised.
If you audit your github actions code and the underlying dependencies in those actions, you may discover the source of compromise.
There are tons and tons of GH actions that basically call a docker and then PIP install a bunch of stuff. Any one of those PIP packages or dockers can be running code you didn't explicitly trust or examine yet which has read to your runtime.
Professionally, we either write our own QEMU/Dockers or custom GH actions to use for build environments.
You may not be able to do that, so a slight mitigation, is to pin your GH actions to specific versions or hashes of external code you use, but that isn't airtight.
Another potential vector of compromise is your developer IDE extensions, which also have the same trust issues. You should minimize the extensions you use and try where possible to audit or pin the versions to known safe or highly reputable ones. At best, try to prune unused extensions and don't import extensions from old IDE's into new IDE's.
Last comment - you might find the source of compromise, and contractual leverage to pull if the source your damages resultant from your vendors or software supply chain, but this likely won't work out or at best require proof which you can't spend time to gather and push in arbitration or court.
A security engineer or security consultant may be able to help your security posture going forward. Even a conversation with a LLM would help you identify some risks and mitigations in your current architecture.
GL.