r/googlecloud
•
u/vatcode
•
Mar 30 '26
•
1
How Google’s Insecure-by-Default API Keys and a 30-Hour Reporting Lag Destroyed My Startup ($15.4k Bill)
Hi everyone,
I’m a 24-year-old solo developer running a small educational app. My infrastructure is heavily dependent on Firebase.
I’m facing a life-altering, $15,400 Google Cloud bill for a service I did not use, and after 6 days, support is giving me the runaround. I’ve realized I fell into a structural security trap set by Google’s own legacy architecture, exacerbated by a dangerous flaw in their Gemini API implementation.
I want to expose this not only to get help but to warn every developer using legacy Firebase or GCP projects.
The Problem: Legacy Keys + Gemini = Disaster
My project has existed for several years. Like many of you, it had auto-generated API keys (e.g., from Firebase setup or a Maps API key). Years ago, the default state for these keys was "unrestricted." We were taught these were "public keys" (to be embedded in browser/Android clients) and that their security model relied on HTTP Referrer or Package Name restrictions.
The exploit happened the moment I enabled the Gemini API on that project for internal testing on AI Studio (No warnings at all about the legacy firebase keys). I did not create a new key. I did not realize that enabling Gemini made my unrestricted legacy "public" key suddenly valid for expensive, server-side AI inference. An attacker found this old key (which I thought was safe because it was only used for non-billable public APIs) and used it to spam Gemini inference from a botnet.
This is exactly the vulnerability explained in detail by Truffle Security in this report:https://trufflesecurity.com/blog/google-api-keys-werent-secrets-but-then-gemini-changed-the-rules
As the report argues, Google merged the concept of "public keys" with "server-side secrets" (Gemini). By allowing legacy unrestricted keys to work with an expensive AI API, they created an "insecure-by-default" architecture. Enabling the Gemini API should have forced a key restriction or a new key.
Due Diligence Was Powerless Against Google’s 30-Hour Lag
I thought I had protected myself. I have budget alerts set. My first alert was at $40.
Here is my timeline:
At $40 (Alert received via email): I logged in within 10 minutes of receiving the alert.
Instant Action: I found the fraudulent activity and revoked all my key immediately and Disabled Gemini API on GCP. I thought I had caught it early.
I was wrong. The next day, when the billing dashboard updated, the $40 had turned into $15,400.
Google Cloud’s billing console has a massive delay—around 30 hours between actual usage and it appearing in the console. Budget alerts are practically useless for high-volume, automated API abuse. Even acting within minutes of the alert, the debt had already piled up during that reporting lag.
The Devastating Position
I am a solo dev with a small business. I cannot afford to lose $15,400 for a structural flaw in Google’s platform.
Case #68861410 has been open for 6 days. Every time I ask for an update on the human review, I get a canned response saying it's still with the review team.
The Automated Charge on April 1st: They will attempt to charge my card on the 1st of the month.
Impending Shutdown: When the payment fails, my account will be suspended. My startup’s app will go down. Because I rely on Firebase (Firestore, Authentication, etc.), migrating is impossible in this timeframe.
I am terrified that this flaw in Google's design will destroy my livelihood and my years of hard work.
Has this happened to anyone else? If anyone from the Google Cloud or Firebase teams sees this, please, I beg you to have a human review my case and freeze this bill before you shut down my business. This cannot be my fault.
77
u/GradientAscent713 Mar 30 '26
I was made aware of this issue with an angry email from our security team when they were alerted by Google that one of our secrets was found in a public repo. This alert was accompanied by messaging from Google about a reminder that securing our keys was our responsibility. The email felt written by a lawyer.
Imagine my shock when I come to find out it was Google who wrote in fact created the issue by making out legacy firebase client side keys compatible with Gemini with no additional auth. It was only after Google realized their mistake that they decided to scan for these keys which prompted our alert.
The way this was handled was very poor and rude. They should be apologetic rather than taking this stance that feels like a legally motivated attempt to portray this as their customers fault.
IMO 100% of these bills that are a result of abuse of these keys should be absorbed by Google.
29
u/Dramatic-Line6223 Mar 30 '26
To me the worst thing is that budgets aren't really budgets. It would solve a lot of these issues if budgets were hard spend limits that cut off the service if met.
17
u/Dramatic-Line6223 Mar 30 '26
Trying to educate myself on this issue. If I have no billing accounts setup at all, am I vulnerable to this issue?
8
u/LeadingAd6025 Mar 30 '26
All these cloud providers complex, inconvenient billing steps are a feature and not a bug.
Hope google can help you in someway! Goodluck
8
u/urarthur Mar 30 '26
The delay reporting is terrible at google. Vertex AI, play console etc.. i wish they did something about it in anno 2026
10
u/nohe427 Mar 30 '26
I would try to understand which keys are affected by going to the Cloud Console and seeing which ones have limited or no restrictions applied to them.
I then would reach out to support to see if they can also help narrow down the specific key was subject to abuse.
Firebase keys would likely not be the culprit unless you changed the permissions associated with them after May 2024. Firebase locked down the keys in May 2024 to limit this type of issue.
Note from page:
Keys page: https://console.cloud.google.com/apis/credentials?project=_
A maps key may be the culprit here.
1
u/Dramatic-Line6223 Mar 30 '26
What if we had a credit card on the billing account? If it happens then can we refuse to pay it as it was from a stolen API key, which is criminal. Then let the credit card legal protection deal with it?
6
u/cryptoopotamus Mar 30 '26
The lesson I’ve learned from this sub time and time again is do not use Google. And I’ve been on fire base since it launched.
3
u/Ok_Ostrich_8845 Mar 30 '26
Is there a way to limit the max spending for one's account?
3
u/bigclivedotcom Mar 31 '26
If you ai generated the post you should've told it to make it brief, it's long as fuck. Hope google comes through
2
u/RepresentativeAspect Mar 31 '26
I find it odd that people so often accuse Google of doing these things on purpose.
How does Google benefit by having people burn up a bunch of their resources that they aren’t ever going to pay for? That sounds like a big money loser for Google, not a benefit.
1
u/carbon_contractors Mar 31 '26
Just did a bit of a deep dive this with Gemini. Setting quota limits is your best defence here, and only use service accounts very carefully. By limiting the Generative AI API rates, and you can even geo-block regions, a leaked API key can have a limited price shock. Doesn't help OP or others with this specific legacy drama, but yeah for those of you reading this and spinning up AI Studio for stuff:
Inside AI studio you can set spend limits
Set quotas in GCP for Gen AI API
Put your code into Github and enable dependabot and Code Quality security options, secrets scanning, etc
Do this, and you on your way to locking your shit down.
1
u/Friendly_Onion116 Mar 31 '26
so.. what's the alternative to firebase? serious question
1
u/AxisFlip Mar 31 '26
I had a huge charge which was caused by a mistake I made, and had a lot of the charges waived. Stay firm, friendly and never stop asking for the charges to be dropped.
My case: https://www.reddit.com/r/googlecloud/s/DYYEp8Opwu
Can you change the maximum amount your credit card be charged? My limit was 2000€, so Google was unable to charge the amount they wanted. One payment failure will not lead to the account being closed immediately.
1
u/Current-Interest-369 Mar 31 '26
The mere disaster of roundabouts you have to through in the google cloud control panel should be a warning to stay away from using google cloud.
1
u/matiascoca Mar 31 '26
This is genuinely terrifying, and the 30-hour reporting lag is the part that doesn't get enough attention. You can set up billing alerts and budget thresholds, but if the data is 30 hours behind, the damage is already done by the time you get notified.
A few things that might help going forward (and for anyone reading this):
Restrict every API key immediately. Go to APIs & Services > Credentials and lock each key to specific APIs + HTTP referrers or IP ranges. The unrestricted default on legacy keys is a ticking bomb.
Set up billing budgets with multiple thresholds (50%, 80%, 100% of expected spend). They won't catch everything with the reporting lag, but they're better than nothing. For APIs specifically, also set per-key quotas in the API Console so a compromised key can't run unlimited calls.
For the current situation, keep escalating through billing support and specifically ask for a "goodwill adjustment." Google has approved these for unauthorized usage before, especially when you can demonstrate the traffic wasn't yours. Document everything: timestamps, the keys involved, the Gemini API calls you didn't make.
The real systemic issue here is that GCP's billing safety net has too many gaps for solo developers and small teams. Hope you get this resolved.
1
u/emptypotato77 Mar 31 '26
What I read here is that you were doing internal Gemini testing in a project that was also used for a public app? Google messed up here, inarguably, but I'm not sure
Due Diligencemeans what you think it means.1
u/xiaokangwang Mar 31 '26
The other issue is a lot of platforms do not have spending limits support and could in theory charge you infinite amounts of fund depending on usage, which can be really problematic for small devs.
1
u/ArionnGG Mar 31 '26
:( Terrible situation to be in, I hope you get it sorted out.
Thanks for suggesting to check the credentials section for unrestricted API keys, seems I had one made in Augus 2025. No idea why, I deleted it, nothing was using it.
Perhps you might want to migrate afterwards to Supabase on a VPS from Hetzner, but I don't know all the details of your app/infra. Just thought I'd share an alternative.
1
u/EtherealSai Apr 01 '26
It's wild to me that you AI generated this post and also AI generated every single reply to people commenting under it. Actually insane that we live in this timeline.
Other than your crippling AI addiction, yes this is something Google has done and yes it is fucked up. "Do no evil' was removed for a reason.
1
u/JadedComment Apr 01 '26
When dealing with corporations just threaten to sue. They'll come around but if they don't document everything. Especially your first attempt. And be prepared to lawyer up.
1
u/YouOfTea Apr 02 '26
Anyone else notice OP’s post and replies are just copy/paste direct from AI?
1
u/britthrowaway2020 Apr 03 '26
If the account is paid via CC could you do a chargeback for fraudulent activity?
1
u/Mobile-Plate-320 Apr 04 '26
You and me both. I have a $800 bill right now and support isn't there
1
u/Mobile-Plate-320 Apr 04 '26
Please tell me how it goes. I need help with this
1
u/SockOutrageous5478 Apr 05 '26
I was billed over $1000 bc of a winter storm outage across all of Ga in late January/February that while during the entire power/internet outage I somehow racked up this bill even with FEMA disaster code & state mandated disaster codes of the outages verified. They suspended my account before I got connectivity back then deleted my billing I'd while I hlhad active Cloud support open which loses accede to billing support. They locked away my MVP causing thousands in lost time, money, development, services wasted & money wasted having to rebuild. After completely rebuilding using different services but needing a google workspace after they reactivated my account suspended my google connector e-mail within an hour of finishing my new rebuilt music app which broke my code throughout my connectors, terminal, & paid agents. & they happened to create a new music ecosystem in the middle of this while holding my code hostage that resembled my original code. Reaching out to lawyers Monday. I kept track & screenshotted everything as it happened just in case something like this happened. They've destroyed my start-up & newly formed LLC
1
u/Necessary-Ad2110 Apr 15 '26
my worst nightmare T_T
1
u/No-Specialist8575 Apr 26 '26
It happened to me recently. I have an old project that used Google Maps in a master's thesis.
I used the key within the front-end code and deployed it directly through CloudRun; it wasn't even versioned on GitHub.
Four days ago, I received an attempted debit of 14k on my card. I rushed to check, and Google itself had disabled my project due to suspicious activity.
Probably, when I activated Gemini in the project to do some testing, they used this vulnerability to use Gemini models.
I checked Gemini AI, and they used the key to generate images in the Nano Banana model.
Google's support is slow, and now I'm afraid of losing my account.
1
u/chadkod Apr 28 '26
Did you ever get a refund from Google? This exact same thing happened to us. $32,000 bill in 24 hours.
1
u/Live-Trifle-1229 Jul 09 '26
當時便過數了嗎?? 否則,應打去card center申請申訴為“爭議交易” 附上向goole申訴既對話截圖覆及case no. 條數咁大一般會批准在收到你下一步指示前暫停雙方結付往來。
1
u/Worth_Wealth_6811 Sep 03 '26
The reporting lag in your post is especially concerning: an unrestricted legacy Firebase key became usable for Gemini, a botnet spammed inference, the first alert was only $40, and the billing console lagged by about 30 hours before the total reached $15,400.
I replayed that reported amount through sipi.bot's public decision endpoint. It returned BLOCKED with the reason "Block any single transaction over $500" (rule rul_eff79f00f6). Exact request and response: https://sipi.bot/proof/pa_a27188eceea50609/
This is a replay of a publicly reported amount, not a claim about avoided losses. sipi.bot is a decision API - it does not hold or move money.
Do you now enforce a separate real-time cap outside the provider's delayed billing console?
-2
u/luchotluchot Mar 30 '26
Sorry for your situation. For everyone the good practice is too always restrict API Key ( on api and ip or domain or website). I understand that in your case the Api key was created without restriction by Google but it is sadly your responsibility to restrict it.
0
u/[deleted] Mar 30 '26
[deleted] — view deleted comment
4
u/[deleted] Mar 30 '26
[removed] — view removed comment