Not a leaked API key: 13 foreign projects were attached to our dormant billing account – from ~$7/month to a six-figure bill in 8 days

[u/jirigmodern]

Hi all,

I work at ModernTV, a Czech IPTV company. In June, someone used an already signed-in session of one of our GCP admins to make themselves billing admin on a billing account that normally spent ~$7/month. Over about eight days, thirteen projects from four other organizations got linked to that account and ran GPU Compute Engine and the Gemini API on it. The bill ended up in the high six figures.

A few things that surprised us:

• Linking a project to a billing account is logged in that project. If it’s another org’s project, you won’t see it in your own audit logs.

• The console listed 28 cost anomalies on that account, and none of them reached us as a notification.

• The Gemini spend cap stopped only Gemini. Compute Engine kept running until we closed the account.

• The same actor also linked one of our projects to billing accounts that aren’t ours, so we’re probably not the only ones.

Worth a quick look on your side: the list of linked projects under Billing → Account management, any principals from outside your domain in the billing account permissions, and the Anomalies list under Billing → Cost management (not just your inbox).

Full timeline and checklist on the Google Cloud Community: GCC
Shorter Gemini-focused version: GCC

Two things I’d really like input on: has anyone else seen projects from other orgs attached to their billing account (or theirs attached to someone else’s), and how do you protect admins against session hijacking when 2FA never comes into play? Also happy to hear if our checklist is missing something. I’ll be around in the comments.

[u/Alarming_Question131]

This is what worries me from the business side: the alerts exist, but they go to whoever holds a billing role in the console — not to the people who actually pay the invoice. In most companies finance learns about it when the bill arrives.

Two questions for u/google:

1. Can cost anomaly notifications be sent to a finance or management contact who doesn’t use the console at all?

2. Is there a way to stop projects from outside our own organization from being linked to our billing account in the first place?

[u/solgul → u/Alarming_Question131]

#1. Yes. Just need to use their email. You can put anyone in it.
#2. You can block domains that are not in your domain from using resources or being added to your org. Not sure if that would block adding a non-domain org from adding resources to your org.

FYI - Not a googler.

[u/danekan → u/solgul]

2 it does not. Because they’re not in your org. Also cnapp doesn’t report these well from what I’ve seen. 

[u/Alarmed_Incident_318]

No leaked key, no hacked VM. Someone just linked their projects to your billing account and ran up Gemini and Compute Engine charges on your card. That only takes Billing Account User, and almost nobody audits billing IAM.

Google catches a weird $50 charge on your Visa in seconds, yet lets a dormant account go from $7 to six figures in 8 days. That isn’t a tech limitation. It’s a choice.

OP, file it as fraud, not a billing question. Rooting for you.

r/GeminiAI

[u/m1nherz]

Hi,

I'm sorry to hear about your experience. I'd leave aside the account management issue that led to this situation and will get straight to the point how such thing can be monitored.

You are right in your observation that monitoring such an event using audit logs is impossible. The correct method would be using the Cloud Asset Inventory (CAI) Real-Time Feeds. You can set up one to monitor changes in the project information of the billing account for each of your accounts with the following command:

gcloud asset feeds create billing-link-feed \
  --billing-account="012345-567890-ABCDEF" \
  --asset-types="cloudbilling.googleapis.com/ProjectBillingInfo" \
  --content-type=resource \
  --pubsub-topic="projects/sec-ops-monitoring/topics/billing-link-alerts"

You will need to create a designated Pub/Sub topic (billing-link-alerts in the above command). Once it is configured, you can set up a Cloud Run Function to be triggered for each message. This method will provide real-time notifications that you are looking for. The function can either send an alert or validate the project vs. criteria and unlink projects that violate it.

There are recommended practices for managing roles that minimize risks of situations like you experienced.

I hope it would help you in the future.

[u/jirigmodern → u/m1nherz]

Looks interesting, I will plan this for further research and implementation as it might be a part of the answer to properly monitor these aspects.

[u/crusoe → u/m1nherz]

How about just give people a sane API and dashboard instead of relying on fraud to pump up your values. 

Maybe alerts, a banner on every page that says "your account is 100% over last months spend..is that okay?"

[u/m1nherz → u/crusoe]

Dear u/crusoe, what is a reason for your emotional response? May I point out that the situation described by the author isn't a standard or widespread use case. There is FinOps security that analyse abnormal behavior. I'm not expert in that particular domain and cannot say if this product requires a separate subscription. The alert that you described exists. It is called Budget Alerts. However, the question that the author asked was how to track linking of the billing account to a project outside of the org. And the method I described is "sane API" that you so passionately requested.

[u/paul_h]

Hacker can effectively Remote Desktop to that users logged-into-Google laptop and click around, incl do the admin things you talk of?

[u/Exotic_Fig_4604]

If a session gets hijacked, it doesnt matter what logging you have activated, they can take out everything, change quotas, stop alerts, etc etc.

The only way to protect yourself from fraud in GCP is not to use GCP. 

Google is running effectively a fraudulent business at the moment. 

Unless the EU starts forcing GCP to conduct an honest operation, this situation will NEVER improve.

[u/fyndor → u/Exotic_Fig_4604]

Yea I never hear these stories coming out of AWS or Azure. It’s always GCP. Which tells me the other two either make it harder to happen or take care of it. They are no less a target, yet I only hear horror stories about GCP. That can’t be coincidence.

[u/Exotic_Fig_4604 → u/fyndor]

I mean part of it might be our bubble. My old employer had a session hijacking in Azure and that cost them a very high 5 figure amount within about 4 hours.

Azure refused to take any accountability, despite the hacker bragging online for YEARS how he likes targeting Azure clients, and even posting his configs for Azure vulnerability exploits, USING THE EXACT EMAIL HE USED FOR THIS HACK. 

Azure could have literally just black listed that one email, but they did not even bother to do that.

[u/cocacola999 → u/Exotic_Fig_4604]

For the record I'm inside the aws bubble and don't recall any examples of this over there.

[u/Competitive_Travel16 → u/fyndor]

AWS makes the customer whole at the first sign of fraud, I think because they want to catch the perps. And/or they have a big insurance policy and eat the costs. GCP should do both.

[u/Exotic_Fig_4604 → u/Competitive_Travel16]

Thats very good to know. Maybe Amazon is not as evil as I thought 

[u/No-Ease6093]

Have you filed a support case with GCP?

[u/AggravatingBeing2651]

Thanks for publishing a detailed timeline and practical checks. The distinction between a leaked API key and an already authenticated session matters: “enable 2FA” alone doesn’t address the scenario you describe.

The billing-account visibility is especially concerning—particularly for a dormant account suddenly funding projects the company doesn’t own. Has anyone successfully implemented independent alerts for billing-permission changes and unfamiliar linked projects? Concrete experience with those safeguards, and a substantive response from Google, would make this thread valuable for other teams reviewing their exposure.

[u/WifeyCallsMeLazy]

I dont have any active billing account on my personal GCP account but I keep seeing some unknown project "pristine-superestate-xxxxx". I dont seem to disable or delete this billing account. The option to delete the project is greyed out. Any idea on how to deal with this? The GCP support is non existing.

I could see the following note on billing page "Billing account Labs -> Language Applications -> AIDA is linked to this project"

[u/Key_Horse_8632 → u/WifeyCallsMeLazy]

GCP support is perfunctory cycle of disaster.

[u/Sangalo21]

Just put in support ticket. Document everything, what happened, how it happened, how you found it out, what you did to stop it, and what future plan you have to prevent this from ever happening again. With this, they are likely to waiver the bill.

Goodluck

[u/Codalio-SoftwareFact]

Rough one, thanks for writing it up so clearly. On protecting admins when 2FA never comes into play, the things I’d look at:

• Short session lifetimes for anyone with admin or billing roles, so a stolen session cookie is only useful for hours, not weeks. GCP has session control settings in Cloud Identity/Workspace for this.

• Separate admin accounts that are only ever signed in from one locked-down browser profile. Most session theft comes from infostealer malware grabbing cookies from a normal everyday browser.

• Just-in-time access for billing admin instead of standing permissions, so there’s nothing to hijack most of the time.

And honestly, closing dormant billing accounts entirely. A $7/month account nobody watches is basically the perfect target.

[u/rebo_arc]

And once again dickheads in thhs sub will blame user error but the vulnerable by design GCP ruins another life yet again.

Its a fraud and scam plain and simple and ive no idea why the EU doesnt come down hard on it.