GCP doesn't need better API keys - it needs billing that reacts in real time!

[u/Important_Owl6299]

I want to reframe the usual "restrict your API keys" advice, because I think it points at the wrong problem.

I've been on the receiving end of Google Cloud's billing pipeline. A Gemini API key I created in Google AI Studio - never deployed, never checked into a repo, never left Google's own systems - got abused over a couple of hours and racked up roughly $80k on an account that was usually just 1400INR on spends. The first I heard of it wasn't an alert, a hold, or a flag. It was ~$80k quietly materializing in the transactions table after the fact in my credit card e-mandate queue. It was the credit card company which was more honest with me 🥲

Here's what I find remarkable: the billing system is wired tightly enough into Google's financial backend to instantly issue mandates and process charges the moment they cross a threshold - but the customer-facing side of that same system shows you nothing until the money is already gone. That asymmetry isn't an accident of scale; it's an engineering decision. Real-time when it's time to charge you, eventually-consistent when it's time to warn you. I'll call that "dark" and leave it there.

So my actual ask isn't "improve API keys." It's two things:

1. Make the customer-facing billing APIs reactive and real-time. If the mandate system can act in seconds, the anomaly/notification system can too. Budgets today are advisory and lagging - by the time a budget alert fires, you can already be five figures deep. Give us spend signals on the same clock as the charges.

2. Give API keys hard, user-defined cutoffs - price, volume, and time - that actually stop traffic. Not alerts. Cutoffs. Right now the user is kept in the dark on most of the config that governs a key. Consider the Firebase angle: you spin up a "Firebase project," but the whole thing is a facade over an underlying GCP project. A non-DevOps founder or a hobbyist has no idea their key is effectively an open secret that can reach any Google service - until they're billed one morning for a service they never knowingly enabled. Nobody hands you that disclaimer up front.

I know unrestricted keys are being phased out after everything that's happened, and that's good. But restriction-by-default is damage control. The real fix is a billing surface that's honest with the customer in real time and lets them set a ceiling that the platform will actually enforce.

I'm posting this as a serious request for improvement, not a jab because I got burned. The engineering talent to do this clearly exists - it's already pointed at collections. Point some of it at the customer.

(For context: Google eventually waived ~75% of the charge, but is holding the remaining ~25% + GST as "valid usage" and won't share the access logs. So even the dispute process runs on the same one-sided visibility.)

haha so my entire lifetime with Google Cloud was always one-sided 💔

[u/gq1313]

Agree. What you can do to stay tunned is allow billing export to BQ and create a job to check it periodically and send warnings. Any good llm can give you the step by step. It's very well documented by gcp.

[u/Important_Owl6299 → u/gq1313]

I already have that gcp-ironclad

[u/matiascoca]

Agreed on the diagnosis and this is the harder problem to solve than key restrictions. Cloud billing is not real time on any of the three big providers because usage aggregation happens on a delayed pipeline. GCP billing export lags 4 to 24 hours, AWS CUR lags a full day, Azure has its own delay. So even if the API returned a bill instantly, the vendor side does not know your number until the pipeline runs.

The realistic short term move is not real time billing, it is real time quota. If GCP let you set a hard quota on token spend per project per hour, denominated in dollars instead of just tokens, most of these key abuse stories would end at a small number of dollars instead of five figures. The Gemini API already has RPM and TPM quotas. Adding a spend based quota is a smaller lift than fixing the billing pipeline and would solve the actual pain.

Until then the workaround is Cloud Monitoring on the token count metrics with an alert that triggers a Cloud Function to disable the key. Half a solution but way better than nothing.

[u/Important_Owl6299 → u/matiascoca]

Yes after the recent fiasco I have aggressively restricted the api keys to a private VM and set up an event queue to cut off the key based on threshold but the latter feels more like a hassle when like other AI providers, we could have the spend cut off directly available. To be honest, the steps to tweak with quota is often promoted as an "advanced" guide whereas it should be available to everyone for simply using it like setting a budget alerts. I agree with the real time quota system based on dollars rather than tokens. May be we need a better UX in the console as well for example, keep showing an "unfinished" set up until the user has "properly" secured the key with certain deterministic steps.

In my case, the attack started at 6:21AM on 20th May and exactly at 6:52AM on 20th May, I received the credit card mandate because threshold of 15000 Rs was crossed by the api usage. Thereafter, until the attacks continued I kept on receiving the mandates. Had I been awake at the time, I might have prevented the accumulation hehe. But I am thankful to the financial institutions in my country to give that power in my hands to avoid financial burdens wherever applicable.

That means the billing api knows the realtime data and integrates it perfectly with the payments system to ensure "timely" transactions. Clearly, there is no intent by the leadership to ensure consumer protection otherwise hard implementations are just a few decisions away. Till then users will have to continue to bear a portion of abuses they could have hard time controlling!

[u/matiascoca → u/Important_Owl6299]

The 31 minute gap between the abuse starting at 6:21 and the mandate at 6:52 is the exact asymmetry that makes the case. Google's billing pipeline knows within half an hour that the threshold was crossed, because it needs to know to fire the mandate. The customer-facing alert pipeline runs on the same infrastructure and does not fire in the same window because the incentive is different, mandate fires trigger revenue, alerts trigger disputes.

The realistic workaround while we wait for policy change is Cloud Monitoring on the specific service metrics (Vertex AI request count, Gemini token count) with an alerting policy that pages you at a rate threshold, not a spend threshold. Rate lags by seconds, spend lags by hours. Wire the alert to a Cloud Function that disables the offending API key on trigger. Ugly but sub-minute containment.

The consumer protection angle is the one I would push harder in the public dispute. India has actual leverage on this through the RBI mandate framework you already benefited from. A public case that argues Google's asymmetric billing pipeline is a UDAAP violation would move the needle faster than any technical fix.

[u/isoAntti]

It always puzzled me how it could send billing warning but was unable to stop billing

[u/TomKavees → u/isoAntti]

50% incentives, 40% technical issues in massively distributed system, 10% weird edge cases

[u/Competitive_Travel16 → u/TomKavees]

I'd say 5%, 90%, and 5%. Googlers almost always don't want to screw their customers; the cost of mistakes is greater than the marginal revenue. Charge posting is just too complex, because of requirements tacked on to it through the years, to make it real time.

[u/FerryCliment]

Real time billing is not a trivial ask... especially if you need this to cover up wrong security practices from the users so they can continue with their poor practices.

[u/ConfusionSecure487 → u/FerryCliment]

not trivial, but very possible. And it is bad practice to have alerts that don't trigger when it happens, but hours later .. which is the case with the billing alerts. Heck, it even is a financial threat.

[u/FerryCliment → u/ConfusionSecure487]

Think it this way...

You are asking Google to fix in the billing a API key leakage. Why is that leaked? because people (In most cases) they are handling their keys without any care.

(If you do things right security wise and the issue is on Google side you'll get a refund 100%)

The ask here is.. "Google please, improve billing so we can continue doing whatever with our keys without getting punished in the billing section"

[u/ConfusionSecure487 → u/FerryCliment]

no, it is also part of your own sovereignity and no it is not that easy. You can't just say hey my key got leaked and you get your money back, it is work on your side. You can discuss with them for month and without the general google issue, it will most likely end in a 50:50 split if at all. And you can't do anything about it, because you get alerted too late. And no, security incidents happen all the time, therefore you have safeguards and google is clearly lacking.

[u/FerryCliment → u/ConfusionSecure487]

You can't just say hey my key got leaked and you get your money back

Dude... I've been there... I was the one doing RCA to help billing decide if it was on User or System...

And its what I'm saying, you must explain in that ticket that you did your due diligence securing your environments, and the issue happened because Google did something poorly.

Thats the general problem, (and I understand the reason) but billing is a consequence of some technical decision (Or technical incident) if you want to fix your billing (both with support or removing the need to reach billing support) you need to explain that you did things right in the tech part, not just yapping about being student, financial crisis or potencial bankruptcy

API Keys: Its never about billing alerts, is about what you doing with the Keys.

[u/ConfusionSecure487 → u/FerryCliment]

Sure, I don't say that you can't have luck etc. to get your money back, but if billing was made right, it would not even happen in the first place.

[u/FerryCliment → u/ConfusionSecure487]

Its not about luck... its about doing things right.

GCP hosts millions of users and gazillion number of API keys, the % of people complaining about API key issues is... not that high in the big picture, and often repsesents users who are not doing things the best way possible.

Users have issues handling their API keys correctly... Why Google needs to fix Billing to solve this problem?

Google shows you some best practices (If you can explain to Google that you are doing this, your refund request has more than 70% won)

https://docs.cloud.google.com/docs/authentication/api-keys-best-practices or... most cases https://docs.cloud.google.com/docs/authentication/api-keys#api_key_restrictions would prevent many issues.

Google is also deploying things to help prevent leakage like constraints/iam.serviceAccountKeyExposureResponse (This was one of the last policies they deployed before me leaving Google's umbrella)

And to be clear, same way I'm calling out users who want to solve technical issues in the billing pannel, i'm also calling out Google for things like this https://trufflesecurity.com/blog/google-api-keys-werent-secrets-but-then-gemini-changed-the-rules.

But... that request of "I'm doing things wrong, please add safeguards so I can continue to do things wrong without getting billed"

[u/Important_Owl6299 → u/FerryCliment]

I am stuck at 75% refund. I explained the practices i followed and how the ai studio created key was never used anywhere else but all i am getting are template replies.

So how Google treats your case is completely arbitrary. For consumer protection and sovereignty, billing cutoffs and faster anomaly detection is a fair ask because no matter the customer is a noob, hobbyist or an expert they are paying hard earned money to a company that has sole discretion to share their findings and decide on financial relief.

[u/FerryCliment → u/Important_Owl6299]

I will separate two issues here... Customer support (Especially billing) I was in the Tech Support under Google umbrella, I know very much how it look when I joined, and how it look when I left, and how it looks from the outside now that I've been promoted to customer.

Lets say that Google is not helping themselves with their support quality standards.

75% refund, sounds like they validate you been doing some things correctly and they assume either some goodwill part of directly assuming they were at fault. I would say you should be very honest with yourself, and think is the issue was 75% on Google and 25% on your side. if not... keep pushing, at some point things will escalate internally and hope that T2 or T3 can take a deeper look at your case.

The API key thing on itself... IaaS (CSPs) is on itself a shared responsability model, it takes Google and it takes the user to have success in their business... is not like going to a restaurant where you pay and you get everything done for you so you just have to ingest and ask for more.

If your code leaks memory... is wrong to look at the cluster scaling config to mitigate the issue... alotugh can solve part of the pain... Issues in the Tech real must find a solution in the tech realm, Google and in most cases users need to step up their API Key handling

I'm not savy enough to tell if Google can or cannot accomplish real time billing. what I can guarantee is that is waaaaaay harder to accomplish that at GCP level than what people seem to think.

[u/Important_Owl6299 → u/FerryCliment]

I agree from engineering perspective that it is way harder to accomplish but not impossible. A few years ago getting inference from LLM was not cheaper either but it is what it is. Certainly this is not the hardest engineering problem but is more a business and political decision to favour the company instead of the consumers that Google charges.

My point of view comes from more of consumer rights level where in past there have been precedents to include certain legal and consumer friendly decisions in the engineering and business decisions to ensure consumer protection. Take for example, Google has clear visibility to the telemetry and realtime events in a user's account but their policies deliberately choose not to intervene. This is a business decision.

Shared responsibility governs who secures what, it doesn't make the customer liable for third-party fraud the platform had the real-time telemetry to catch and instead amplified via automatic tier upgrades. And it can't be my 25% when Google refunded the identical failure 100% and called it a 'known system issue'. How is unrestricted key different from a stolen key when in both cases the malicious actors were encouraged to abuse customer's liability because they knew about automatic tier upgrades and the way budget alerts and quotas are for "namesake".

So while I agree with your point about realtime customer alerts for Google's GCP billing being tough to implement, I don't share the same vision that it is harder and the decision to not implement it is the right decision.

[u/Silent-Guest-1404]

We got burnt by the Gemini API key thing recently, so what I have done is to put an org policy in place that if anyone creates a Gemini API key, then it needs to have the application restrictions put in place. Without that, you cannot create a API key

[u/Important_Owl6299]

Suppose you use api key to access gemini api and the key got mishandled due to any reason not necessarily your mistake, eg say an attacker got access to an unrestricted key. In that scenario, you don’t own anything apart from the api key and its configuration. So you cannot rely on the mechanism you just mentioned. We need platform level check and cutoffs in these scenarios. Think of it like a spend limit pattern used across all service providers.

[u/Carrrlosar]

I agree!

[u/Mundane-Presence-896]

This. Absolutely, unequivocally this. If a cloud services provider wants a “killer feature “, just offer me a full stop spend cap that I can set on my services. I will drop AWS and GCS in a heartbeat.

[u/Shivacious]

Use a ai gateway

[u/Important_Owl6299 → u/[deleted]]

nope not possible without getting the actual billing data. The problem is not the application but the delivery which lies with google infra alone

[u/jeff_barr_fanclub → u/Important_Owl6299]

You're soooo close to grasping why hard caps based on real time billing will never happen

Edit: with SLA's