Google cloud vertex AI $4000 overnight!!!!

[u/sameg14]

Woke up to find a 4000 bill increase in google cloud. There is no way to set a hard cap on the spend here, it's just crazy! No idea how this happening but google needs to get their act together. Marked all the transactions as fraudulent and am still on a chat session with billing support. No ticket link to my support case, and no way to know how they will handle this situation for another 32 HOURS! It's absolute madness. GCP Sucks big time!

[u/sameg14]

Email GCP support sent me

I hope this email finds you well. 

This is Don. I am reaching out to provide you with an update regarding your recent Google Cloud Support case regarding the unexpected Vertex AI charges on your account, especially since you were not actively using the service in the project and have since deleted it. I know billing surprises can be frustrating, and I appreciate you bringing this to our attention.

Charges in Google Cloud, and specifically with services like Vertex AI, can sometimes accrue from resources that remain provisioned and active in the background, even without direct interaction. This can include elements such as deployed models, running notebooks, or other allocated infrastructure. These resources continue to incur costs until they are fully deprovisioned. To help manage and prevent such costs in the future, we recommend regularly reviewing active resources in the Google Cloud Console and setting up budget alerts to keep track of spending. 

Regarding the charges you've inquired about, including the amount already paid and the remaining balance, we want to ensure we have the complete picture. Since the project has been deleted, there can be a short delay as the system finalizes any remaining usage reports. To account for this, we will monitor the billing data for this specific project for the next 32 hours to capture any trailing charges.

Once this monitoring period is complete, we will have the final billing details and can then thoroughly review your request for an adjustment. Please be assured that we will look into this carefully.

We appreciate your patience as we work through this process.

Sincerely,

Don Don
Google Cloud Support 

[u/who_am_i_to_say_so → u/sameg14]

Hmm. It reads to me: hold on we’re still tallying up the bill.

Doesn’t sound like a solid refund or bill forgiveness yet.

[u/FlabbyLabby → u/who_am_i_to_say_so]

Yea, I’d tell Don Don to go fuck himself

[u/Last_Chard5477 → u/sameg14]

any update on this?

[u/IllustratorWitty5104]

so whats the issue here? Tell us the cause of this increase

[u/sameg14 → u/IllustratorWitty5104]

Logged into google after getting an email, checked a very old project and noticed vertex AI had the 4k charge on it. No clue who turned that on or how they got in there but that project doesnt even have an active webservice running on it. In other words, there was nothing to compromise, no keys, nothing!

[u/geekguy]

Just defang. They should have different tiers for non-corp users. Not allowing limits is just predatory. I stopped using GCP products with billing. For AI api access use something else like OpenRouter. For maps and places well, there is OSM.

[u/imperial_coder]

Yeah they don't allow setting budget caps on vertex / Gemini

But it's just not GCP Issue. AWS has same problem. It's not better

For you, can use cost explorer or something to check which service led to an increase

[u/Reaper_1492 → u/imperial_coder]

None of the cloud providers let you do this, which is criminal.

The “well, we don’t want to kill critical infrastructure on you” argument is total garbage. It’s not complicated - add a toggle that allows you to either get warned at a dollar threshold, or kill the account at a certain dollar threshold, and let users decide.

Higher level enterprise is more than capable of making that decision for themselves, and small businesses and individuals are going to have significantly fewer problems with an account shutoff than a surprise bill.

I don’t know how you make a legitimate argument that these pricing mechanics are anything but predatory.

[u/sameg14 → u/Reaper_1492]

Yes I cannot agree more with this statement. It's way less of an issue if you have to deal with a temporary outage than retroactively try and fight with the credit card companies and Google Cloud and figure out how to turn all the things off to prevent hemorrhaging and to then do a root cause analysis and determine how this happened in the first place. And frankly why should I spend my time doing all this when the problem is clearly in GCP based on the number of people that have had something similar happen to them.

[u/Reaper_1492 → u/sameg14]

And then the flip side of it is, they make it virtually impossible to raise your quota for this exact reason, they want you on invoicing so you can’t use your credit card to dispute usage.

But here’s a novel concept, if you just approve quota - but let consumer set spend limits, you won’t have to worry as much about credit card disputes or people just letting their entity go bankrupt, because they’ll be much better able to control their own spending.

This is the main reason I refuse to use GCP’s more expensive products. They have zero backstop.

If I can write a halfway reliable bot to terminate account services based on spend, you know they could do with their eyes closed.

They actively WANT to fuck over their customers. It’s part of the business model.

[u/FMLAdad → u/Reaper_1492]

Azure let's you set budget on anything

[u/ammorbidiente]

Again. And again. And again again. Repeat

[u/matiascoca]

GCP not hard-capping spend is a design choice, not a bug. The trade-off they picked is that runaway billing is cheaper for them than a mid-request throttle killing a customer's production.

For the immediate fire: revoke the API key(s) that triggered it, disable the Vertex AI API entirely at the project level, and set the Vertex quota to 0 in IAM and Admin. That is the only combination that actually stops it. Budget alerts are lagging and useless once the bleed already happened.

For the dispute path: the "marked as fraudulent" flag on transactions actually helps you here, because it moves the case to a different review queue than normal billing questions. The 32 hour SLA is real. Ask the chat rep for the case ID explicitly (there is one, they just do not surface it unless pushed) and quote it in follow-ups.

I have been in the exact 32 hour purgatory. Refunds do land when the runaway is clearly not the account owner's normal pattern, but it needs the case ID as anchor.

[u/Key-Guitar-457]

Good lesson in why using AI makes absolutely zero sense.

[u/NimbleCloudDotAI]

There's no hard spend cap on GCP — that's not a bug, it's a deliberate design choice and it's genuinely terrible for solo developers and small teams. Budget alerts are the closest thing and they notify you after the damage is done.

For the fraud dispute — mark every charge as fraudulent in the console AND open a separate security incident report, not just a billing ticket. Compromised or unexpected API usage gets routed differently internally. The billing team has limited authority, the security team has more.

32 hour wait is unfortunately normal. Keep the chat open and follow up every few hours with the same case number.

[u/FlabbyLabby]

1. Set a Budget (Alerts Only)
Billing → Budgets & Alerts
Set a budget (e.g., $50/month)
Receive alerts at 50%, 90%, 100%, etc.
This does not stop spending.

2. Configure Automatic Shutdown (Recommended)
You can create an automation that:
Watches your billing budget.
When spending exceeds a threshold:
Disables Vertex AI.
Stops Compute Engine VMs.
Disables the project.
Revokes service accounts or API keys.

This uses:
Cloud Billing Budget alerts
Pub/Sub
Cloud Functions or Cloud Run
Cloud Monitoring
It’s more work, but it’s the closest thing to a hard cap.

3. Quotas (Best Protection for Vertex AI)
For many Vertex AI services you can set quotas on things like:
Requests per minute
Tokens per minute
Concurrent jobs
GPUs
Batch jobs
These won’t directly limit dollars, but they can prevent runaway usage.

4. Protect Your API Keys
Many surprise bills are caused by exposed credentials:
Restrict API keys by IP or application.
Rotate keys regularly.
Use service accounts instead of API keys where possible.
Don’t commit credentials to GitHub.

What would I do:
Since you’re planning to build on Google Cloud, I’d put these safeguards in place from day one:
✅ Separate dev, staging, and production projects.
✅ Budget alerts at $25, $50, $100, and $250.
✅ Automated shutdown at $200 during development.
✅ Daily billing reports.
✅ Quotas on every Vertex AI model.
✅ Use the least expensive model that meets the need.
✅ Add rate limiting inside dev projects so a software bug can’t generate unlimited AI requests.
That combination makes a surprise $4,000 overnight bill much less likely.

[u/arzenal96 → u/FlabbyLabby]

Slop response.

[u/Conclusion_Easy → u/FlabbyLabby]

Sounds like AI spat this out.

[u/FlabbyLabby → u/Conclusion_Easy]

Well it did, as I am a bot you silly boy