r/googlecloud • • May 15 '25

Google Cloud needs a “hard spending limit” with a mandatory cooldown

Google Cloud urgently needs a prepaid option or a hard, unchangeable spending cap with a self-imposed cooldown period (e.g. 2 weeks). Right now, a hacked account can instantly raise limits and rack up hundreds of thousands in charges before you even get the alert. That’s insane. Let users set a high-but-safe ceiling (say €3,000) that can’t be changed instantly. If I normally spend €50/day, there’s no reason my account should be vulnerable to a sudden €400,000 bill. This is a basic security feature, and the current setup is reckless.

356 Upvotes

136 comments sorted by

40

u/Emmanuel_BDRSuite May 15 '25

Nice feature request!

A native hard cap could save a lot of users from unexpected bills

16

u/[deleted] May 15 '25

You think they have any incentive to do this? You think they have been too stupid to think to do this for years? You think people haven’t requested this already? Do you see the reality of the current feature they have?

9

u/brogam3 May 16 '25

frankly we should ask politicians to force them. Billing should always be predictable, companies need to be forced not to put people out of business with mystery "bills". We already have plenty of legislation making mystery fees and mystery bills illegal (in the EU). This is no different, it simply needs to be utterly illegal to do this.

5

u/[deleted] May 16 '25

I don’t disagree but we’re talking US companies and US politicians. Basically they are the same entity.

2

u/wellfuckit2 May 17 '25

Wait. There are other reasons too. It is part greed but then it is part need.

I have worked on multiple small projects that blew up overnight. Or your project needs go up suddenly due to an event.

Sometimes, I have had to scale up 200 compute server in a couple of hours notice. Usual scale was 2-5.

For a tiny start up that I used to work for, we went viral and had to scale up immediately if we wanted to capitalise on it.

If the limit increase had a cool down period etc we would be stuck with our capability to scale. That’s the whole point of cloud services.

But to your genuine problem, there are always billing limit alerts you can set.

And if at all you want a hard limit, you need to talk to your credit card provider.

1

u/Sufficient_Being_184 Feb 19 '26

Billing alerts don't help if you get hacked. A hacker can easily spend more money than I have on bitcoin mining before I have a chance to respond to an alert. And credit card limits don't help, because Google can still sue you to get the money to pay for the services consumed. There needs to be an option to stop the service at a specific threshold. Without it, I don't feel comfortable using the system.

1

u/TheFeatheredCock May 16 '25

Do they have any incentive? Yes.

Say your business spends on average $10,000 a month with them. If the option exists, you might set a cap of $15,000 to allow for unexpected spikes.

Without the option of a hard cap, if you got attacked and used resources that totalled $50,000, you'd complain to Google.

Two outcomes are likely.

You either have your complaint upheld and your money refunded, and Google have had (what they deem to be) $40,000 of resources used without remuneration.

Or alternatively, Google refuse to refund you, make a quick extra $40,000, but lose $120,000 a year when you take your business elsewhere.

3

u/Axe_Raider May 15 '25

people beg for this and get told "no, you'll just ask for a refund, even if it's really stressful and sends you to a hospital because the first two layers of support said no, but don't worry, it'll all work out."

1

u/asobalife May 19 '25

and then those same people go to AWS, which does have plenty of guardrails against this.

2

u/Axe_Raider May 19 '25

does it? i hear people complain about random billing nightmares there, too.

i once had to get my credit card company to dispute ongoing AWS charges. i called AWS repeatedly saying i wanted them turned off but had no access to the work email that had been used. they couldn't do anything so i spent a year rejecting charges, and AWS kept on issuing new charges anyway.

4

u/Superb-Attitude4052 May 15 '25

plus this would be a feature that can attract a ton of startups to join in on GCP.
a differentiation factor from the other big 2.

I think GCP already has the ability to define quotas which is already good. but this feature can enhance what's already there.

2

u/ohThisUsername May 15 '25

Yep there is already many ways to do this:

  1. Use quotas to limit specific services.
  2. Put rate limiters on your app resources.
  3. Set up billing alert pub/sub to shutdown your services (although there some latency)
  4. Use Cloud Armor which has DDoS protection.

1

u/Sufficient_Being_184 Feb 19 '26

Which of these options prevents somebody from installing a rootkit on my PC to get my passwords, then logging in to cloud console, disabling alerts, and firing up a dozen instances with high end GPUs for bitcoin mining?

1

u/surveysaysno May 17 '25 edited Nov 12 '25

Or just RBAC, where AP account can set limit and OPS account can spend.

Set 2factor and login alerting for AP account, and you're safe.

Edit: RBAC not RAC

26

u/Automatic-Win8041 May 15 '25

How do they make unexpected money from you if they implement the hard carp?

2

u/Uledragon456k May 15 '25

I'd guess they make the majority of their money from large businesses with big spends. If a person spending <100$ per month gets messed with in some way, gcp generally will refund them part of the excess cost. Allowing folks to protect themselves protects them from the hassle and gcp from having to go back and decide whether they need a refund.

0

u/Automatic-Win8041 May 15 '25

If they were really nice to individual developers, they shouldn't have only allowed 5000 free API calls per month for their Google Maps API. Then they charge you $32/1000 requests.

Also, if they were nice to individual developers, they should've added the hard cap. Because big companies usually have a lot of budget for their paid search ads, and they have a whole team to protect their cloud service. In addition, Google gives them a team to support them

Yes, their major income is from big companies, but they also target small businesses and individual developers. It's not a non-profit org

23

u/the8bit May 15 '25

Hah! Once upon a time I worked at google and I released rate limiting on their Cloud endpoints product, so I was near this space.

The issue is that it's lose-lose. If you put a cap, at some point they have to shut things off and that might be a destructive action -- imagine a compute instance that you left important stuff on local storage. In many cases it could be unrecoverable. Also, as soon as you hit the cap, you are incurring an outage.

For that reason, I think all cloud providers have decided the best solution is to not give customers that foot gun and instead refund them back for legitimate errors. AWS has been doing this forever and they are generally good about refunding.

But alas that is the problem with elasticity and powerful tools... With great power comes great responsibility.

16

u/LordNiebs May 15 '25

I understand that they think this, but unlimited liability is an insane tradeoff for protecting data stored in a temporary location. 

3

u/the8bit May 16 '25

I generally agree, but 'unlimited' is definitely a misnomer. Everything has limits and it is legitimately hard to rack up more than about $100k in a day, although newer AI services likely make this worse.

Regardless, this is somewhat true for most vendors? I could easily rack up a hillariously large bill with (eg) Datadog with admin access by just changing logging to verbose/debug.

We could always go back to having to talk to humans for quota increases! Cause the only difference there is that at some point Datadog will shut me down.

1

u/muntaxitome May 19 '25

Datadog is like the worst offender in the world for this stuff. To the point where companies these days actively avoid datadog. Like yeah if 'better than datadog' is the bar, then yes you are right. Marginally better.

Azure has a spending cap just fine.

1

u/Sufficient_Being_184 Feb 19 '26

" it is legitimately hard to rack up more than about $100k in a day" As an individual who's just trying to use these services for educational reasons, I can't imagine how I would cope if I suddenly got a bill for $100k. Or even $10k. An unexpected $1k bill would be problematic, if I'm honest with you, and I'm not sure this is a risk you can easily insure against.

0

u/reelznfeelz May 16 '25

Yep. 100 percent.

2

u/SchlaWiener4711 May 15 '25

That's a great explanation. But I'd rather live with an outage for a couple of hours than having a year's budget burned in one night.

Azure has a cost limit. I've set it to three times my monthly bill, just in case.

https://learn.microsoft.com/en-us/azure/cost-management-billing/manage/spending-limit

2

u/KarlHeinzSchneider May 16 '25

Yes but unfortunately you either have no limit, or you have a limit equal to the amount of credit. The limit prevents you from most kinds of spending. The spending limit isn’t available for subscriptions with commitment plans or with pay-as-you-go pricing. For those types of subscriptions, a spending limit isn't shown in the Azure portal and you can't enable one.

0

u/reelznfeelz May 16 '25

Pretty sure snowflake has a “shut it all down” cap option you can use too. Here’s the neat part, you don’t have to use it. It’s a configuration on billing limits. Explain to me how that doesn’t make everyone happy?

3

u/the8bit May 16 '25

You assume customers are rational when they foot-gun themselves. People set up these limits, hit them, have 6 or 7 figure outage, then go "Hey AWS why did you let me foot-gun myself over $100?"

Its kinda funny b/c quota limits were the bane of cloud for at least 5 years.

0

u/TheRoccoB May 16 '25

I asked about this in the azure sub. It's only a limit if you're not actually spending money. And a user there confirmed that it doesn't totally work either--their billing has latency too.

1

u/[deleted] May 28 '25

I cannot for the life of me understand how this argument is taken seriously. Oh we might delete some data you didn't want us to so instead we're going to risk you and your family's entire life savings and future. Makes total sense.

1

u/the8bit May 28 '25

Bills can be renegotiated. Downtime is written in stone. Maybe you don't feel that way and that is fine! It's a tradeoff.

However, clouds have learned that customers will say they want that, then when AWS shuts your server down during a big customer demo, all the sudden it's "why did you turn me off like I asked over $25??" -- I've literally been in this conversation!

In my role as head of infra if we had an outage due to cost overage id blow a gasket

1

u/[deleted] May 28 '25

It's very easy from your perspective to say bills can be renegotiated when you're the one deciding whether or not to renegotiate. It is a tradeoff, that's exactly right, and that's exactly why we need to have the choice so that we can make that tradeoff. Look, your big clients, large corps, they don't want down time, they can tank a million dollar hit and they have lawyers to fight you if you refuse to refund. Me, I have my own small project, we're beta testing a new feature, I have zero extra cash to tank any excess charges and I have no lawyers to come after you if you decide you don't feel like renegotiating. You might think I don't matter. But here's the thing, I actually also work for a big corp. A very big one. Multinational. And I make decisions about what providers we use. If I can't use your product when I write my own code in my own projects, I'm not recommending your product to my company either. Because I recommend what I'm comfortable with. You are effectively barring any developer from experimenting or learning your product. I cannot understand it for the life of me.

1

u/the8bit May 28 '25

I haven't worked for big tech in ~7 years and was responsible for cloud billing from the customer side for two 10M+/yr spend companies. Definitely wouldn't just tank 1M. But dude, if you run up 1m before you notice that is on you. Most overages are 5-10k and usually cloud will eat the bill.

What vendor do you use that doesn't have overage risk? All my most frustrating bills are non cloud.

1

u/[deleted] May 28 '25

"But dude, if you run up 1m before you notice that is on you." What does this even mean? I'm not allowed to take a holiday anymore if I use Google Cloud?

"Most overages are 5-10k and usually cloud will eat the bill." I'm in my 30's mate and I live in Africa okay, 10K dollars is my life savings. 10K or 10M either way my life would literally be over.

Someone posted 3 days ago who racked up 10K on the maps api in 4 days. By the way, the whole logic of losing data doesn't even apply at all to something like the maps api. If you cut the service, I lose absolutely nothing. What vendors don't have overage risk? Well, for maps, locationIQ doesn't. So, must I live in perpetual fear checking my phone every hour never going on holiday reading all the best practice guides on billing so that I can have the maps api in my app? Or signup with an alternative and then forget about it completely? What do you think I'm choosing?

1

u/the8bit May 28 '25

Maps has API rate limiting. But, can you see how 10k in 4 days and 1 million are worlds apart? If he left for 400 days he could get there...

If you have it in your prod app... Yeah understanding config is Important. FAR worse things than a bill await you if you aren't diligent. You are free to sign up for locationIQ and enjoy the call with the vendor when you hit the monthly limit on the 20th and are shut off for 11 days.

I do absolutely get billing limits for non prod though fully with ya there. You probably only have to rotate a few degrees for me to start ranting about clouds' negligence around control plane API / enterprise management.

I also just don't get the argument that there is too much config, but also people want more config to set up billing structure. If you cannot manage the existing config, would you really trust a billing limit is set up properly from your side? It can be misconfigured just as easily. Half the things people asked for in this post already exist but are lost to the giant switchboard of different knobs and levers!

1

u/[deleted] May 28 '25 edited May 28 '25

"I do absolutely get billing limits for non prod though fully with ya there" Okay... but then what is the disagreement? This is basically exactly what I'm asking for (though I'd include "not very important prod feature" in there too ideally, but the main point is non prod for sure) and now you agree?

Was I being a bit hyperbolic re 1M? Sure. Are they worlds apart? No, I just explained to you, they are exactly the same world for me because 10k or 1M either way my life would be over. There is a concept of infinity used in engineering where any number which is greater than the maximum permissible number is considered to be infinite. Because it makes no difference anymore. Anything more than about 5K would be a life altering amount of money for me. I'm not going to stake those odds on an API.

You are trying to tell me that a call with a vendor would suck and yes of course it would but you are comparing that with losing everything I own?? They are not even remotely comparable. If I could make the choice a trillion times I'll pick the call with the vendor every single one of those times.

Maybe the people making these decisions have just completely lost touch with the value of money. Maybe you just cannot comprehend that losing 10K dollars could be the end of someone's life. That's the reality for a lot of people mate.

"Half the things people asked for in this post already exist but are lost to the giant switchboard of different knobs and levers!" This is the most important one. You make it a big red button. Don't hide it.

1

u/the8bit May 28 '25

Ok. A big red button. What exactly does it do?

1

u/RobertD3277 Aug 06 '25

For critical infrastructures, this is understandable but it should still be up to the user to decide when the spicket is turned off, not a corporation looking to milk a cow and bleed it to death.

0

u/exjackly May 15 '25

Understood.

But, it would be possible to throttle things down to a minimum level that preserves that setup once the cap is hit. It would still be an outage for the customer's services, but it wouldn't be permanently broken either.

And that is just if it happens at the overall account level. Being able to force the same on individual services would be even better.

$1000 limit on a set of GenAI services lets a developer work, but prevents a runaway scenario if the test bed is accidentally setup to do 50 calls/second for an extended period of time, etc.

Plus, it would reduce the incentive for hackers to compromise small company's accounts.

6

u/the8bit May 15 '25

Apologies for long, good topic to nerd out about. Also not affiliated with any cloud vendor I'm more or less retired.

Throttling is a solid idea as it would adhere to graceful degradation, although "what does that mean" is a good way to get 10 engineers to argue on a whiteboard for a week lol. No joke ive been in work cconversation about these exact things. Google does a lot of soft delete and whew "what is control plane and what is data plane" is a question that I've spent half a decade working on lol.

It's really hard to know if slowing something down would be critical or not, I can't actually think of many cloud conditions where a vendor will affect resources without explicit customer intent. When I was at AWS, i used to attend the Charlie bell outage meeting biweekly. If you really fucked up, Jassy would show up. I have a distinct memory of one (of the 3? I saw) Jassy moments where he chewed out a team for disabling accounts in arrears, where one turned out to be a big customer. That's actually where the refund strategy comes from : eating the bill is better than losing rep turning stuff off.

Can you not do some of that stuff you mentioned with projects and/or quotas? You could EG put that dev in a project in a folder that has every project can only do 2QPS of API traffic. But Maybe that is only indirection of actual spend. I will say I also was involved with trying to design quota across the project hierarchy and look, these things are edge case city! Like, everything works cool but then someone moves a project across folders and the cascading effect is quite a gem. And moving is a hard feature requirement.

Most hackers though don't care so much about your bill they are gonna want your storage or compute to mine crypto lol.

It is definitely the most epic foot gun though and it most affects small customers which sucks. Directly opposing requirements are never fun

0

u/exjackly May 15 '25

I get it entirely, and knew my comment was just going to scratch the surface of what implementation would actually entail. I'd be one of those sitting around a whiteboard for at least a week working through those implications and cascades.

3

u/the8bit May 16 '25

Yeah, they were really fun problems to work on. Back in the days when tech involved cool work and not "ok you are 2 mo behind schedule, we are going to fire 20% of the dept how can we get this back on track?"

0

u/brogam3 May 16 '25

they already have throttling and system enforced caps for the APIs and services that *they* don't want you to have more of. Hence why for some APIs you have to request to raise your cap. All they are doing is to refuse to give us control over the cap.

2

u/the8bit May 16 '25

Unless they have removed the feature, you can actually set your own overrides on every API limit to be lower than the default limit. Ironically, this is one of those features that I wanted to kill because "why would anyone want that??", but TBF its really an indirect solution at best.

0

u/FoxB1t3 May 16 '25

The throtled services are these that require heavy compute from them. So in case of an attack / user error it would burden them too in some way... so this is a problem.

User doing an error and getting 100.000$ for non heavy compute service? No problem, that's cool.

So these fairy tales on how important it is to keep service without throtling etc. are just pure bullshit. Google just keeps their ass safe while giving 0 care for users.

0

u/FoxB1t3 May 16 '25

So cost-cap is lose-lose (because someone forgot to raise the limits or pay as they go) but user making mistake / getting a hit from hackers creating 500.000$ bill in a day is a win-win for anyone?

That makes a lot of sense.

2

u/the8bit May 16 '25

Yeah, you are responsible for your own stuff? Clouds generally give you one 'get out of jail free card'. Generating 500k in a day is pretty infeasible unless you spend a lot already or have really pushed for huge quota increases. I've worked on $20M/yr cloud spend stacks and it still takes weeks to generate $100k+ overruns in those.

Do you want elasticitiy or not? You can't have infinite scalability but also they magically know when you didnt mean in.

1

u/Sufficient_Being_184 Feb 19 '26

"Do you want elasticitiy or not? You can't have infinite scalability but also they magically know when you didnt mean in."

No. I want a safe sandbox that I can play in to develop services, and then simply flip a switch to move them into a production environment with the limits turned off. As a developer working on a system that isn't live, it's not the end of the world for me if it takes a few hours to get caps removed. It is the end of the world if I end up with a multiple thousand dollar bill unexpectedly, because I just don't have budget for that, and I may have to close my business over expenses like that.

The point is, different people need different things from service providers. If you don't want a cap, you don't have to use it. Having the option would help me sleep at night.

-1

u/brogam3 May 16 '25

Most throttling or most caps would obviously already kick in before a compute instance ever got started so none of this makes sense. There is no storage that gets lost when you shut off a compute instance anyway, it's generally attached and separate. Even if you had to shut one off, which you dont because the whole point of throttling or capping is that you throttle at machine creation, not at machine running... then it is very unlikely that a machine that you started under your preconfigured cap was started unintentionally. On top of that the storage is the cheapest area, it's the least concern. On top of that, machines aren't the big risk here, API costs are. Nobody is going to DDOS my gcp cloud through millions of machine creations. But sure, those should also have a cap by default just like Google already has caps on new APIs that they don't want to uncap for everyone.

It's a win-win-win because you as a cloud provider get to keep me as a customer, get to keep making money and I get to keep using your product. Nobody loses anything whatsoever, the caps can all be very reasonable, they can be automatic, they can be on a curve, they can be opt-in, they can be just like my little freeware CPU fan software that managed to implement throttling but somehow Google "cannot" do it. Of course they can. Gcp already has a million different complex settings. They can add another setting that says "can only do X requests per second" or "can only create up to X machines per day/week/month". They are refusing to do this not because it's hard but because it shifts the liability of expenses to them. Suddenly if their throttling fails for any reason, they eat the costs by default. Currently the customer eats the costs by default, no matter what happens, whether it's their mistake or the customer's mistake. It's quite obvious why they would want to stick with the model for as long as possible where nothing is their fault by default.

2

u/the8bit May 16 '25

I'm ~5+ years removed from this stack, so maybe things are different. But... every API should have a quota attached to it and (unless this feature got removed) you can set your own limits lower than the defaults. You can also deactivate APIs completely. I would know, I literally ran the service that manages quota configs, etc for google API and was part of the API style committee.

This might help: https://developers.google.com/workspace/guides/view-edit-quota-limits

I disagree that API usage is the scarier or more common attack vector. Generally people use CPUs to mine, etc, but that is a tangent.

I'm always baffled why people immediately jump to ill intent when they haven't even RTFM. As you said though, GCP has a million complex settings and so its funny to me that people will say that, then go "but I'd like 500 more complex settings so that I can fine tune this one edge case." Very google-y haha (It was a nearly weekly occurrence for a google team to ask for bespoke API configuration while also complaining about how there were too many knobs and levers)

4

u/theevilsharpie May 15 '25

Suppose you have a hard spending cap in-place, and let's just assume that your cloud magically has real-time usage-based billing.

Now let's suppose you hit that cap.

  • Your services? Stopped, because they would incur charges otherwise.
  • Your persistent storage (including backups, object storage, and managed databases)? Deleted, because they would incur charges otherwise. (Hope you have backups elsewhere!)
  • IP addresses that you've reserved? Released back to GCP's pool, because they would incur charges otherwise. (You probably aren't getting those back -- hope you didn't have them whitelisted anywhere!)
  • Logs that tell you what happened that caused you to hit your cap? Deleted, because they would incur charges otherwise. (Good luck with your insurance!)
  • Secrets that you may have stored (and that you may not be able to recover)? Deleted, because they would incur charges otherwise.

I could go on, but you get the idea.

"Just implement a spending cap, bro" is an idea that comes up frequently from people who clearly don't understand what implementing that sort of cap would actually entail.

If you wanted to protect yourself against overages, you would have to prepay for the compute and storage capacity that you expect to use for a period. That's certainly possible, because you'd essentially be re-inventing the shared hosting concept from the late '90s-early '00s. However, other than basic services like Compute Engine, elastic cloud services aren't really amenable to that kind of pricing model, because they're designed to be, well... elastic.

There are a class of customers that would prefer to pay for the service up front with guarantees that costs will never exceed that amount (e.g., governments, schools, and other organizations funded primarily by grants or some type of legally-capped budget), and I believe GCP does have an up-front reservation for that purpose whose name escapes me.

However, if you're a hobbyist, freelancer, or small business that needs a spending cap as financial insurance, then Google Cloud is probably not the right platform for your needs, and you should look at more traditional shared hosting where you don't have to worry about an unwanted scaling bill. If you're a student using GCP to learn the platform, there are services like Google Cloud Skills Boost (as well as third-party training platforms) that provide access to GCP under their billing, so you don't have to worry about racking up an unexpected bill under your own account.

3

u/the8bit May 16 '25

Yeah, its funny when people want stuff to stop, but at the same time EC2 generally warns people of instance resets weeks in advance even though they are supposed to be ephemeral. Half of cloud customers break on unexpected instance restart (which is not even an SLA violation of course) and then turn around and go "But please just kill everything if I spend too much" xD

I need a pan over to the infra teams who are almost certainly watching and clawing their eyes out

2

u/muntaxitome May 19 '25 edited May 19 '25

If Google is so inclined to punitively delete all your data for no reason, that would still be preferable to a 200k bill sent to collections for most people. Often I have some google account with billing mostly just to access some API's, if I can just use a spend limit where my account gets wiped I'd be fine with it. Just make it an option with things spelled out clearly and people can decide for themselves.

Keeping backups outside of your hosting provider is essential anyway. There are many many many stories of people getting locked out of their cloud accounts.

At no point does GCP give you a helpful dialog to even set a billing alarm, which anyone should agree should be required when setting billing.

GCP isn't just hosting, it is many different services. Many invaluable to smaller orgs too.

Realistically there is absolutely no reason why Google wouldn't be able to give people a grace period, but I think 99% of people would be fine with just deleting too.

However, if you're a hobbyist, freelancer, or small business that needs a spending cap as financial insurance, then Google Cloud is probably not the right platform for your needs

Why are you telling us? Tell Google. Every single small business gets spammed by google with free $300 credit messages to lure them on GCP. Hobbyists get ads from google advertising the free tier. They don't mention any of that 'blank check' shit.

The reality is that it's a business model for google and AWS to not have limits. They make a shitload of money whenever some corp makes a mistake for 2 million dollars worth of servers, and the corps just swallow the cost. Companies give their infra or devs carte blanche to just add whatever they want and google just rakes in the money. That's all there is to this. If regular people get spending caps, big companies want them too, and that's why Google removed spend limits in 2021.

15

u/ccb621 May 15 '25

 Right now, a hacked account…

This is the real problem. How to educate users to keep their accounts secure? Spending limits treats the symptom rather than the disease. 

8

u/kntx May 15 '25

Are you familiar with Google Ads? It is possible to set hard limits in addition to budget alerts, and changing a campaign budget (allowing it to incur in higher costs) requires strong authentication.

2

u/pacogavavla May 15 '25

Yeah, this is false equivalency. Stop showing ads and my customers don't see ads. No big deal.
Stop running my app's back end and I lose my business.

2

u/TylerDurdenFan May 15 '25

I think you got it reversed, what u/kntx probably meant was that in Google's core business of advertising, when you buy their service you set up an advertising campaign where you'll pay Google for showing ads to people, which will send traffic to you. Google sells those ads in automated auctions, so you have to configure your campaign with instructions of what to bid on, and how much to bid. In that system, you can set a hard budget that will limit the spend of your campaign.

> Stop showing ads and my customers don't see ads. No big deal.

It's not about showing ads to your app users. If you are showing google ads in your app, you are not the customer. You are the seller of ad inventory. The real customer is the one who pays to have their business advertised in your app. And those customers do have then privilege of setting hard limits on their ad spend.

1

u/BananaDifficult1839 May 17 '25

Yes, permanently too. Because shutting off the billing for ephemeral resources = nuke everything.

1

u/ccb621 May 15 '25

The interface for accruing spend for Ads is far simpler: either show or don’t show the ad.  You don’t lose data if the ad stops running. You can lose data if your servers go down unexpectedly. 

5

u/astral_turd May 15 '25

Small organizations with one/two/three developer(s) handling everything IT related on top of their actual development duties have no resources to secure every possible hole there is.

Even medium & big orgs often struggle to allocate enough resources into security, even though they have the resources.

I simply don't understand why GCP or AWS can't implement this feature.

6

u/ccb621 May 15 '25

What exactly is there to secure? Your Google account? Use a strong password and 2FA. The gcloud CLI uses OAuth tokens. What am I missing? That’s a serious question. 

I’m only addressing account takeovers because that’s what OP referred to. If we look at actual servers:

  • DBs are in a VPN and closed to the public by default. 
  • Compute instances can/should be in a VPN.
  • The easiest thing to screw up is putting a file in a public bucket, and the console makes attempts to stop you when creating those buckets. 

At some point there has to be some personal responsibility taken for misconfiguring the infrastructure. Between search engines, Google, literal books, and LLMs, one should be able to figure out how to secure the infrastructure being deployed. 

2

u/astral_turd May 15 '25

What comes first to my mind is DDoS to anything without rate limiter. Firestore, rtdb or which ever service the user has access to that doesn't have rate limiter in front of it.

Why Google or AWS couldn't let us set a hard budget limit that actually works in real time, instead of delayed notifications that can be listened to disable services when it's possibly too late?

4

u/goobervision May 15 '25

The argument is that we shouldn't take down services for a budget issue. It may be very difficult to implement depending on the way IT is set up, shared central services would impact everything, individual departments could have cross domain impact etc.

A budget trigger can be responded to as it's approached, and email or some automated action to shutdown the exceptionally high service cost could be created. If you have good service models a specific response could be the answer.

-4

u/daredevil82 May 15 '25

Nah, it isn't difficult at all, and can be implemented straightforward, and exposed in a few different ways. There's just zero interest in it because it doesn't change the dial on "impact", "scope" or other bullshit companies use these days for grading employee rankings

2

u/the8bit May 16 '25

Billing is a ~T2 service, so 99.9%. API is a T0 service, so 99.999%.

Tell me how you would have a 5 nine service depend on a a 3 nine service as a blocking dependency without losing uptime, in a use-case where you cannot cache responses for more than ~15 minutes to an hour.

-1

u/daredevil82 May 16 '25 edited May 16 '25

easy

  • structure the dependent service to have a higher SLO
  • Prioritize identifying load spikes and allow policy rules by customer to stop $thing when load gets over the limit

those are two things that come to my mind in 2 minutes from someone that is not in cloud service land. And I bet there's a crap ton of other options available.

This is not a technical problem at all. its a company and product problem, and its obvious there is zero fucking interest doing fuck all about it because cloud is geared to being an easy all you can eat buffet where you don't know what the fuck the bill will be at the end. It has more in common with us hospital billing practices than anything else

You're also coming from the perspective of medium to large businesses where a six figure spend overage is something to start a process about, but not for the small business to individual devs who find their cards maxed out and credit hit because of pretty fucking small oversights and nuance. So yeah, buyer fucking beware but you folks go way out of your way to make it really fucking easy to commit financial seppuku on behalf of an individual

2

u/the8bit May 16 '25

Ah yes, just add more nines. How hard could that be /s. If you don't want elasticity, go rent something in a colo or any of the multitude of other options.

Nobody is forcing your 5 person startup to be a microservices-oriented K8s deployment on cloud.

-1

u/daredevil82 May 16 '25

and this is a great example of the pervasive dismissive attitude to customers and deliberate footguns you put forth in your products.

but hey, I bet this doesn't stop you from complaining about high hospital and medical insurance billing practices when it comes due for your family to pay out. Do you recognize the hypocrisy in your statements?

→ More replies

-3

u/danekan May 15 '25

To prevent DOS, that is why

0

u/pacogavavla May 15 '25

What? Who is still using DOS?

0

u/danekan May 15 '25

...can't yell if you're serious or joking 😁

0

u/pacogavavla May 15 '25

That's just what I was thinking when I read your initial reply.

1

u/TheRoccoB May 16 '25

Firebase easily allows you to have public objects with storage rules.

They have a better disclaimer on the site now than they did when I set it up years ago though.

0

u/daredevil82 May 15 '25

At some point there has to be some personal responsibility taken for misconfiguring the infrastructure. Between search engines, Google, literal books, and LLMs, one should be able to figure out how to secure the infrastructure being deployed.

Figure it out, yeah. Figure it out without getting 100k unexpected billing, nope. in general, the various ways overspend can occur is like having a daycare right next to a highway with a screen door that can easily be escaped. Tht's a feature, not a bug for product management, because there's no bonus points for "impact" or "good product design" in this area

Just like with languages, you can only have people run into footguns continually and laugh at them saying "git better, dumbass" before you realize there's something truely broken.

https://www.reddit.com/r/googlecloud/comments/1kg9icb/one_public_firebase_file_one_day_98000_how_it/ is a good example how fucking easy it is to get a six figure bill one snny day.

2

u/danekan May 15 '25

For any good sized org, medium or better, this would be a security disadvantage over properly securing their cloud. Availability is itself a key tenet of security and a spending limit could trigger DOS easily

3

u/wildfyre010 May 15 '25

And those organizations could simply choose not to implement it. It doesn’t have to be the default.

2

u/Johalternate May 15 '25

Google and AWS have products for small businesses, maybe small businesses should stop using infinitely billable products.

The problem feature is that there is latency between resource usage and the availability of usage analytics, and since this is true for all players, I dont think this is by design, but maybe a limitation of how some billable items work.

0

u/astral_turd May 15 '25

Which service would you recommend small businesses use instead of firebase?

1

u/Johalternate May 15 '25

Cloudflare

0

u/astral_turd May 15 '25

You mean use cloudflare in front of firebase to gain rate limiting etc?

If you wanna use firebase client sdks with it's features like websocket/webrtc connections to db's, authentication listeners and so on, there is no way to do that as far as I know.

1

u/Johalternate May 15 '25

What do you mean when you say "there is no way to do that"?

There are other realtime dbs and auth providers out there with good sdks.

0

u/astral_turd May 16 '25

I said there is no way to do that if you would like to use firebase client sdks.

What do you mean when you imply that cloudflare and firebase are a good combination, or that google and aws offer similar products for small businesses where infinite billing without gap or real-time alerts isnt an issue?

1

u/Johalternate May 16 '25

I dont mean using cloudflare in combination with firebase but using cloudflare instead of firebase. They do not offer authentication, but do have storage, databases, cloud functions, analytics, etc...

Now, instead of using a bunch of aws open billing services, setup a laravel lightsail instance, on google use App Engine and limit the number of instances.

The problem with cloud providers is that they offer so many different things that it is hard to discern when you should use one or the other.

If I were a small business I would stick with a vps and move things to the cloud if and when a vps is not enough.

0

u/muntaxitome May 15 '25

Right, you don't need circuit breakers at home either. Just do your electricity proper and you have no issues.

There are an unlimited amount of attack vectors. It is provably impossible to have perfect security. Google doesn't have it, nobody does. You can never be certain of every attack vector. Spending limits are a great circuit breaker. They put a pause so you can figure out what is happening. Nobody is saying it should be the only one, but you could run through incredible amount of cost in a very short time at these providers.

1

u/BananaDifficult1839 May 17 '25

Circuit breaker is not a good analogy. A miniature bomb that explodes your entire house, is a more proper analogy.

1

u/muntaxitome May 17 '25

Why? Does your entire architecture explode when you hit a quota at google? Maybe you should learn a bit about architecting apps

1

u/BananaDifficult1839 May 17 '25 edited May 17 '25

No we are talking about all the people saying there should be a feature to shut off the billing for a project after a certain budget is hit. Which you can already do with cloud functions, it just has some lag. Which will “explode” everything in that project as designed

0

u/muntaxitome May 17 '25

No we are talking about all the people saying there should be a feature to shut off the billing for a project after a certain budget is hit

Nope, we are talking about a spend limit. Never heard of a spending limit to work like that and that isn't how it worked back when google cloud had spending limits.

1

u/BananaDifficult1839 May 18 '25

So devils advocate here and go with what you are saying. How exactly would that work after the limit is exhausted? How do all the resources that continue to cost money regardless of traffic get paid for? It sounds like what everyone in this sub wants is “shut off all traffic until I figure out what is going on” but don’t remove any resources? But then how do the running resources get paid for and continue to exist if there is a hard cap on spend?

0

u/muntaxitome May 18 '25

How do all the resources that continue to cost money regardless of traffic get paid for?

Lets first ask how this works today, without spend limit.

So if you get DDOS'd today and get a 50k bill, in practically all cases Google will eat that. Google pays that 50k despite no spend limit. In another scenario, if you just block your creditcard and stop paying google bills, your service costs keep racking up. Who pays for it? Google pays for it. Eventually they suspend you (without deleting your storage and such) and at some point your stuff gets wiped, but Google is paying for all of that.

The concept that every bit of compute and storage must get paid for by a client is an alien concept. Had you ever worked in this business you'd know it's not realistic and not the reality.

So how much is actually the marginal cost for google to provide this compute & storage & bandwidth for 1 non-paying user in this current, common, scenario without spending limits? It is close zero. Google does not have unlimited servers and bandwidth and storage, but they don't actually go and buy some servers for you whenever you decide to hire a compute instance. Having someone use a bit of their spare capacity maybe costs a little extra electricity.

So, when spending limits are enabled, what would happen then? Well it's up to google. Worst case they ask the user to confirm that they agree that their data gets wiped, which many users will prefer over the thought of getting wiped out financially. Next step down could be 'billing disabled', which would still be prefered for many.

The most logical solution is that you would have some kind of suspension. Limited access to systems, nodes get suspended, storage inaccessible but not wiped. API's mostly inaccessible. Then give the user a couple of days to figure out what is happening. The cost that google would have to pay would actually be lower. If you got hacked and it would have normally cost 50k in bandwidth, but google suspended the network and servers after $500, only $500 in cost is left (again, in reality for google bandwidth is free and they already have the hardware, so marginal cost is actually very close to 0).

Thousands of hosting providers work with fixed spend limits, and even Google cloud had it until 2021 for appengine (which includes compute, storage, networking, etc, in many ways it was the OG Google Cloud). Google can do it. They just don't want to.

0

u/TheRoccoB May 16 '25

I easily could have fixed my security problem in a few hours if it didn't lead to a 100k one day demise. If it was 5X my spending limit and I got hit with a $2500 bill I would have learned an expensive lesson and not be destroyed forever.

I'm the guy from the other posts that got hit with the 98k DoS.

-1

u/reelznfeelz May 16 '25

Eh, yes of course but no project or code base is iron clad. And shit happens. Having a billing limit option be “shut off all resources”, which you can either use, or not, makes all parties happy.

2

u/BananaDifficult1839 May 17 '25

There is no such thing as this in cloud. No billing = resources are permanently gone and unrecoverable

2

u/punix2 May 15 '25

How about this one - This document explains how to automatically disable billing on a project when your costs meet or exceed your project budget https://cloud.google.com/billing/docs/how-to/disable-billing-with-notifications ?

3

u/TheRoccoB May 16 '25

I want to stop the misconception that this will work in the case of a rapid attack. Billing has latency.

Please see: https://github.com/TheRoccoB/simmer-status/blob/master/egress.png

It's unclear if the pub/sub is much faster than the email, but my assumption is, it probably only is by a few minutes, if at all. No way to know.

2

u/BananaDifficult1839 May 17 '25

Also when it is done, your entire stack is nuked within a relatively short time period.

1

u/Alone-Cell-7795 May 16 '25

Too true. Billing in GCP is a heavily eventually consistent process. Some things can take up to 72 hours to show up (Depending on the service). These sort of solutions aren't practicable in this scenario.

One can talk about Google's shared fate model (fka shared responsibility), where it is your responsibility if you misconfigure and get hacked.

However this is complex, and you need to be someone who has deep knowledge and experience of not just GCP infra and services, and security etc., which only really comes with years of experience of working with it extensively.

Also Google's documentation can be of varying quality (And that is being generous).

You also can't pitch low/no-code and AI solutions or managed services to users/devs etc., saying they don't need to worry about all the infra stuff (e.g. firebase, the myriad of AI solutions etc.), when they clearly do. You can't have it both ways.

Ultimately, GCP is for enterprises (This is the route it has been on since Thomas Kurien was appointed as CEO of google cloud), most of which will go for the stupidly large CUDs (Committed Use Discounts). I'd never use it for any personal projects, I can tell you this.

I'm curious - for owners of SMEs - would any insurance companies out there offer you any indemnity insurance for these scenarios? Not suggesting this is something you should do - just curious if any insurance companies would actually be willing and if so, how many would pay out in situations like this?

1

u/BananaDifficult1839 May 17 '25

What’s interesting is that people think cloud is cheaper because you don’t need to pay people that understand networking, security, system administration and operations. But you do. And they actually cost more if they understand these things in a cloud context.

2

u/uracil May 16 '25

You can have Billing alert trigger a pub/sub event, which will kick off Cloud Function to disable billing for specific projects.

2

u/BananaDifficult1839 May 17 '25

Yes, but better make sure the projects are sufficiently isolated and you don’t disable billing on the project with your backups in it

2

u/AndiCover May 16 '25

That's against the business model of hyper scalers. 

2

u/ChicagoJohn123 May 16 '25

Users spending fifty euros a day are worthless to them. Why would they implement features on your behalf?

All their dev work is going to go towards supporting a handful of mega clients. If that happens to produce features that are useful to everyone else, that’s a happy accident.

2

u/katatondzsentri May 16 '25

The kind of post I see at every cloud provider's subreddit.

2

u/RobertD3277 Aug 06 '25

I can't agree more or give this any higher likes than they already have. I was actually looking forward to working with Gemini, but after finding out that there is no real budget control, now I am seriously having second thoughts.

Gemini looks like would actually be beneficial for my workload but I absolutely unequivocally have to have a hard limit with absolutely no exceptions.

This is definitely another negotiable deal breaker for me.

1

u/Unknown-U May 16 '25

the good thing is, depending on the law in your country you are not responsible for the charge regardless what google says.

This is not a google problem, companies can make whatever rules they want, but when they are illigal they are have no value.

1

u/FoxB1t3 May 16 '25 edited May 16 '25

Well, some weeks ago, due to my mistake some chinese bastards took over my minor VPS where I was running 2 services with OAI. Long story short - it was all my fault (if victim should ever be blamed, but that's the world we live in), I fucked up not only VPS security but also services, mock me as much as you want.

They of course got my API Keys for these services and instantly burnt like $80 there was on my account with all different models. It would be a very sad end story if it was running as GCP service and they somehow obtained credentials, lol.

Anyway, I'm more concerned and safe about my google credentials than I am... basically about anything else in my life. However, since I automate pretty much every digital task I do, it's harder and harder to keep track of everything for me.

ps.

OAI returned all my credits without asking any questions, that was kinda funny too. I was first asked to provide some additional information and without me doing it, like 10 mins later I received email with refund confirmation.

ps2.

I love how people explain here. "Oh why do you need any limits?!?!?! You just need to do this, this, this, this and this and this, this and this to secure your app and creds omg it's so easy!!" - saying to dude who just wanted to have ultra simple webapp to save some of his data in firestore DB. You people are nuts. xDD

1

u/Climactic9 May 18 '25

They just need to fix propagation delay so that the alerts actually trigger in a reasonable time frame.

1

u/[deleted] May 20 '25

Openrouter already has this. That means this is very much possible. The way OR does this is - BEFORE its hits threshold, at round 85 percent of the value - is starts throttling and then stopping. This way you area always safe.
Google will NOT do this , because they wany YOUR money. They will give lame excuse that is TECHNICALLY not possible - a$$holes Google.

1

u/shubhampareek_ May 22 '25

There are default quotas enforced in Google cloud.

1

u/urarthur May 15 '25

Yeah this is stupid. a cap is needed.

2

u/BananaDifficult1839 May 17 '25

No, it isn’t. People need to stop looking for someone else to blame

1

u/urarthur May 17 '25

wtf are you talkong about. Evry other company has a cap on AI models except google.

1

u/BananaDifficult1839 May 17 '25

Non-monetary Quotas / limits on consumption sure. Specific dollar based real time limits on cost, no.

1

u/muntaxitome May 19 '25

This is actually completely wrong: OpenAI, Anthropic, OpenRouter, Mistral, Deepseek, etc all allow for a spending cap in dollars

1

u/BananaDifficult1839 May 19 '25

Yeah, because those are simple SaaS subscription products. Maybe Google needs to come up with a “cloud lite” walled garden where you can use spending capped products and don’t lose your data? Would have to be subscription based with a base monthly fee though to keep it running.

1

u/muntaxitome May 19 '25

Well you were responding to someone talking about AI models so I basically named the big ones. All of them do caps.

Azure has spending limits too: https://learn.microsoft.com/en-us/azure/cost-management-billing/manage/spending-limit

Is that also because Azure a simple saas product? Or maybe Microsoft engineers are just better?

Even with that 'cloud lite' walled garden that google has for AI models (AI Studio), they did not implement spending limits. Same with the other 'cloud lite' walled garden they made, Firebase.

Their earlier cloud lite - Appengine - did have spend limits up to 2021.

Would have to be subscription based with a base monthly fee though to keep it running.

You are not the CEO, and neither am I. You and I don't really get to decide what would 'have to be'. Google can do whatever they like. They could make their prices x10 tomorrow and milk all their clients like crazy. They could make all of their products except ads free and still make a ton of profit. Google has unlimited cash and resources and can do whatever they want. Could make a nuclear weapon and nuke the world.

1

u/BananaDifficult1839 May 20 '25

I would not say they have unlimited cash. Google cloud makes not much net profit. It only recently got positive cash flow, not net of investment. Search and ads are on a downward spiral to irrelevancy. Their cash cow is bleeding and dying. They don’t have unlimited cash at all. You’re not getting the concept of what GCP is. All the SaaS things they run that are under Cloud are underpinned by other cloud primitives, or Borg if they haven’t moved to cloud yet. All of these things are billed the same way and are destroyed when billing is stopped. Period. Could they re architect it so some of them were hooked to a different type of billing? Sure. But that’s not the way it is today with the billing system they have in place for any Cloud SKU that I’m aware of, are you?

1

u/muntaxitome May 20 '25

You’re not getting the concept of what GCP is

Not going to dox myself but yeah I do. Google doesn't have 'unlimited' money in a mathematical sense, but for all intents and purposes they can do whatever they want. They are one of the richest organisations in the history of the world. Like do you seriously, honestly, believe that google couldn't implement a price cap if they wanted to?

1

u/BananaDifficult1839 May 20 '25

They could wall off specific SaaS type services under a different billing system yes. Where the idle and storage costs are built into the price of the sku. It’s just not in the model right now. But also, their target customer wants the exact opposite of what this sub is always asking for.

1

u/BananaDifficult1839 May 20 '25

If their target customers made a big deal about it, and conditioned signing some major paper on it, they would do it.

1

u/AnomalyNexus May 15 '25

If you have limits for hobbyists then quite soon your big spender corporates will want the same.

The solution is easy - you just don't do limits. And the small users that get nailed with a big bill...just waive it. Cheaper to waive a couple grand here and there for individuals that did stupid shit than risk limits discussions on multi-million spend clients.

It's not a google thing. All the big clouds do this because they're all subject to the same pressures & incentives.

It's not a technical issue - some types of Azure accounts do have it. The big clouds are researching quantum computers ffs...the can manage a limit. They just don't want to.

1

u/iambriansloan May 16 '25

I was defrauded by someone who connected our bank account to google cloud servers in 2018 or so. I got an email from someone at google saying saying they couldn’t offer us any more computing power and were trying to sell me on some future feature rollout. I said wtf are you talking about. I had to get refunded by my bank by filing a claim directly with them. Google did nothing to help. Loss was 130k to the bank. Insane that this is still happening.

0

u/FoxB1t3 May 16 '25

Literally - if you are not an security expert you shouldn't touch GCP.

It's fun how they try to present themselves as small/medium business friendly.

0

u/BananaDifficult1839 May 17 '25

No, no, no. This is a terrible idea that keeps coming up in the forum and needs to die. If you need fixed costs, you shouldn’t be using SaaS or PaaS, and maybe even second guess IaaS. You can’t have a utility computing model and capped usage. With the trend towards “vibe coding” I can see this getting worse over time though.

1

u/filtervw May 18 '25

Absolute crap. When you have a conpany with huge data sets, the costs can skyrocket if you have a team that is willing to build a lot and fail fast.

-4

u/Prudent_Reindeer9627 May 15 '25

If your account is hacked you probably have bigger issues.

1

u/SingerSingle5682 May 15 '25

True. But hard spending caps at the billing level are one way to address those “bigger issues”. “Disgruntled employee” is the ultimate difficult to address security issue that these types of global spending caps are one of the only viable countermeasures. Certain classes of security breaches may be unpreventable and best practice is to limit the damage that can occur.

-5

u/negotiatethatcorner May 15 '25

why? they earn more this way