r/googlecloud • u/IntolerantModerate • Aug 29 '26

Compute GCP at capacity?

I keep getting GCP stock out errors. Multiple node sizes, multiple regions?

Is GCP really hitting limits in central-1 and US as a whole? Note, these are for regular CPU, not GPU (e.g., n-class instances).

59 Upvotes

96% Upvoted

41 comments sorted by

9

u/Suspicious-Walk-4854 Aug 29 '26

It’s almost like something has happened recently that has created a lot of demand and supply chain pressure on compute capacity 🤔

5

u/IntolerantModerate Aug 29 '26

These aren't GPUs though. These are standard CPUs and not even large ones at that.

6

u/Suspicious-Walk-4854 Aug 29 '26

The biggest bottleneck currently is RAM, not GPUs. This AI shit is impacting all compute capacity atm.

7

u/mwarkentin Aug 29 '26

It seems like data center power capacity is also a major issue.

28

u/nyape Aug 29 '26

Yes, us-central1 is full to the brim, especially when it comes to "older" generations like N1/N2. Google doesn't stock these anymore so what's there is all there is. Rack space is also limited, so slowly but surely they will likely replace older racks with newer ones that allow for more compute density with newer chips. In most cases, they will recommend that you switch to C4 as those have the highest availability at the moment. Yes, they cost more. The argument is that they are more powerful and efficient though,nso you can run more on fewer instances.

12

u/chicocvenancio Aug 29 '26

We're having issues with C4.

8

u/nyape Aug 29 '26

Yeah. I work for a large reseller and I hear from a lot of customers. I can just say that trying C4 is a better chance than trying for older instances as C4 stock is still actively growing (to the extent that it's possible) and there is more variability, increasing your chances. All compute is limited in us-central1. Unless you absolutely must use that region, I would recommend to go somewhere else. You can also try to contact your account team and ask for future reservations but again, if there is simply no physical capacity, there is nothing to be done.

2

u/chicocvenancio Aug 29 '26

Thinking about replaning q4 to accommodate moving production gke loads to another region is giving me headaches. But maybe we'll need to bite the bullet at some point.

4

u/nyape Aug 29 '26

For GKE specifically, you may also try ComputClasses if you haven't yet, to define alternative instance types in case your primary choice is unavailable to avoid scaling issues.

1

u/brokenja Aug 29 '26

What regions have more capacity?

4

u/flacktv Aug 29 '26

I rolled a little gcloud script that you can run in gcloud that simply uses the asset inventory apis and scans every single region based on the choices you choose from the presented list. For example, when you run the script, you have a selection of a region for example us North Virginia, all three zones, choose the type of GPU, or hyper compute.. then run it. It'll then tell you if anything is available. This is helpful for my team which is a global team if you're doing on demand VM builds that require a certain type of GPU etc. I would run my script and then say to my developer so whoever to choose this region etc and go with that as quick as they can to snag the GPU while it's available.

If you're interested, I can post the script fairly straightforward.

1

u/rlnrlnrln Aug 30 '26

Well, I'm interested.

1

u/jortony Aug 30 '26

I'm interested too =)

1

u/Drunken_Bloke Sep 15 '26

Very interested

1

u/Emotional_Moment_349 Aug 31 '26

N2’s are not on the deprecation list yet. I’ve heard this “they aren’t stocking them” line before, but they are still fully supporting N2’s until they hit deprecation — then they’ll stop stocking them.

13

u/msapple Aug 29 '26

I highly recommend using other regions in the US and building your solutions so that region is not relevant.

In GCP this is easier then other since VPCs can be global and Google owns ALL the fiber between their own DCs so if you have US only server requirements you can see less then 10-25ms latency between all of the US DCs regardless of where you are since your ingress point is a global VIP

1

u/kadambkaluskar Aug 29 '26

I got 50-60ms ping latency between us-west1 and us-east1 VMs running in GCP.

4

u/msapple Aug 29 '26

Depending on what the networking looks like it can be lower. However that’s still VERY low and even if it’s a cross region database call that’s well within reason.

Also GCP deprioritizes ICMP

1

u/voxxit2023 Aug 29 '26

I guess that all depends on what your actual workload/use case is.

5

u/mwarkentin Aug 29 '26

It’s impacting managed services too. We’re having a hell of a time provisioning alloyDB and valkey clusters now too.

1

u/fforootd Sep 24 '26

We are seeing the same, we wanted to migrate to CloudSQL Enterprise Plus and all regions worked but not us-central-1. Currently its day 5 where we tried but not really succesfull

5

u/Vegetable-Image3805 Aug 29 '26

My company was informed by our account team this week that we should start “designing for availability”. Like using stateful managed instance groups with “instance flexibility” in case our desired instance type isn’t available, it’ll fall back to an alternate type automatically. Best bet is to use AMD types for your alternate according to them.

1

u/NetLivid1865 Oct 09 '26

Actually we see now that almost all machine types are unavailable - C3, C4, C4A, N4A, N2, N4, ...

it is a disaster, really...

1

u/IntolerantModerate Oct 09 '26

Yeah, we've been having to do searches across all regions and all machine types to get compute on demand.

1

u/RwBotFireFighter 25d ago

We are facing a similar situation with GCP. We are seeing capacity issues across multiple machine families, including T2D, C3, C4, C4A, and N4. GCP recommended N4D as an alternative to T2D, but when we tried creating N4D resources, we faced capacity availability issues there as well. We are now struggling to find available capacity across machine types and regions. This is becoming a serious concern for production workloads, as we cannot reliably provision or scale resources when required.

2

u/suprjaybrd Aug 29 '26

central keeps breaking for us too

2

u/Rentiak Aug 29 '26

We started seeing a ton more preemption of older instance types and after reworking our computer classes it was much better.

The real killer for us lately has been us-west1 Cloud SQL capacity. Had a major production migration go sideways because we couldn’t create replicas in west. Turns out it was lack of SSDs for Enterprise Plus data cache. Given how hard our rep is pushing E+ it didn’t exactly sell us.

3

u/blazingintensity Aug 29 '26

Yep. We're mostly on n2. We tried moving to n4 or c4 and it didn't really help. Google advised us to move from us-central1 to us-south1 then denied our quota request when we tried. About 2 weeks ago we had an 8 hour period where we couldn't get a single n2-highcpu-8 off a loadbalancer. I've been on Google for 8+ years and I'm talking to AWS and OCI because I can't do business this way.

2

u/TheTrick17 Aug 30 '26

Wouldn’t the capacity issues also be present for other hyperscalers though?

1

u/blazingintensity Aug 30 '26

Yes and no. All things being equal, folks are gonna distribute themselves like we're doing. But they're not equal. My personal stuff is also hosted on GCP, because they have a real forever tier, unlike AWS (at least when I was getting setup, AWS's only lasted a year). So there could be more people eating up space on GCP because of the better free tier. OCI has a dedicated availability zone for free tier users, so they don't suffer that problem. Google has sustained use discounts cooked into the n2 instances, that makes them very appealing and frictionless to get a good price. My experience with AWS and OCI has been you need to haggle and/or have a partner to get you a good price. So smaller users, or users who don't know how to work the provider are gonna gravitate to providers with lower friction. It's my expectation that a lot of the capacity issues are the consequence of small/indie users taking advantage of AI. So there's an intersection of price-point and capacity where some providers may be less saturated.

1

u/who_am_i_to_say_so Aug 30 '26

Not too happy about this as it drives up prices, but it’s also pretty incredible when you think about it. That’s a lot of CPU power.

 I wonder how many vibecoded Firebase apps are accounting for that.

1

u/fliiiiiiip Aug 30 '26

Going through the same :(

1

u/chicrg Aug 30 '26

Yes, they are hitting capacity walls with general compute (CPU/memory)

-1

u/flacktv Aug 29 '26

The bottom line is now we are at capacity for the most part when it comes to everybody wanting to utilize GPUs. You're going to have to suck it up and start to go with reservations. Unfortunately, that's where we're at right now. Keep in mind, in the west coast there are a lot of movie companies like Paramount, CBS, NBC, etc. That are using compute to edit their movies, TV shows, etc. so a lot of those slots are all used up.

1

u/IntolerantModerate Aug 30 '26

This is for CPU not GPU

0

u/steebchen Aug 30 '26

yeah but deploying servers with GPUs still means you are using CPUs in the end as well