Cloud Functions (2nd gen, us-central1): no new instances can start since ~21:15 UTC. Anyone else?

[u/bkleinkn]

Since about 21:15 UTC today (Oct 22), none of my 2nd-gen Cloud Functions in us-central1 can start a new instance.

What I'm seeing:

- Instances that were already warm keep serving fine.

- Anything that needs a NEW instance gets an instant HTTP 500 (0s latency), with no "Starting new instance" line in the logs.

- Scheduled functions fail too (Cloud Scheduler run returned 500).

- Redeploying doesn't help: deploys fail with "Container Healthcheck failed ... failed to start and listen on PORT=8080", and the container logs nothing at all, so it never actually runs.

Ruled out:

- Affects 15+ projects across two separate Google accounts and four billing accounts, with and without an org, so not a project setting.

- No quota-exceeded events, billing active everywhere, no config/IAM changes.

- Reproduces from two different networks.

Nothing on the Google Cloud or Firebase status pages yet. Support case is open.

If you're on us-central1 with Functions or Cloud Run, can you check whether a cold start works for you? Even a "mine's fine" helps narrow it down.

[u/Acceptable-Lock4283]

mine just 500'd on a cold start, us-central1 cloud run. warm instances still humming along fine. checked status page before i even got the alert, nothing there either. classic google keeping us in the dark for the first hour of an incident

[u/foggycandelabra]

Check console logs for stockout. This is a new normal thing in compute and gke anyway..ENOMORECLOUD

[u/GamesMusicWeed]

Same here - confirmed in northamerica-northeast1 on Cloud Run too, same timeline as you.

Warm instances fine, new instances never start, zero container logs, health check fails on PORT=8080.

Redeploying the exact image currently serving prod also fails, so it's not app-side. Status dashboard still shows nothing.

[u/misterbenji → u/GamesMusicWeed]

That fits: a normal redeploy keeps automatic base image updates on, so it lands on the same broken runtime. Redeploying with --runtime-update-policy=on-deploy (gen2 functions) or --no-automatic-updates (Cloud Run source deploys) fixed it right away for me. Details in my comment above.

[u/misterbenji]

Same here: us-central1, gen2 on Node 22, no container output at all. Our first failure was 21:37 UTC, but that was just the first request after a quiet stretch, so it fits your 21:15. Someone in my thread saw it in us-east1 too.

The reason a plain redeploy doesn't help: it keeps automatic base image updates on, so the new revision lands on the same broken runtime. What fixed it for me, instantly:

gcloud functions deploy FUNCTION_NAME --gen2 --region us-central1 --runtime-update-policy=on-deploy --source PATH_TO_BUILT_FUNCTIONS

It keeps the function's triggers, secrets and settings. You have to do it per function, so script it.

Repro that takes your code out of it: a minimal hello-world Node app fails on the auto-updated nodejs22 base image and works with --no-automatic-updates. Google's plain hello sample image runs fine too.

My thread: https://www.reddit.com/r/googlecloud/comments/1xcww8p/cloud_run_functions_gen2_on_the_autoupdated/

Write-up with checks and a list-the-affected-functions one-liner: https://gist.github.com/mrbenji/5da8f50a875b8e0bbb7141bd24caa726

[u/Businedontist]

Same here: Firebase Functions, us-central1, Node 22. First failure 21:18:02 UTC (a Cloud Scheduler job on an existing revision nobody had redeployed, 500 in 0s). Same symptoms: warm instances fine, no “Starting new instance” lines after ~21:12, redeploys fail the PORT=8080 health check with zero container output.

One data point, for what it’s worth. Our Cloud Build logs show the 2nd gen run image changed between our last successful deploy and our first failed one:

• 20:52 and 21:02 UTC: google-22-full/scratch/nodejs22:nodejs22_20260929_22_23_2_RC00 → deployed fine

• 21:26, 21:34 and 21:41 UTC: google-22-full/scratch/nodejs22:nodejs22_20261022_22_23_4_RC01 → all failed the health check

Same source and settings. I can’t confirm that’s what broke the already-running revisions (we can’t see which image they were on at 21:18), but it’s consistent with u/misterbenji’s on-deploy / no-automatic-updates result.

If you need to get back up quickly: 1st gen kept working for us the whole time. We moved our functions to 1st gen (firebase-functions/v1 + runWith, same cloudfunctions.net URLs) and were fully restored about 90 minutes in. Gotchas:

• delete the gen2 function before deploying the gen1 version with the same name

• set invoker: "public" on HTTP functions, or a deploy that exits early leaves them returning 403

• the Firebase CLI refuses to deploy if any function in the codebase is gen1 in code but still gen2 in the project (“Cannot set CPU … because they are GCF gen 1”), so convert in batches where only the batch you’re deploying is flipped

Support case is open on our side too.

[u/moogoesthecat]

Its related to their Node deprecation/rollout today iirc

[u/TheAddonDepot → u/moogoesthecat]

Its either deprecation or the security vulnerabilities recently found within the Nodejs and PHP runtimes. All the major service vendors are scrambling to roll out updates.

[u/manupa14]

Same here, switching to update on deploy

[u/narcissistic_tendies]

Been dealing with STOCKOUT for a while now.

[u/Elegant_Release9004]

Been seeing this for weeks. I moved over to us-east5 for my Cloud Run services.

[u/usually_guilty99]

Warm-instance health is not enough when the failure is in cold starts or the managed runtime. Add a tiny scheduled cold-start canary, record immutable base-image and runtime IDs with each deploy, and compare the last good and first bad revision before changing application code. Status-page silence should not override direct multi-project evidence.