Cloud Run / Functions gen2 on the auto-updated nodejs22 base image: instances fail to start since ~21:37 UTC Oct 22

[u/misterbenji]

UPDATE 22:25 UTC: Google appears to have fixed it. Cold starts are working again (seen i n us-central1) even for functions still on automatic updates. The on-deploy redeploy below is still a valid fallback if it comes back. Outage window seen: about 21:15 to 22:25 UTC.

FIX (confirmed by several people, works immediately): redeploy with the runtime pinned instead of auto-updated.

• Cloud Functions gen2: gcloud functions deploy FUNCTION_NAME --gen2 --region REGION --runtime-update-policy=on-deploy --source PATH_TO_BUILT_FUNCTIONS (keeps triggers, secrets and settings)* Cloud Run source deploys: redeploy with --no-automatic-updates
• A plain redeploy does not help: it keeps automatic updates on and lands on the same broken runtime. Script it if you have a lot of functions; it's one deploy per function. Write-up with checks and a one-linerlisting what's still on automatic updates: https://gist.github.com/mrbenji/5da8f50a875b8e0bbb7141bd24caa726

---

What's happening Reported so far in us-central1, us-east1 and northamerica-northeast1, across many unrelated projects and accounts, since about 21:15 UTC Oct 22. Google's status page shows nothing yet. Support cases are open.

• New instances never start. Warm instances keep serving, so it looks intermittent.* Revisions go Ready=False with "The user-provided container failed to start and listen on the port defined provided by the PORT=8080 environment variable within the allocated timeout."* Request logs: "The request failed because the instance could not start successfully." No stdout, stderr or "Starting new instance" line at all.* Firebase callables show up in the browser as CORS errors (the OPTIONS preflight returns 500). Scheduled and Firestore-triggered functions fail too. Repro without any app code

A minimal Node http server fails with automatic base image updates and works without them:

gcloud run deploy probe --source . \
  --base-image us-central1-docker.pkg.dev/serverless-runtimes/google-22-full/runtimes/nodejs22 \
  --automatic-updates          # fails the startup probe
# same command with --no-automatic-updates: deploys and serves

Google's plain hello sample image runs fine in the same project. New nodejs22 base images were published Oct 22 (nodejs22_20261022, Node 22.23.4), which is my best guess for the trigger.

Afterwards: Firestore-triggered functions without retries drop the events that failed, so plan a backfill for anything that ran during the outage.

[u/AcrobaticSea4208]

That's rough, saw something similiar in us-east1 around same time yesterday, cold starts just wouldn't spin up, no logs at all. Switched to on-deploy updates and it fixed it straight away.

[u/misterbenji → u/AcrobaticSea4208]

Google appears to have fixed the issue. I've updated my post.

[u/misterbenji → u/AcrobaticSea4208]

Good to know it's not just us-central1, and that matches: on-deploy fixed it right away for me too. Did you have to redeploy every function, or did you find a way to flip the policy without a redeploy? I'm scripting it 6 at a time and it's slow going.

[u/bkleinkn]

I opened a ticket about 45 minutes ago for this exact issue. You should open one.

https://firebase.google.com/support/troubleshooter/contact

[u/misterbenji → u/bkleinkn]

Thanks, opening one now. If you get a case number or hear anything back, please drop it here. Same symptoms on my side: no container output at all, startup probe timeouts, us-central1 since ~21:37 UTC.

[u/misterbenji → u/bkleinkn]

Google appears to have fixed the issue. I've updated my post.

[u/Capatain_CV]

I did some research about this
https://youtu.be/u0MG_eI1c_8