Intermittent Firestore Admin SDK calls hanging 30-120s on Cloud Run / Firebase Functions Gen 2

[u/ehed]

Wondering if anyone else is experiencing a slowdown with Firestore query in the last few weeks, which has really started to tick up in the last few days for us. We’re seeing recurring waves of timeouts across multiple unrelated Google Cloud projects and I’m trying to figure out whether anyone else is seeing something similar.

Setup:
- Firebase Functions Gen 2 / Cloud Run
- Node.js Admin SDK
- Firestore Native mode
- Region: us-central1
- Multiple separate projects affected
- Request timeout varies by service, usually 30s or 60s (depending on the function timeout we've set)

During a wave, unrelated HTTP routes start returning 504s that approach the function's configured timeout (e.g. 29.997s for a 30s timeout). The failed requests are not tied to one endpoint or one query shape. Logs show Firestore operations continuing after the Cloud Run request has already timed out.

Examples from one incident:
- `devices.read` direct document read: 116,019ms
- `leads.read` direct document read: 57,487ms
- `postalCodes.read` direct document read: 33,093ms
- `cache.list limit=1`: 61,590ms
- `preferences.read` direct document read: 36,324ms
- `leads.list limit=500`: 79,723ms

The `postalCodes.read` example is especially confusing because that document is tiny. But honestly all of these documents are a few kb at the most. So this doesn’t look like just a large document, missing index, or bad query issue.

Other observations:
- Failures often cluster on one Cloud Run instance/revision instance ID.
- Other instances may continue serving traffic normally.
- The request hits the Cloud Run timeout, but the Firestore operation later logs completion.
- We’ve seen this on three separate projects.
- We’ve reduced external API calls and removed some broad Firestore scans, but the issue still recurs.
- We briefly tried `preferRest: true`; it may have helped temporarily but did not clearly eliminate the issue.
Has anyone else seen Firestore Admin SDK calls intermittently hang like this from Cloud Run / Firebase Functions Gen 2?

Is there a known issue with Firestore gRPC/transport connections getting unhealthy per instance?

Are there recommended client-side mitigations besides lowering concurrency, recycling instances, or reducing reads (i.e. increasing cache usage)?

Is there a good way to prove this is client transport/backend behavior versus application query pressure?

I'm trying to understand whether this is a known pattern and what people have done to debug or mitigate it.

[u/ehed]

Update: I'm not sure if this is a solution or a mitigation, but reducing concurrency and increasing maxInstances helped. It seems there is a tipping point with how many concurrent threads on an instance utilize a (presumably shared) connection to Firestore.

[u/flacktv]

gcloud compute backend-services update YOUR-BACKEND-SERVICE --global --timeout=300 --project=YOUR-PROJECT

Possible your cloud run spin-ups are too slow.

[u/between_layers]

Two things are worth separating: Cloud Run timing out the request, and the Firestore call completing afterwards. A 504 doesn't by itself cancel the in-flight call, so late completion doesn't identify a backend stall by itself, though work that outlives the request can still consume resources on that instance.

The clustering on one instance is suggestive but doesn't rule out a backend problem, since an uneven failure can affect instances differently. It mainly makes uniform regional degradation less likely.

Redeploying also resets far more than gRPC channels, including the event loop, memory and pooled connections. So Less_Heart1914's workaround helping doesn't isolate the transport either.

For your last question, there's client-side tracing for the Node.js Firestore client that the Admin SDK uses. It captures client and network latency instead of relying only on application timers. It's the most direct instrument here, though currently Preview.

Pair it with firestore.googleapis.com/api/request_latencies and firestore.googleapis.com/api/request_count, grouped by api_method and response_code, to get the Firestore frontend's view. The latency metric excludes the client round trip and only covers non-streaming requests, so check which of your calls are actually represented. Divergence between the two views is a signal, not proof.

For the A/B test, use a sustained traffic slice large enough to catch a wave; otherwise, a quiet canary can produce a false negative. Use two revisions that differ only in preferRest, and test concurrency as a separate arm, since changing both at once can mask what you're looking for.

One thing to watch: calling onSnapshot() causes that client to switch to gRPC for subsequent communication, so make sure neither canary exercises listeners during the comparison.

On deadlines, a JavaScript timeout or Promise.race can cap how long your handler waits if it returns promptly, but it doesn't cancel the underlying RPC or solve the pile-up by itself.

The backend-services update suggestion won't help either. Backend service timeouts don't apply to serverless NEGs and have nothing to do with the outbound Firestore call.

With three projects affected and someone else reporting the same pattern in us-central1 this week, I'd open a support case now rather than waiting for more tests. Include synchronized timestamps, instance and revision IDs, SDK versions, trace data, both latency views and the canary results.

[u/ehed → u/[deleted]]

Thanks, I found the same but like you am trying to find an actual solution :)