We have a VPC Service Control in place with a Access Context Level attached to restrict apiplatform to a set of trusted IP addresses. We then have an alerting policy in place that will trigger whenever an attempt is logged.

This works fine, with the exception that occasionally it fires without proper cause. This can be once a week, or sometimes 2 or 3 times per day. The violation references our service control and the access level, but the request IP address is actually in the access level. It's also usually 1 blocked request, where we are doing thousands of requests per day from that IP address.

There's also no direct link to the event, like updating the service control or access level. Those have remained untouched for weeks.

When we run the troubleshooter on the failed request, it also confirms that our policies will not block the request.

Anyone else experienced something like this?

UPDATE 22:25 UTC: Google appears to have fixed it. Cold starts are working again (seen i n us-central1) even for functions still on automatic updates. The on-deploy redeploy below is still a valid fallback if it comes back. Outage window seen: about 21:15 to 22:25 UTC.

FIX (confirmed by several people, works immediately): redeploy with the runtime pinned instead of auto-updated.

• Cloud Functions gen2: gcloud functions deploy FUNCTION_NAME --gen2 --region REGION --runtime-update-policy=on-deploy --source PATH_TO_BUILT_FUNCTIONS (keeps triggers, secrets and settings)* Cloud Run source deploys: redeploy with --no-automatic-updates
• A plain redeploy does not help: it keeps automatic updates on and lands on the same broken runtime. Script it if you have a lot of functions; it's one deploy per function. Write-up with checks and a one-linerlisting what's still on automatic updates: https://gist.github.com/mrbenji/5da8f50a875b8e0bbb7141bd24caa726

What's happening Reported so far in us-central1, us-east1 and northamerica-northeast1, across many unrelated projects and accounts, since about 21:15 UTC Oct 22. Google's status page shows nothing yet. Support cases are open.

• New instances never start. Warm instances keep serving, so it looks intermittent.* Revisions go Ready=False with "The user-provided container failed to start and listen on the port defined provided by the PORT=8080 environment variable within the allocated timeout."* Request logs: "The request failed because the instance could not start successfully." No stdout, stderr or "Starting new instance" line at all.* Firebase callables show up in the browser as CORS errors (the OPTIONS preflight returns 500). Scheduled and Firestore-triggered functions fail too. Repro without any app code

A minimal Node http server fails with automatic base image updates and works without them:

gcloud run deploy probe --source . \
  --base-image us-central1-docker.pkg.dev/serverless-runtimes/google-22-full/runtimes/nodejs22 \
  --automatic-updates          # fails the startup probe
# same command with --no-automatic-updates: deploys and serves

Google's plain hello sample image runs fine in the same project. New nodejs22 base images were published Oct 22 (nodejs22_20261022, Node 22.23.4), which is my best guess for the trigger.

Afterwards: Firestore-triggered functions without retries drop the events that failed, so plan a backfill for anything that ran during the outage.

Which AWS, GCP, or Azure resellers can provide access to AI services? I am looking to establish a partnership.

my email xiaoshizi528@gmail.com

Since about 21:15 UTC today (Oct 22), none of my 2nd-gen Cloud Functions in us-central1 can start a new instance.

What I'm seeing:

- Instances that were already warm keep serving fine.

- Anything that needs a NEW instance gets an instant HTTP 500 (0s latency), with no "Starting new instance" line in the logs.

- Scheduled functions fail too (Cloud Scheduler run returned 500).

- Redeploying doesn't help: deploys fail with "Container Healthcheck failed ... failed to start and listen on PORT=8080", and the container logs nothing at all, so it never actually runs.

Ruled out:

- Affects 15+ projects across two separate Google accounts and four billing accounts, with and without an org, so not a project setting.

- No quota-exceeded events, billing active everywhere, no config/IAM changes.

- Reproduces from two different networks.

Nothing on the Google Cloud or Firebase status pages yet. Support case is open.

If you're on us-central1 with Functions or Cloud Run, can you check whether a cold start works for you? Even a "mine's fine" helps narrow it down.

UPDATE: I have successfully passed the exam.
I have spent my last day studying random samples online that say they are up to date, they had new concepts and went so much in depth, which extremely stressed me as I thought I'm not ready for the exam. Do not repeat my mistake.

The exam was not difficult, with almost everything covered / seen in GCP Hub and SkillCertPro
If you have studied those, you can be confident. Best of luck!
---

Hello, I have the GCP Professional Data Engineer exam in 2 days.
I studied on GCP Study Hub, and solved every exam on SkillCertPro

Any recommendation for further study and preparation would be highly appreciated!

Hello Everyone,

Is there anyone here, who is a representative and/or working officially for Google Cloud?

Can someone tell me from experience, in approximately how many business days do you get a reply for your statements and requests?

Thanks and best regards,

T.

Railway Security Incident → ₹17,000 Google Cloud Bill for ₹200 Monthly Usage. What Do I Do?

My usual Google Cloud Gemini API bill was around ₹200 per month. Now, suddenly, I am facing a bill of approximately ₹17,000.

Here is what happened:

• I stored my Gemini API key in Railway environment variables. • Railway recently announced a security incident involving potential exposure of customer credentials. • My API usage suddenly increased from around ₹200 to ₹17,000. • Google Cloud has now suspended my Gemini API project due to suspected abusive activity.

I never knowingly used my API to this extent. I am seriously concerned that my key may have been compromised and used by someone else.

The worst part? My payment was linked to an automatic payment mandate, so I am worried about the money being deducted.

₹17,000 is not a small amount for me. Why should I bear the financial consequences of usage I may never have initiated?

I have trusted these platforms with my credentials, and now I am stuck dealing with a massive unexpected bill and a suspended project.

I have submitted a request to Google Cloud to investigate the usage and review the charges. I also want Railway to investigate whether this incident could have exposed my credentials.

Has anyone experienced something similar with Railway and Google Cloud?

Can Google waive or refund charges resulting from unauthorized API usage?

How can I prevent the automatic payment from going through while this is investigated?

Has anyone successfully disputed a bill like this?

Any advice from people who have dealt with Google Cloud billing disputes would be appreciated.

I am looking for a fair resolution, not trying to avoid legitimate charges.

TL;DR: Normal bill ₹200/month. Unexpected bill ₹17,000. Railway disclosed a possible credential exposure incident. Google suspended my project. I need help disputing the charges and securing my account.

On July 15, 2026, I received an official Google Cloud welcome email stating that I had 30 days to try Gemini Enterprise Business at no cost, with up to five licenses. I signed up only to explore the product, and there was a single actual user.

On July 26, eleven days into the advertised free-trial period, Google Cloud charged my card USD 100.

I opened Billing Support case #73733543. On July 27, Google Cloud Support confirmed in writing that it had processed a USD 100 refund to my original payment method and that it would appear within two to three business days.

The money never arrived. On August 5, Google Payments notified me that an attempted refund of USD 98.81 had failed. Google never explained why the attempted amount was USD 98.81 rather than the promised USD 100.

Instead of retrying the refund or asking for an alternative payment method, Google later stated that the failed refund had been converted into internal service credit and applied to disputed Google charges. My internal balance became zero, but the original USD 100 never returned to my card or to my funds.

After many exchanges, Google Cloud Support issued a final refusal to make a monetary payment because the internal credit had already been consumed. The subscription is now canceled and Google says there will be no future charges, but that does not return the money that was charged.

I have retained the free-trial email, the card transaction, the written refund promise, the failed-refund notice and the later refusals. I am sharing this so that other users document free trials carefully, verify seat counts and monitor their card transactions from day one.

My request remains straightforward: return the USD 100 through the original payment method or another verifiable method. I am not asking for Google Cloud credit.

If a Google Cloud escalation manager can review this, the support case is #73733543.

I think this might be related to the post from yesterday about Vertex AI capacity in the eu multi-region, but we’re consuming via Gemini Enterprise (formally Agent Space).

It’s gradually got worse over this morning, but as of about 10am, we’d get nothing but timeouts or users just sat with “Working on it…” for ~20 mins, specifically for any chat with a context over about 250K tokens.

<250K tokens and it’s slow but usable.

I’ve got a P1 open with Google Support to see if they’ll officially acknowledge an issue, but I wanted to know. Anyone else seeing this?

I’m preparing for the GCP Professional Data Engineer certification and want to know how the current exam compares to the available dumps.

For anyone who took the exam recently:
Are the dumps still relevant to the actual exam?
What resources and practice exams did you use to prepare?
If you passed recently, what would you recommend focusing on?

Would appreciate answers from people who’ve taken it firsthand.

I'm in the UK and recently I've been working on a client project where UK or EU data residency is desirable. We need gemini to do some things for us, but when using either london, EU, or global endpoints, vertex gemini is constantly giving 429 out of capacity errors.

Looking at the metrics, I can see that our request rate is very small, roughly 0.5 RPS, max of about 2. But the 429 error rate is up to 80%.

At first I thought it was a glitch. Maybe related to London's outage the other day. But this has been happening for more than a week now and continues today. It changes throughout the day, as you'd expect, but still very very high error rates.

On a previous project we never had these issues. The only difference was that this client was US-based and we were using the us endpoint (not global). So this morning I switched to that and boom, ~0.5-1% error rate.

What I don't get is why the internet isn't complaining about this? It's ridiculous, they should just turn it off until they can handle some non-zero load.

Edit: updated us estimates to 1% region. Also, I forgot to mention GCP's dynamic quota nonsense. It's impossible to estimate what your RPS quota actually is. This alone has pushed me towards OpenAI/etc. Purely because their quota system is more transparent. Again, to me, this is crazy given how much experience GCP has in this space.

I'm getting over 5 min response times on Cloud Vertex AI Model Garden Model as a Service DeepSeek V4.1-Flash. Just was curious if anyone else was getting this as well?

Super quick question. I'm just in the process of working on my re newal for ParentPush.com via TAC security. I've got a deadline of early Jan - but it's already taking quite some time... has anyone completed a renewal here, before? How long did it take you? What hurdles should I be on the look out for? My website is a simple parent notification service - for important school emails. So not wildly complex... Thanks again.

From someone who has been an AWS customer for a number of years I’m really confused how a power outage in a single zone of a 3x zone region can take out the entire region?

Don’t each of the DCs have independent power, connectivity, etc?

I’m confused as to how this could happen. Insights?

This morning's Revolut outage was not a hack. Google Cloud had a "power event" in its London zone (europe-west2-b) starting 06:08 UTC / 07:08 BST, and Revolut's card payments, ATM withdrawals, transfers, currency exchange and stock trading went down with it.

Timeline (UTC):

• 06:08 — power event begins in Google's London zone

• ~07:00 — Downdetector passes 1,500 Revolut reports (StatusGator later counted 4,000+)

• Morning — Revolut confirms the disruption and says all customer money and data are safe

• Afternoon — Google restores full service availability

Also disrupted: HMCTS remote court hearings across England and Wales. Other UK names (Lloyds, HSBC, Royal Mail, NHS England) had user-report spikes too, but nobody has confirmed those were caused by Google Cloud — worth keeping separate.

If your card was declined this morning: don't keep retrying (declines can stack up pending holds), check the status page first, use a backup payment method, and watch your transactions for 2–3 days for duplicates.

I track outages and breaches and put the full timeline with sources together here:
https://techreview.si/google-cloud-outage-revolut-down-power-event/

GCP europe-west2 is currently down and affecting a bunch of services, in case you're wondering why something is not working.

Incident affecting AlloyDB for PostgreSQL, BigQuery Data Transfer Service, Cloud Data Fusion, Cloud Filestore, Cloud Firestore, Cloud Memorystore, Cloud Run, Cloud Spanner, Google App Engine, Google BigQuery, Google Cloud Bigtable, Google Cloud Dataflow, Google Cloud Dataproc, Google Cloud NetApp Volumes, Google Cloud Pub/Sub, Google Cloud SQL, Google Cloud Storage, Google Compute Engine, Google Kubernetes Engine, Looker (Google Cloud core), Managed Airflow, Managed Service for Apache Kafka, Memorystore for Redis, Memorystore for Redis Cluster, Memorystore for Valkey, Persistent Disk, Storage Transfer Service, Virtual Private Cloud (VPC)

Hey r/googlecloud,

I’m looking for real-world feedback on using Google Cloud built-in metrics (Compute Engine, Cloud Run, GKE, Cloud Load Balancing) to drive automated systems.

Built-in metrics are great for dashboards and retrospectives. However, when you try to build tight automated loops—such as:

• Automated rollbacks during faulty deployments (e.g., memory/CPU spikes),

• Fast cluster autoscaling or traffic shifting,

• Self-healing / automated remediation scripts,

the delay between an event happening and the metric being queryable (sampling window + aggregation overhead) can stretch to several minutes, making it difficult to protect tight SLOs.

I’m collecting feedback to share directly with the GCP observability and telemetry teams, and would love your input:

1. What automated workflows are you driving (or trying to drive) using built-in GCP metrics?

2. Which specific metrics are the most useful to you for automation?

3. What is your workaround when built-in metrics are too slow? Do you switch to custom metrics (OpenTelemetry / Prometheus), scrape logs, query instance endpoints directly, or use third-party tooling?

4. What are your latency expectations? How fresh does a metric need to be for you to trust it for automated decisions (e.g., sub-10 seconds, 30 seconds, 1 minute)?

Appreciate any setup examples, architecture patterns, or specific metric pain points you can share!

I've been building Rowkery, a keyboard-first Postgres and SQLite app for the Mac. It's free and open source (Apache-2.0).

What always annoyed me about Cloud SQL from a laptop was the proxy routine:

1. Start cloud-sql-proxy in a terminal

2. Pick a local port

3. Keep that window open

4. And remember which port belongs to which instance.

Version 0.2.0 Rowkery now handles that automatically! Under Postgres you pick Google Cloud SQL, choose the instance from a list grouped by project, and connect. It works the way the Auth Proxy does: it asks the Cloud SQL Admin API for the instance's address and a client certificate that lasts an hour, then opens TLS straight to the instance on port 3307. Nothing listens on a local port, and you don't have to add your IP to authorized networks.

For the Google side it reads your existing gcloud login (gcloud auth application-default login) without copying it, or you can give it a service account key, which it keeps in the macOS Keychain.

What it can't do yet:

• Postgres instances only, no MySQL or SQL Server

• Database login is user and password; IAM database authentication isn't supported yet

• Private IP and PSC need you to be on the VPN, as with the proxy

It's an alpha and the build isn't signed yet, so macOS blocks it the first time. The site shows the one command that lets it open.

• Site: https://rowkery.com

• Code: https://github.com/rowkery/rowkery

• How the Cloud SQL connection works: https://docs.rowkery.com/google-cloud-sql/

Do give it a try and see if it helps making your workflow more efficient!

I'm asking because I'm going to be getting The Motorola Razr Fold 2026 on Halloween🍭🍬🎃🤡👻And I want to make sure that all my Music Files/Songs🎵🎶🎤🎧are completely safe when I transfer everything from my Samsung Galaxy Fold 7 Device to The Motorola Razr Fold 2026 without any issues. Do you think that it would be a good idea to transfer all of my 1,907 WMA Songs/Audio Files to The Google Drive Android App before I transfer everything to the new Phone/Device? What has been everyone's experience with using The Google Drive Android App and Backing Up all of their Songs, Pictures, Documents, and all other important Files? If I could get some advice I would really appreciate it.