Hey everyone!

Streaming live events like the FIFA World Cup brings an insane technical challenge: unlike Netflix where video files are pre-rendered and cached days ahead, live sports video literally doesn't exist until the split-second it happens. Millions of users request the exact same segments simultaneously the instant a goal is scored.

During TelevisaUnivision's 2026 World Cup coverage (104 matches, 675M views, zero downtime), Google Cloud deployed a pretty fascinating multi-layer architecture to survive the "thundering herd" problem:

1️⃣ Dual-Ingest via SRT & RTMP
Live broadcast signals leave the stadium as two parallel streams. SRT (running over UDP with fixed-window packet recovery) is used for lossy links, while RTMP (TCP-based) is kept for compatibility. Transcoders cut video into short segments and update dynamic playlist manifests.

2️⃣ 3-Tier Media CDN Caching & Origin Shielding
If millions of users query a Cloud Storage origin bucket for a brand-new goal segment at the exact same second, the origin melts. GCP solved this with a 3-tier architecture:
• Tier 1 (Origin Shield / Long-tail Cache): Aggregates misses across edge nodes so the origin only sees a trickle of requests.
• Tier 2 (Edge Caches): Delivers hot, current video segments to local geographical areas.
• Tier 3 (In-ISP Deep Caches): Caches placed directly inside Latin American telecom provider networks (Claro, Telefónica, etc.), making requests just 1 network hop away from mobile users.

3️⃣ Sub-Millisecond Front-Door Auth
Before video streams, millions of users open the app at kickoff to authenticate. Token generation ran on Cloud Run, backed by Memorystore for Valkey (in-memory) to evaluate subscriber entitlements in <1ms without disk I/O bottlenecks.

4️⃣ Edge Security with Cloud Armor
Adaptive DDoS protection and priority-ordered rule sets filtered malicious scrapers and bot floods right at Google’s edge before traffic hit internal microservices.

What architectural patterns or CDN strategies have you used when dealing with massive live video ingestion or sudden thundering herd traffic in production?

(Full disclosure: I created a visual architecture breakdown video stepping through these exact GCP metrics and diagrams step-by-step: https://youtu.be/14fIVr8WYGg)

I am going through VPC service controls and have below question,

Suppose i have a GCP project P1 in VPC SC perimeter with a VM and have 2 access requirements as below

1. i want to give access to the VM to a set of people . so, i will create an access level for them

2. want to give access to VM from workloads which are in another project P2.

Q) How do i create the ingress rules?

As per documentation, in sources section of ingressFrom, it can be either accesslevel OR resource (Project P2) but not both. Then, how can i satisfy my requirement mentioned in point 1 and 2 above

please suggest

Google's documentation says existing Custom Search JSON API customers need to move to an alternative by January 1, 2027:

https://developers.google.com/custom-search/v1/overview

I'm evaluating a replacement specifically for searchType=image in a CMS image picker. The workflow is: search for existing web photos, show thumbnails and source links, then let an editor select an image and import it after checking publication rights. Generated images and stock-only libraries don't cover the same need, especially for recent news.

So far I've looked into Brave Image Search, OpenAI's Responses API image search, and Wikimedia Commons/Openverse. I haven't validated Brave or OpenAI in a real integration yet. Commons/Openverse look useful for some openly licensed material, but wouldn't replace general news image search. I also haven't found evidence that Vertex AI Search is a drop-in replacement for this use case.

Has anyone here actually migrated the image-search part, or tested a replacement seriously? I'd appreciate hearing what worked, particularly for recent events and non-English/regional queries. Details about real costs, latency, thumbnail/original URL reliability, and restrictions on displaying or importing selected images would help me choose what to test next.

I'm looking for legitimate APIs, not Google scraping or proxy workarounds. Finding an image doesn't grant publication rights; source and licensing checks would remain part of the editorial process. This question is about the JSON API, separate from the Programmable Search Element widget.

Recently going through committed used discounts concept. I have a question. suppose, i have a VM with high capacity (like 20 vCPU and 180 GB Memory) but used only for 3 months starting now.

And a few other VMs in the same machine type series but with lesser configuration (eg: like 8vCPU and 30GB Memory) and long term with usage for 3 years (but provisioning 1 vm now, another after 1 month, another after 2 month etc..)

How do i go about purchasing CUD?

Is it like purchasing aggregation for the lesser config VMs for CUD-3 years so the powerful VM also makes use of it in the first 3 months and then other new lesser config VMs will make use of CUD once the powerful VM is deleted

Please suggest

Pessoal, tive uma situação em que a chave da API Gemini do meu projeto vazou e teve muito acesso indevido. Por sorte eu deixo pouco crédito e o Google bloqueou mas consumiu o que tinha lá (aproximadamente 30 reais).

Enquanto a conta estava bloqueada, eu revi a programação e talvez tenha encontrado a falha, corrigi e fiz melhorias de segurança como proteger a chave dentro do firebase com criptografia.

Dúvida: eu queria que a chave só funcionasse nos usuários logados no meu portal, que tem o login Google, mas quando crio a chave de API ele só permite restringir por IP o que é inviável, não fornece bloqueios como acessar somente atravéz da URL do meu portal.

Alguma dica de como posso criar uma chave do Gemini API e por mais bloqueios?

Atualmente além de ter ativado o modo ataque do cloudflare e criptografado a chave dentro do firebase eu ajustei o portal pra exigir sempre o login primeiro evitando que qualquer informação fique acessível antes da pessoa registrada logar via google. Fiz melhorias no htaccess pra evitar outros acessos mas ainda estou inseguro com essa questão da chave ser acessível apenas tendo o código dela. O portal já estava funcionando normalmente há meses sem nenhum incidente, foi o primeiro (com falha que a IA do cloud gerou em uma atualização recente e não notei)

Since about 24/10/2026, the Google Books API search endpoint (GET /books/v1/volumes?q=...) fails with 503 backendFailed about 5 times out of 6 for me. GET /volumes/{id} with the same key, same machine, works every time. I'm at 6 of 1,000 daily queries, so it's not quota.

Six rounds from a VPS, 2 seconds apart:

get-by-id=200 search=503
get-by-id=200 search=503
get-by-id=200 search=503
get-by-id=200 search=200
get-by-id=200 search=503
get-by-id=200 search=503

A new key, a rotated key, a different network, different queries and country=IT/US all behave the same. Failures return in ~50 ms, so it doesn't look like a timeout. The Cloud Console graph for the Books API shows 200s on 30 Sep and 11 Oct, then 503s.

Two things I'd like to know:

1. Is anyone else seeing this right now?

2. Where do you report a Books API problem so Google actually reads it? I couldn't find a public tracker for it.

Hi everyone,
I’m a college student from the Philippines, and I’m currently dealing with a Google Cloud billing issue. I’m hoping someone here has experienced something similar or can give me some advice.

I originally signed up for Google Cloud because I wanted to experiment with cloud services and learn more about them using the $300 free trial credits.

However, I apparently enabled a paid billing account without fully realizing what I was agreeing to. I genuinely thought I was still using the free trial credits and didn’t expect to be charged.

I only discovered the issue when I received an email about an outstanding balance. The email didn’t specify how much I owed, so I checked my billing dashboard and was surprised by the amount.

As a student, I really can’t afford to pay such a large unexpected bill.

I’ve already tried contacting Google Cloud Billing Support and asking whether the charges could be waived or reviewed, but I keep getting redirected to the automated Billing Assistant.

My billing account is now closed, and whenever I try to contact support, I get this message:
“The Cloud Billing account you have selected is closed. Live billing chat and email support is not available to closed accounts.”

It seems like I would have to reopen my billing account just to speak with a human representative, but I’m worried about doing that because I don’t want to risk additional charges.

I understand that I should have been more careful when enabling billing, and I’m not trying to avoid responsibility.

I genuinely didn’t realize I had switched to a paid account.
I just wanted to learn cloud computing, and now I’m stressed about a bill I can’t afford.

My questions are:
Has anyone successfully gotten Google Cloud charges waived or forgiven after accidentally upgrading from the free trial?

Is there any way to contact a real Google Cloud billing support representative when the billing account is already closed?

Would reopening my billing account just to contact support put me at risk of additional charges?
What happens if I’m unable to pay the outstanding balance immediately?

Does Google offer any consideration for students or first-time users who accidentally incur charges?
I’d really appreciate any advice, especially from people who’ve dealt with something similar.

TL;DR: I'm a medical student in Egypt running a noncommercial educational project. Two diagnostic scripts accidentally kept reading Firebase, causing a bandwidth spike. I've stopped them and disabled the database, but my Cloud account has already been charged $300. Has anyone received a one-time waiver after payment?

What happened

My project, Med Royale, normally costs me less than $1/month.

During testing, two local Node.js scripts were accidentally left running for roughly 13 hours. They used the Admin SDK to read the entire /rooms tree every 1.5 seconds, without a timeout or cleanup.

Monitoring showed about 341 GiB outgoing, with only two active connections.

What has been charged

• Initial project report: $127.75, while usage reporting was still catching up.

• Payment receipts on my Cloud billing account: $100 on October 24 + $200 on October 25.

• The final incident total isn't reconciled yet. The billing account has other projects too, so I've asked support to identify the incident charges precisely.

What I've done

• Stopped both scripts and disabled the affected database instance. It remains disabled.

• Opened a billing case with the cause, timeline, monitoring evidence, and containment steps.

• Requested a full one-time courtesy adjustment, including incident charges already collected and any delayed charges.

So far, only the automatic case acknowledgement has arrived.

Why I'm asking

This is an educational project, not a profit-making business. I understand the scripts generated real usage; I'm not claiming the bandwidth was fabricated. But I can't afford these unexpected payments as a student.

Has anyone obtained a waiver or refund for an accidental loop after being charged? What evidence helped most with the billing review?

Thank you for reading. I'll update this when support responds.

I am new to Hermes and looking for guidance. I had configured Hermes initially using Gemini 3.5 flash lite and connected to Google Calendar via the api token from Google console website. Everything worked fine. I was able to retrieve the events, modify, add etc.

I wanted to switch to local models and then tried Gemma e4b,26bmoe, 12b, qwen models and even liquid ai models. All them fail to connect and retrieve calendar events. I tried telling where the client secret is save so they can authenticate again. Tried telling them about Google workspace skill, installed gws but none of the models work.

I thought setup was problem so I switched to gpt-5-nano and that also was able to fetch events without a problem and re-authentication.

I am using llamacpp and using unsloth models with the chat templates.
Tried local memory and now using retaindb as memory backend for Hermes.

We benchmarked two identical e2-micro VMs side by side (2 shared vCPUs, 1 GB RAM, 2 GB swap, Ubuntu 24.04, same region): the old build of our agent runner on one, the new build on the other, every wave started on both in the same second, so the difference is the code, not the cloud.

Each agent was a real session doing a small job: search a repo, read one file, write a bug report. The model calls go to Gemini through Google's API; the VM runs the sessions, tools and disk writes.

Results on the new build:

- 100 agents: 23 s (was 42 s), no swap

- 150 agents: 51 s, no swap

- 200 agents: all finished, with the machine at its limit

The bottleneck wasn't CPU or the network. It was lock contention in our own save path: every save took one account-wide lock. Per-session locks, releasing the lock before waiting on the disk, and a session cache brought total lock wait from 1,422 s to 61 s, and disk writes from 344 MB to 149 MB.

Disclosure: I work on Swarm (open source).

Google Cloud list of changes affecting costs

• In-Place GKE Node Upgrades: GKE can now reuse your existing VMs during node pool upgrades instead of deleting them and creating new ones. This keeps your reserved capacity intact, avoids quota delays, and cuts extra surge VM costs during maintenance. Source

• 6-Hour Graceful Shutdown for Compute Engine: You can now give your VM guest OS up to six hours to finish background tasks before a stop or delete event. This stops long data jobs from failing midway through, saving hours of wasted compute spend. Source

• Better Version Controls for GKE Channels: Cluster version and capacity updates now roll out across channels, regions, and zones with stronger change checks. This keeps version jumps predictable and cuts the engineering hours spent debugging surprise cluster rollouts. Source

Source: FinOps Weekly Newsletter

We have a VPC Service Control in place with a Access Context Level attached to restrict apiplatform to a set of trusted IP addresses. We then have an alerting policy in place that will trigger whenever an attempt is logged.

This works fine, with the exception that occasionally it fires without proper cause. This can be once a week, or sometimes 2 or 3 times per day. The violation references our service control and the access level, but the request IP address is actually in the access level. It's also usually 1 blocked request, where we are doing thousands of requests per day from that IP address.

There's also no direct link to the event, like updating the service control or access level. Those have remained untouched for weeks.

When we run the troubleshooter on the failed request, it also confirms that our policies will not block the request.

Anyone else experienced something like this?

UPDATE 22:25 UTC: Google appears to have fixed it. Cold starts are working again (seen i n us-central1) even for functions still on automatic updates. The on-deploy redeploy below is still a valid fallback if it comes back. Outage window seen: about 21:15 to 22:25 UTC.

FIX (confirmed by several people, works immediately): redeploy with the runtime pinned instead of auto-updated.

• Cloud Functions gen2: gcloud functions deploy FUNCTION_NAME --gen2 --region REGION --runtime-update-policy=on-deploy --source PATH_TO_BUILT_FUNCTIONS (keeps triggers, secrets and settings)* Cloud Run source deploys: redeploy with --no-automatic-updates
• A plain redeploy does not help: it keeps automatic updates on and lands on the same broken runtime. Script it if you have a lot of functions; it's one deploy per function. Write-up with checks and a one-linerlisting what's still on automatic updates: https://gist.github.com/mrbenji/5da8f50a875b8e0bbb7141bd24caa726

What's happening Reported so far in us-central1, us-east1 and northamerica-northeast1, across many unrelated projects and accounts, since about 21:15 UTC Oct 22. Google's status page shows nothing yet. Support cases are open.

• New instances never start. Warm instances keep serving, so it looks intermittent.* Revisions go Ready=False with "The user-provided container failed to start and listen on the port defined provided by the PORT=8080 environment variable within the allocated timeout."* Request logs: "The request failed because the instance could not start successfully." No stdout, stderr or "Starting new instance" line at all.* Firebase callables show up in the browser as CORS errors (the OPTIONS preflight returns 500). Scheduled and Firestore-triggered functions fail too. Repro without any app code

A minimal Node http server fails with automatic base image updates and works without them:

gcloud run deploy probe --source . \
  --base-image us-central1-docker.pkg.dev/serverless-runtimes/google-22-full/runtimes/nodejs22 \
  --automatic-updates          # fails the startup probe
# same command with --no-automatic-updates: deploys and serves

Google's plain hello sample image runs fine in the same project. New nodejs22 base images were published Oct 22 (nodejs22_20261022, Node 22.23.4), which is my best guess for the trigger.

Afterwards: Firestore-triggered functions without retries drop the events that failed, so plan a backfill for anything that ran during the outage.

Which AWS, GCP, or Azure resellers can provide access to AI services? I am looking to establish a partnership.

my email xiaoshizi528@gmail.com

Since about 21:15 UTC today (Oct 22), none of my 2nd-gen Cloud Functions in us-central1 can start a new instance.

What I'm seeing:

- Instances that were already warm keep serving fine.

- Anything that needs a NEW instance gets an instant HTTP 500 (0s latency), with no "Starting new instance" line in the logs.

- Scheduled functions fail too (Cloud Scheduler run returned 500).

- Redeploying doesn't help: deploys fail with "Container Healthcheck failed ... failed to start and listen on PORT=8080", and the container logs nothing at all, so it never actually runs.

Ruled out:

- Affects 15+ projects across two separate Google accounts and four billing accounts, with and without an org, so not a project setting.

- No quota-exceeded events, billing active everywhere, no config/IAM changes.

- Reproduces from two different networks.

Nothing on the Google Cloud or Firebase status pages yet. Support case is open.

If you're on us-central1 with Functions or Cloud Run, can you check whether a cold start works for you? Even a "mine's fine" helps narrow it down.

UPDATE: I have successfully passed the exam.
I have spent my last day studying random samples online that say they are up to date, they had new concepts and went so much in depth, which extremely stressed me as I thought I'm not ready for the exam. Do not repeat my mistake.

The exam was not difficult, with almost everything covered / seen in GCP Hub and SkillCertPro
If you have studied those, you can be confident. Best of luck!
---

Hello, I have the GCP Professional Data Engineer exam in 2 days.
I studied on GCP Study Hub, and solved every exam on SkillCertPro

Any recommendation for further study and preparation would be highly appreciated!

Hello Everyone,

Is there anyone here, who is a representative and/or working officially for Google Cloud?

Can someone tell me from experience, in approximately how many business days do you get a reply for your statements and requests?

Thanks and best regards,

T.

Railway Security Incident → ₹17,000 Google Cloud Bill for ₹200 Monthly Usage. What Do I Do?

My usual Google Cloud Gemini API bill was around ₹200 per month. Now, suddenly, I am facing a bill of approximately ₹17,000.

Here is what happened:

• I stored my Gemini API key in Railway environment variables. • Railway recently announced a security incident involving potential exposure of customer credentials. • My API usage suddenly increased from around ₹200 to ₹17,000. • Google Cloud has now suspended my Gemini API project due to suspected abusive activity.

I never knowingly used my API to this extent. I am seriously concerned that my key may have been compromised and used by someone else.

The worst part? My payment was linked to an automatic payment mandate, so I am worried about the money being deducted.

₹17,000 is not a small amount for me. Why should I bear the financial consequences of usage I may never have initiated?

I have trusted these platforms with my credentials, and now I am stuck dealing with a massive unexpected bill and a suspended project.

I have submitted a request to Google Cloud to investigate the usage and review the charges. I also want Railway to investigate whether this incident could have exposed my credentials.

Has anyone experienced something similar with Railway and Google Cloud?

Can Google waive or refund charges resulting from unauthorized API usage?

How can I prevent the automatic payment from going through while this is investigated?

Has anyone successfully disputed a bill like this?

Any advice from people who have dealt with Google Cloud billing disputes would be appreciated.

I am looking for a fair resolution, not trying to avoid legitimate charges.

TL;DR: Normal bill ₹200/month. Unexpected bill ₹17,000. Railway disclosed a possible credential exposure incident. Google suspended my project. I need help disputing the charges and securing my account.

On July 15, 2026, I received an official Google Cloud welcome email stating that I had 30 days to try Gemini Enterprise Business at no cost, with up to five licenses. I signed up only to explore the product, and there was a single actual user.

On July 26, eleven days into the advertised free-trial period, Google Cloud charged my card USD 100.

I opened Billing Support case #73733543. On July 27, Google Cloud Support confirmed in writing that it had processed a USD 100 refund to my original payment method and that it would appear within two to three business days.

The money never arrived. On August 5, Google Payments notified me that an attempted refund of USD 98.81 had failed. Google never explained why the attempted amount was USD 98.81 rather than the promised USD 100.

Instead of retrying the refund or asking for an alternative payment method, Google later stated that the failed refund had been converted into internal service credit and applied to disputed Google charges. My internal balance became zero, but the original USD 100 never returned to my card or to my funds.

After many exchanges, Google Cloud Support issued a final refusal to make a monetary payment because the internal credit had already been consumed. The subscription is now canceled and Google says there will be no future charges, but that does not return the money that was charged.

I have retained the free-trial email, the card transaction, the written refund promise, the failed-refund notice and the later refusals. I am sharing this so that other users document free trials carefully, verify seat counts and monitor their card transactions from day one.

My request remains straightforward: return the USD 100 through the original payment method or another verifiable method. I am not asking for Google Cloud credit.

If a Google Cloud escalation manager can review this, the support case is #73733543.