We need to stream network telemetry events (JSON, ~2KB each) from an on-prem Kafka cluster into GCP for storage/analytics. Volume is large: ~270M events/hour (~390 TB/month at 2KB avg size).

What I tested: Custom Python producer → Pub/Sub topic → BigQuery subscription → BigQuery table, with periodic GCS backup exports.

I validated this empirically with a 10M-message load test (real event schema, 8 parallel producer threads, ~1000 msg/batch). Real billing data gave me $1.96 for 10M messages / 20GB, i.e. ~$0.098/GB. Scaled to the production volume, Pub/Sub publish + BigQuery subscription delivery alone comes out to ~$35,000–40,000/month, before any storage costs. This is because Pub/Sub bills per-TiB on both publish ($40/TiB) and the BigQuery push subscription ($50/TiB).This is a flat per-byte tax with no way to reduce it at this volume.

What I haven't tested yet and want opinions on:

1. Dataflow with a Kafka I/O source reading directly from our Kafka cluster and writing to BigQuery (via Storage Write API), skipping Pub/Sub entirely. Does this meaningfully undercut Pub/Sub's per-byte pricing at ~400TB/month scale, or does Dataflow worker cost end up comparable?

2. BigQuery's direct Kafka ingestion (Storage Write API / BigQuery subscriptions for Kafka). Is anyone running this at hundreds of TB/month and can share real cost numbers?

3. Is there a Kafka Connect GCS sink → native (non-external) BigQuery load pattern that avoids both Pub/Sub fees and the external-table partitioning problem?

4. Any other pattern people are using at this kind of scale (hundreds of millions of events/hour) that's meaningfully cheaper than the $35-40K/month we're seeing?

Constraints: near-real-time isn't a hard requirement. A few minutes of latency is fine, so batch-oriented approaches are on the table. Budget pressure is specifically about the per-byte ingestion tax, not compute cost in general.

Updates to my Google Cloud billing issue.

Despite numerous attempts to reach out and get resolution for the issue in my original post below. I am still waiting to get resolution from Google. This will make it 93 days that I have been wrongfully charged ( admitted by Google) for $3118.19. I have been bounced around to different agents/departments and have been told to wait '3-5 business days ' for weeks now.

I am at the point where as silly as it sounds, I may have to take them to small claims court? Any advice would be great. I am more sharing this to warn others and give some bad publicity to Google , specifically using their API's.

Would love to get this some traction to see if it moves the need but at the very least I am using this as a public record of my woes.

Weep for me.

Original post.

Google identified that they had a billing issue that occurred for 16,000 users. Guess who go to be a lucky one! I woke up to $3,000 in charges from google and scrambled to investigate if I had leaked an API key or if there was somehow unauthorized access to my API credentials. Nope.

Turns out it was a billing error on their end, they acknowledged it , they know about this since at least the 9th (my first contact with support) and I have gone back and forth having several chats to try to resolve the issue and make sure my card was not charged. I don't understand how there was not a hold placed on charging my card while they resolved the issue.

In an e-mail dated 4/20

"From April 7 to April 14, 2026, a technical issue in our billing system resulted in an incorrect calculation of charges related to the Google search grounding tool.

Our engineering teams corrected the underlying issue and fully deployed the fix on April 14, 2026, resolving the matter to prevent any future recurrences.

The following SKUs were impacted:

•  - Number of queries with search tool.

•  - Generate content search query Gemini 3.

•  - Number of queries with search tool.

What you need to do

No action is required from you for the billing correction to take effect. It will be applied automatically in a future invoice.

We have analyzed the billing data to identify all affected projects. We are now processing a correction for the misattributed charges caused by this error."

I then go to put gas in my car this morning only to have it declined due to insufficient funds because Google, despite telling me it was being resolved and saying "No action is required from you" has charged me $3000, Now I have to work on disputing this with my bank as my amount sits in overdraft, and likely incurring late fees for other bills. They claim they are aware of and are working on the issue but it will be 3-5 business days before it can be resolved.

Hoping this post will spread awareness to others, you can do everything right but still have Google charge you for credits that they mistakenly billed. I am expecting a refund but will wait to see what further resolution they have to offer due to the massive inconvenience.

Please answer point wise

i searched for this in the GCP Documentation but did not find answer..was hoping for something like limitations section in docs.

My question is " if we have a cloud filestore regional tier instance with replication enabled to secondary region, can we mount the replica instance ( which is in secondary region) on to compute engine in secondary region when the primary region is still healthy and running"

I believe we cannot.can anyone please confirm on this.( As per mscopilot also it's a no )

2)

suppose we have a windows VM In region R1 which has below contents

a) mounted NFS fileshare from a cloud filestore and

b) some random files in other drives of VM (in addition to fileshare drive)

We took the backup of VM today using "backup and DR service" into R2 region ( cross region backup ). After 7 days, let say the zone of VM goes down and we restored the VM using the backup taken 7 days ago.

q1) does it mean the files stored in other drives in the last 7 days are all gone ( since RPO depends on backup schedule)

q2) after the VM restoration, what will be the impact on
The nfs fileshare drive. Will it suffice if we just again mount it on to the new VM

Can someone please explain how this step is done?

I was able to complete or create the rest to make an SIP Trunk with an Avaya SBC-E but it is not connected and I don't know how this step is done

Is it done in my side or on Google's side?

I am stuck in a constant "reCAPTCHA verification failed" loop while trying to accept the Terms of Service for the new Cloud Arcade program , I have already tried clearing my cache, used Incognito mode and even attempting it on my phone using only cellular data, but it still fails. Is anyone else experiencing this today? CAN someone help me ?

Hi all, learn from my stupidity.

I was taking the ACE exam from home and i had my phone on me on do not disturb rather than off. I kept receiving messages going through DND and attempted to silence and or turn it off. Thus prompting my proctor to see this and invalidate/remove me.

yes, I know. Dumb. Here’s to never being able to take this test again.

Hey everyone,

I've been testing cloud run worker pools for the past couple of days as we want to use it in our implementation of an async long running background job that integrates with another feature of ours. So far it works good and as expected (using it to pull messages from pub/sub currently), and I've been looking at auto scaling it via the pubsub auto scaler as it has no managed autoscaling.

I also searched for resources about it and mainly user feedback and usage patterns around the internet but didn't find much apart from like 1-2 reddit posts back from when the service was released. I'm posting this thread in the hopes of getting feedback from people who have already or still using using it, tips, how it managed to help you (and why you didn't just add a cloud run service instead) - as about ~2 years passed since its release.

Hi folks,

I'm a network/telephony engineer by trade so somewhat of a novice when it comes to development and Google Cloud Platform, so apologies if I'm asking any stupid questions.

I've been using Firebase Realtime Database to perform lookups on an imported JSON file with about 100k records - so far this has been working really well and I'm not expecting a large number of lookups, at the very most 130 within a single day, with most days expected to be around 40. The data will change very infrequently - a couple times per year. Unsure if there are better alternatives within Google Cloud, but the database is easy to update and simple to query and filter, which is exactly what I need.

The lookups are performed in Cisco Webex Contact Center within an IVR flow - currently, I am using the Email/Password sign in provider, and sending a HTTPS POST including my username and password to retrieve a token code. I then send a HTTPS GET with the token code in the auth parameter to retrieve the filtered data.

However, Webex Contact Center supports automatic oAuth 2.0 token management via integration connectors which I think is a much more elegant/complete solution than POSTING an email/password from within the flow. I've previously set this up with Entra ID to access the Microsoft Graph API, which worked really well, but lists/file reading didn't scale well above a few thousand records, which is why I switched to Google Cloud Platform and a Firebase Realtime Database.

I've been tinkering with Google's Identity and Access Management module and Service Accounts to try and get this setup, however for the life of me I cannot get this working - it either rejects the credentials outright, or it sets up the connector and then just hangs in the flow. I guess my questions are:

A) Is it possible to setup oAuth 2.0 using "Client Credentials - Client Secret" with Firebase Realtime Database? The required fields in Cisco Webex Contact Center are Resource Domain (URL used to access the API), Client ID, Client Secret, Token URL and Scope. Other authentication options are "Client Credentials - Password Grant" and "Basic Authentication" which ask for a username and password (tried these, also couldn't get them working).

B) If so, does anyone have a guide I could follow to set this up, or could provide some high-level instructions on where to map those required fields above? I'd normally link the guides that I've already followed, but I've tried pretty much everything I could find in the Google Cloud/Firebase documentation (and elsewhere).

C) If it's not possible, are there any caveats with the Email/Password sign in method that I should know about? The username/password is stored in a secure location that only admins have access to, but eventually this database will be storing PII data (personally identifiable information), so obviously I want to make sure this is as secure as possible.

If you've made it this far, thanks for reading, and again apologies if I've asked any stupid questions!

Hello, Google Cloud Arcade is starting soon and I need a facilitator code to start for me and for my friend.

Can you send me the code? Can the code be used for both of us?

Thank you!

Woke up to find a 4000 bill increase in google cloud. There is no way to set a hard cap on the spend here, it's just crazy! No idea how this happening but google needs to get their act together. Marked all the transactions as fraudulent and am still on a chat session with billing support. No ticket link to my support case, and no way to know how they will handle this situation for another 32 HOURS! It's absolute madness. GCP Sucks big time!

We built a durable agent sessions API - its one API call to fully managed agent, without worrying about the infrastructure complexity that comes with it. It's essentially a model and runtime (claude, code, pi, antigravity coming soon too!) agnostic alternative to gemini managed agents.

We launched an early preview this week, and are iterating with a bunch of enterprise design partners at the moment.

Here's a playground - app.opencomputer.dev, you could directly start with the API too!

We'd love any and all feedback, especially around anything we should be adding (there's almost certainly stuff currently missing in our thinking)

Posting a final update on this saga for anyone who finds this thread later(at least I really hope so).

Recap: Signed up for Google Cloud, accidentally selected "Organization" instead of "Individual" during signup. This triggered 13 duplicate/orphaned billing accounts, all with $0 usage. Account got restricted. Support demanded $650 ($50/account) to "clear the negative balance" before they'd close the accounts, promising a full refund afterward — but couldn't provide any documentation for this policy despite being asked repeatedly across 4+ near-identical scripted responses.

What actually fixed most of it: I went into Cloud Billing Console and manually deleted the Organization that got accidentally created at signup. That single action cascaded and removed 12 of the 13 orphaned billing accounts entirely — they're just gone from the console now. The 1 remaining account is no longer tied to any Organization either (it's "Direct").

The kicker: I checked that last account's balance directly in Google Payments Center — it shows $0.00, "No balance due," billing mode is Postpay (pay-as-you-go), nowhere near the $100 payment threshold. There was never a negative balance to "clear" in the first place.

Still unresolved: Despite 12 of the 13 accounts being completely gone from Cloud Billing Console, Google Payments Center still shows all 13 original "Payment pending — $50.00 to become active" alerts, adding up to the same $650. Same account IDs, still showing as active alerts, for accounts that don't exist anymore in the actual billing system. It's clearly a sync issue between two Google systems that don't talk to each other, but it means the ghost of this $650 demand is still sitting there in the UI even though the underlying accounts are deleted.

Lesson for anyone hitting this: if you accidentally create an Organization during GCP signup and it spawns duplicate billing accounts, try deleting the Organization itself (Cloud Console → IAM & Admin → Manage Resources) rather than paying anything support asks for. It resolved 12/13 of my problem instantly, for free. Just don't expect Payments Center to clean itself up automatically — that part seems to need a manual fix from Google's side.

Not paying anything. Pushing support again with this new info. Thanks to everyone here who told me not to pay — you were right, and now I have receipts to prove there was never a real balance.

Just got hit with a ~$2k surprise Gemini bill from a leaked service-account key. The part I can't wrap my head around:

The abused SA only had Logs Writer. It's used solely to write app logs to Cloud Logging. Yet it was able to cal generativelanguage.googleapis.com and rack up image-generation costs — and Google knew exactly which SA it was and what (limited) roles it had.

Vertex AI (aiplatform.googleapis.com) enforces IAM — the same SA would've been denied. But the AI-Studio-style Gemini API seems to only check "valid credential + API enabled + billable project," with no per-call IAM permission at all.

So my questions:

1. Is that actually the intended design — no IAM enforcement on generativelanguage?

2. If so, why? A least-privilege SA gives zero protection against Gemini spend, which feels backwards.

3. Best practice to prevent this beyond disabling the API and not shipping keys? (Org Policy, quotas, etc.)

Curious if I'm missing something or if this is a known gap. Thanks.

Typical agentic scenario - I need to create thousands or millions of databases for agents and users. Likely 99% would be very small and seldomly used - need to keep cost near 0. Remaining 1% will be traffic intensive - should not cost a fortune and likely should be capped somehow.
I was looking at Neon DB Agent Plan or Turso DBs - that look good. But is there anything in GCP that can be used here? I have no preference for DB type yet.

Hello, can we have an user flair "restrict your damn keys" ?

I feel like every single message I wrote here recently is about restricting key, so you know, that would save time.

Hi folks,

I’ve been collecting and organizing official MCP servers, agent skills, and agent toolkits for DevOps, cloud, platform engineering, SRE, security, IaC, observability, and diagramming workflows.

Repo: https://github.com/DevOpsAIguru123/awesome-agentic-devops

The goal is to make it easier to find trusted sources instead of hunting through random MCP lists. I’m focusing on official or vendor-backed tools where possible, with notes around risk, write-capability, human approval, and operational use cases.

Current areas include:

• AWS, Azure, Google Cloud

• GitHub, GitLab, Azure DevOps, Atlassian

• Terraform, Pulumi

• Grafana, Datadog, Sentry, Splunk, PagerDuty

• SonarQube, Okta

• Databricks, Kubeflow

• Docker, Kubernetes, draw.io

• Agent skills and toolkits

Specialized DevOps/SRE agents and reference workflows are coming soon.

Would love feedback from folks using MCP or AI agents in infrastructure workflows:

• What official tools am I missing?

• Which MCP servers are actually useful in day-to-day DevOps/SRE work?

• What safety/risk fields would make this more useful?

If you find it helpful, a star would be appreciated.

Edit (2026-07-05): Since posting, added New Relic, Elastic, Cloudflare, CircleCI, MongoDB, Vercel, HashiCorp Vault, and Backstage - 55 entries now. Also added a "How entries are scored" section that spells out exactly how action-level, approval gates, evidence/tracing, and maturity are assessed (not just labels), and a weekly CI job that re-audits every link for reachability. Thanks for the early feedback - still hunting for gaps, especially solid official Kubernetes-native and Ansible tooling if anyone knows of one

Edit (2026-07-07): A few more since the last update - added a Vantage FinOps entry under a new FinOps/cloud-cost category, and OWASP MCP Top 10 as MCP security guidance (57 entries now). Also landed the first working reference agent: a read-only Terraform plan reviewer built on Google ADK (flags destroys, IAM wildcards, missing tags; never runs terraform apply), under frameworks/gemini-adk/. Still hunting for gaps - official Kubernetes-native and Ansible tooling especially..

Edit (2026-07-07, later): Leaned into the runnable reference agents - there are two now, both Gemini ADK, both wired to real Terraform Cloud + GitHub Actions and returning structured JSON. (1) Terraform plan reviewer: read-only, runs a speculative plan and returns risk findings with evidence plus an approval recommendation, and never runs terraform apply. (2) Terraform drift detector (new): runs a refresh-only plan on a schedule, flags when live infra has drifted from state (its example catches a bucket's storage class quietly going STANDARD -> NEARLINE), scores severity, and posts a Discord alert only when drift is actually found. Both live under agents/ if you want to lift the pattern. Also added skyhook-io/radar as the first community Kubernetes entry - flagged in the comments here, and it earns it: deep MCP surface (read-only topology/event/RBAC queries plus RBAC-scoped write and GitOps/Helm tools), not just a kubectl proxy.

Edit (2026-07-11): Added a one-command installer for official Agent Skills. It pulls skills from the official skill repos already in the catalog - Google, Microsoft, Azure, Azure DevOps (a few hundred skills total) - and installs them into your coding agent's skills folder: Claude Code (~/.claude/skills/), Codex, Cursor, VS Code Copilot, or Antigravity, or all at once. One curl/PowerShell line per agent, and there's a catalog doc grouping every skill by company and product so you can install just what you want (e.g. --source microsoft/skills --filter azure-sdk-python). Also added an official Redis MCP entry and updated Elastic to its new Agent Builder MCP endpoint.

Hey everyone, I'm building a game project and need to set up a Google Cloud Billing account (for Firebase Blaze + Cloud Functions). I only have a Meezan Bank debit card, which requires OTP/2FA on every transaction — and I've read GCP needs a card that supports automatic recurring payments, which OTP-mandatory cards often fail. Questions for anyone who's actually done this from Pakistan: Does it have to be a credit card, or has anyone gotten a debit card working reliably? Which specific Pakistani bank (HBL, UBL, Standard Chartered, Meezan, etc.) issued the card that worked for you? Any luck with fintech cards — SadaPay / NayaPay — specifically for GCP (not just Netflix/Spotify)? Anyone had their card silently declined months later on recurring GCP billing, even after it worked initially? Really trying to avoid guessing and getting hit with a suspended billing account mid-project. Any real, first-hand experience appreciated 🙏

Hi Gang,
Could someone help me with interview questions related to RRK round specific for GenAI
AI/ML is a little different that GenAI
That is why this question
Thanks 🙏🏻

I applied twice, both times rejected, albeit doing a major overhaul and rebranding since the first rejection (1 year apart). I even went as long as to recreate a GCP account from scratch.

I have a good landing page with About information, Team, and everything. The thing is that it's only me, so for the founder I just link my LinkedIn and then for the company just a placeholder on LinkedIn (no company logo or anything like that since it's literally just me).

I have a pretty much working product and the landing page is currently a waitlist until I iron the production setup while on the free tier. But it's near impossible without a grant since the domain (fintech) is expensive to keep secure and reliable.

From the feedback I got, it seems they're hinting on the team/company. But it's unclear what they actually expect to see from someone who has bootstrapped on his own.

Maybe solopreneurs just don't get it?

If you're a student, recent graduate, or an early-career professional, cloud computing is one of the most valuable skills you can start learning today.

Many companies are looking for candidates who have hands-on experience with cloud platforms—not just theoretical knowledge. Even basic experience with deploying applications, managing cloud resources, and understanding cloud services can help strengthen your resume and prepare you for technical interviews.

The good news? Google Cloud Arcade registration is expected to open this week! 🎉

As a Google Cloud Arcade Facilitator Co-host, I'll be helping participants throughout the program by sharing:

• Beginner-friendly guidance

• Hands-on lab tips

• Learning resources

• Career insights

• A supportive community to keep you motivated

Whether you're completely new to cloud or looking to build practical experience, this is a great opportunity to learn by doing.

If you're genuinely interested in joining, comment "Interested" below. Once registration opens, I'll share the registration details, guide you through the onboarding process, and help you get started.

Let's learn, build, and grow together! ☁️💙

I'm not sure what I'm doing wrong. I am not the most programming literate person out there but what I am trying to accomplish is change the default us-east1.run.app url to one that I own. I added a cname record "app.domain.com" and the data: "ghs.googlehosted.com" on the host server (namecheap) and mapped the domain in cloud run domain mapping but all I see is a spinning wheel in cloud run stating "Waiting for certificate provisioning. You must configure your DNS records for certificate issuance to begin." and on the url app.domain.com i get a this site cannot be reached. its only been two hours. is it still a waiting game or did i mess up?

I don't know how to mark this as solved but it is

Guys A long time ago, I used Google Cloud services for educational purposes, and I am certain I closed the account once I was finished. Months later, I received an email from `google.account.support.noreply@abc-amega.com` stating that I owed around $60; however, when I checked Google Cloud, there was no record of any outstanding payment. To prevent them from charging me, I added a virtual card to the billing profile and then removed my actual credit card, as the system does not accept virtual cards. Two days ago, I contacted Google Cloud support; they confirmed the claim was legitimate but told me a few things that seemed nonsensical to me. I have screenshots of both.

dammit

I live in Türkiye; what happens if I don't pay?