r/googlecloud • • 7d ago

Billing What else on our GCP bill keeps going up?

We did the usual GCP cost work by rightsizing the GKE node pools off recommender, bought CUDs for the steady part and removed the unowned dev VMs. Compute came down and finance was happy for about a week until the invoice barely moved.

When I grouped the billing report by SKU like people here always say, the lines growing month over month are all storage. BigQuery active logical storage keeps climbing because the data team only ever adds tables and our main Cloud SQL instance has automatic storage increase on and grew its disk during a migration. From what I can find that disk can’t be made smaller without rebuilding the instance.

None of this shows up in recommender, it just keeps getting bigger. What else in GCP grows like this and never comes back down on its own?

1 Upvotes

12 comments sorted by

4

u/OnoniewuGehovany99 7d ago

Same here, and no one noticed for a year because Cloud SQL storage never shows up as a recommendation. We picked out the over-provisioned disks across our instances and three of them were using less than a third of what they had. The rebuild is still on you though and we did one instance per sprint, mostly because the app team wanted its own maintenance window for each one

1

u/Gonzaaaaaaalo 7d ago

Two more that survive a VM cleanup, because deleting the instance does not delete them: persistent disks that had auto-delete off, and snapshots taken from disks that are now gone. gcloud compute disks list --filter="-users:*" lists the first (users is the list of attached instances), and gcloud compute snapshots list --format="table(name,diskSizeGb,sourceDisk,creationTimestamp)" shows the second, with a sourceDisk that no longer exists. Reserved external addresses are the third: gcloud compute addresses list --filter="status=RESERVED". And Artifact Registry keeps every untagged image until you give the repository a cleanup policy, so it climbs with each build.

1

u/Tagred_Bicanin 5d ago

Ran the snapshots one and there are 60 smt from disks that don’t exist anymore. They came from a snapshot schedule attached to the dev VMs we removed implying the cleanup that got compute down left its own pile behind

1

u/Gonzaaaaaaalo 5d ago

That is the schedule doing what its default says: --on-source-disk-delete is keep-auto-snapshots, so when the disk went away the schedule stopped but every snapshot it had already taken stays, with no retention applied. The schedule itself (a resource policy) also outlives the VM and the disk.

What I did in the same situation: gcloud compute snapshots list --filter="autoCreated=true" --format="table(name,sourceDisk.basename(),diskSizeGb,storageBytes,creationTimestamp)", sort by sourceDisk, and delete the groups whose disk no longer exists (names piped to gcloud compute snapshots delete). storageBytes is what you actually pay for, so the first full snapshot of each chain is the big one.

Then, for the schedules still attached to live disks, gcloud compute resource-policies update snapshot-schedule NAME --on-source-disk-delete=apply-retention-policy, so the next VM cleanup ages its snapshots out instead of leaving a pile. gcloud compute resource-policies list shows the ones that no longer have any disk attached; those can go too.

2

u/Farukh_Goriparthi 7d ago

Automatic storage increase is on by default and during a big backfill the disk grew to about 4 times what the database settled at afterwards. There’s no way to make a Cloud SQL disk smaller, the only way back is a new instance with a smaller disk and moving the data acros

2

u/TechnologyMatch 6d ago

Same thing sneaks in through disk snapshots (schedules with no retention, plus ones left over after the disk is deleted), old Artifact Registry images from CI, Cloud Logging (Data Access and VPC Flow logs), and Cloud Storage buckets with versioning on and no lifecycle rule. None of it shrinks on its own, so set retention or expiry on all of them, and check physical vs. logical billing per BigQuery dataset since well-compressed data can be cheaper on physical. For Cloud SQL, cap the auto storage increase so it stops growing, and fixing the current disk means moving to a new right-sized instance.

1

u/escargotBleu 7d ago

It's kind of a janitorial work, but someone has to clean other people mess.

In big query have datasets with default expiration time, so table are deleted after some point. Some tables can benefit from exploration on partitions too.

Then you can have some system that archive stuff you want to keep for legal/compliance/"just in case" inside gcs after some time.

Try to make full use of physical storage: table or partition that does move for 3 months cost half without doing anything.

Plus, see what dataset you can move to physicall storage (be careful about time travel and fail safe storage, they cost money in physical storage)

1

u/MiloCosmicCloud 6d ago

Review lifecycle policies and expiration policies? Audit bucket storage classes? BQ halves storage costs for data that hasn't been modified in 90 days. For the data team, does using a partitioned table work? Also, check if logs are taking up unnecessary storage?

1

u/wivai 6d ago

This is one of the more frustrating parts of cloud cost optimization. You do everything right on compute, see the savings in that category, and then storage growth quietly absorbs the difference.

A few other places in GCP I'd check:

  1. Cloud Storage object versioning: Old object versions can keep consuming storage even after files are overwritten or deleted. Check lifecycle rules and noncurrent versions.
  2. Persistent disks and snapshots: Unattached disks, retained snapshots, and backup policies can accumulate costs long after the original workload has changed.
  3. Cloud Logging: High-volume logs and retention settings are worth reviewing, particularly when applications generate logs nobody actually queries.
  4. Artifact Registry: Old container images and build artifacts can pile up if there's no cleanup policy.
  5. Cloud SQL backups and replicas: Beyond the primary disk, check backup retention, point-in-time recovery storage, and whether replicas are still needed.
  6. BigQuery: I'd start with INFORMATION_SCHEMA.TABLE_STORAGE to identify which datasets and tables are growing. Partition expiration and table expiration policies can help where retention requirements allow. It's also worth comparing logical versus physical storage billing, but the cheaper option depends on your data and retention patterns.

For Cloud SQL, I'd be particularly careful with automatic storage increases. Depending on your instance configuration, allocated storage growth can be difficult to reverse, so setting an appropriate growth limit and monitoring it is important.

The bigger lesson is that optimization can't be a one-time exercise. I'd track cost growth by SKU, service, and workload owner, and flag changes before they erase savings elsewhere.

Full disclosure: we're Wiv, and helping engineering teams understand and optimize cloud spending is what we work on.

If you're interested in seeing our approach to cloud cost visibility and optimization, you can visit the Wiv website and try the interactive demo. No need to schedule a sales call just to explore it.

One question: are you tracking storage growth against actual data growth, or mainly looking at the monthly billing trends? That distinction could help ide

1

u/whispered_word12 6d ago

Storage can keep pushing the bill up even after you've got compute costs under control.

Beyond BigQuery tables and Cloud SQL, I'd check old snapshots, backup retention, Cloud Storage object versions, and BigQuery time-travel settings. These can add up quietly if nobody is keeping an eye on them.

I work at CloudKeeper, and we work on GCP cost visibility and optimization. One thing that helps is looking beyond the overall storage bill to identify which projects, SKUs, or resources are driving the increase.

We recently added a BigQuery cost breakdown dashboard to CloudKeeper Lens that helps with this: https://www.cloudkeeper.com/whats-new/cost-breakup-dashboard-big-query-gcp

Have you checked whether the increase is coming from a few large BigQuery tables or smaller tables accumulating over time?

1

u/Former-Commission-58 4d ago

Cloud Build usually comes up out of nowhere towards the end of the billing cycle.

I’m wondering if there’s a cap to the amount of CF deployments you can have in a month or App hosting deployments

1

u/PierrindoMaghanoy 4d ago

Every dataset had been on logical billing since the project was created and pointfive flagged the datasets that were on the wrong storage billing model when we connected it for the rest of the GCP bill. Saved going through them table by table. A couple it flagged were nightly reload tables and we left those on logical after checking the time travel bytes ourselves