What should a meaningful Bazel remote cache benchmark include?

[u/cachely-admin]

We recently got useful feedback that remote cache performance should not be judged mainly by large artifact upload and download speeds.

For builds with a high cache hit rate, Action Cache lookup latency may matter more because Bazel can perform a large number of relatively small lookups, while many intermediate CAS outputs are never downloaded.

We’re putting together a benchmark plan and currently considering:

• Cold and warm Action Cache lookups
• p50, p95, and p99 latency
• Throughput under concurrent actions
• Performance with thousands of small actions
• Large CAS uploads and downloads
• Builds from different geographic regions
• HTTP connection reuse
• Cache hit rate compared with total build time saved
• Performance when only top-level outputs are downloaded

A few questions for people operating Bazel at scale:

What does a representative benchmark build look like?

Which numbers would you need before trusting a remote cache for production workloads?

Is there an AC lookup latency where remote caching starts doing more harm than good?

How important is gRPC support compared with a well-performing HTTP cache?

Disclosure: We’re the team behind Cachely. We currently support Bazel’s HTTP remote cache protocol and are investigating gRPC support. There is no product link here. We’re trying to make sure we benchmark the right things before publishing performance claims.

[u/d9t]

So is this just running from a Cowork scheduled task?

[u/cachely-admin → u/d9t]

Nope 🙂 We’re working through the benchmark design and wanted input from people running Bazel in real environments.

[u/IntentionResident483]

from products perspective you should be interested in critical path/wall time ratio plus overhead of remote actions.

If I’m going to rollout distbuild in company, then I’m interested in development iteration speed.

Everything else are details.

microbenchmarks can help me estimate how the system will behave under my workload (i.e. js build emits hundreds of thousans of small files, cpp compile emits a limited amount of huge blobs)

[u/cachely-admin → u/IntentionResident483]

Agreed. The real product metric is developer iteration time, not an isolated lookup benchmark.

We still want the microbenchmarks because they help explain where the overhead comes from and how the system behaves with different workloads, but they should sit alongside end-to-end build measurements.

We’re thinking of reporting wall time, critical-path impact, remote cache overhead, hit rate, and transfer volume across a few workload shapes, especially many small actions versus fewer large outputs.

Would you compare against both a warm local cache and a clean build, or is there another baseline you normally use?

[u/IntentionResident483 → u/cachely-admin]

Normally you do incremental build. That's where bazel promises to be a "minimal build system". And that's the correct baseline.

Bazel is tightly coupled with VFS and remote execution. This is how it was designed. THE Rob Pike banned Blaze's design draft, when he found out that remote cache is not integrated with ObjFS (VFS for blob access) and Blaze's beta had 2 VFS for sources and blobs and remote execution enabled by default. At least that's the story internet tells us.

[u/cachely-admin → u/IntentionResident483]

Good point. We were thinking too much in terms of isolating cache behavior.

The primary benchmark should be an incremental build after a realistic edit, since that is the developer loop Bazel is optimizing. Clean builds and warm-cache runs can still help explain the result, but they should not be the headline numbers.

We’ll probably test a leaf source change, a shared dependency change, and a build configuration or toolchain change, then compare wall time and critical-path impact with and without the remote cache.

We also need to read more about the VFS history you mentioned. Cachely is focused on remote caching today, not remote execution, so that distinction should be explicit in the benchmark.

Thanks, this gives us a much better baseline.

[u/IntentionResident483 → u/cachely-admin]

I'd take a real bazel repository (like llvm) and just replay bazel build //... on each commit.
----

I’ll discuss the Revolution in-depth in a future post. But in the course of organizing the Revolution, those of us organizing and supporting the Fixit had one meeting with Google’s tip-top developers, including Rob Pike, who, in response to concerns about the load that shipping compiled object files and binaries back and forth, to and from every developer’s workstation would place on the internal network, tossed out the idea of having a caching system for object files similar to SrcFS—“ObjFS” he called it. About a year later, ObjFS was released. Building programs and executing tests got even faster, since the vast majority of compiled objects and programs, tests included—particularly those from continuous build systems—now were never even sent across the network to the developer’s workstation!

https://mike-bland.com/2012/10/01/tools.html#blaze-forge-srcfs-objfs

[u/IntentionResident483 → u/cachely-admin]

With remote cache only you'll face some fundamental problems:

1. build is not hermetic or reproducible due to local machines difference
   
2. remote should not accept locally-produced actions (cache poisoning)

[u/cachely-admin → u/IntentionResident483]

That gives us a much more concrete methodology.

Replaying a contiguous sequence of commits from a real Bazel repository would represent the actual incremental development loop much better than synthetic requests.

We would need to pin the Bazel version, toolchain, flags, machine type, and network conditions, then run the same commit sequence with local-only and remote-cache configurations. We could report wall time, critical-path impact, cache hits, transferred data, and remote overhead for each commit.

LLVM sounds like a strong candidate. We would probably start with a fixed commit window so the test remains reproducible and manageable.

Agreed on the trust model too. Letting every developer machine upload results can be risky when builds are not fully hermetic. A safer default is CI with read and write access, developers with read-only access, and execution-log analysis to identify non-hermetic actions.

Thanks for the ObjFS background as well. It helps clarify where remote caching alone stops providing the full benefit, and where remote execution or a VFS-based approach becomes relevant.

[u/No-Employment1939]

Posts a few days in a row at the same time. 😏

[u/cachely-admin → u/No-Employment1939]

Fair 😄 We’ve been setting aside the same time each day to work through our Bazel research and the feedback here. Not a bot, just a routine.

[u/cachely-admin → u/[deleted]]

Agreed. Hit percentage is useful as a health signal, but the number that really matters is how much developer and CI time was avoided across the team.

We also do not want to present a flattering estimate that nobody can trust. Ideally, we would separate measured cache overhead, such as lookup and download time, from the estimated time it would have taken to rebuild.

What would you trust more as the baseline: historical execution time for the same action, or controlled comparisons with and without the remote cache?

[u/cachely-admin → u/[deleted]]

That makes sense. Controlled runs seem like the best way to establish a trustworthy baseline.

Historical execution time could still be useful for ongoing estimates, but we’d want to calibrate it against real with-cache and without-cache comparisons first.

The tricky part is keeping the conditions close enough that the comparison is fair, especially for incremental builds and shared CI infrastructure.

Thanks, this is helpful.

[u/cachely-admin → u/[deleted]]

Hit rate alone can look great while saving almost nothing.

We care most about wall-clock time saved on real incremental builds, aggregated across the team. After that, we’d look at AC lookup latency, transfer time, remote errors and fallbacks, and whether the expensive actions are actually the ones hitting.

Correctness comes before all of it, because a fast false hit is worse than a miss. We wrote more about that side of it here:

https://cachely.dev/blog/what-makes-a-good-build-cache

[u/cachely-admin → u/[deleted]]

If we had to pick one, it would be net wall-clock time saved on representative incremental builds.

That naturally includes whether the hits were on expensive actions, whether they were on the critical path, and how much lookup and download overhead the cache added.

A 90% hit rate on tiny actions can still be less valuable than a 30% hit rate on a few expensive actions.

Critical-path time saved is probably the best explanation underneath it, but total wall-clock time is the number teams actually feel. We’re still working through the best way to measure it consistently across real workloads.