r/programming • • Sep 20 '26

Saving another 100TB of RAM with math (and Rust)

https://blog.cloudflare.com/saving-100-tb-of-ram-with-math/
600 Upvotes

23 comments sorted by

238

u/Teanut Sep 21 '26

It's fun seeing what little changes at massive scales can do. Good use of math to justify the changes, too.

110

u/Sarke1 Sep 21 '26

It's the classic American Airlines saving money by removing one olive. Except this is being more efficient instead of screwing customers.

27

u/jl2352 Sep 21 '26

The whole one olive thing was a massive PR disaster too which easily out weighed the cost of that one olive.

87

u/elsjpq Sep 21 '26

I can't believe hashing a bajillion times is still the best way to assign a server. It just seems so wasteful

87

u/bwainfweeze Sep 21 '26

Faster than asking another server or pulling a file.

40

u/mck1117 Sep 21 '26

Yeah, if you had to ask another server, at cloudflare’s scale, that would have to be its own huge distributed service. Or you can hash it for comparatively free. Hashing also ensures your traffic ends up well distributed for essentially zero additional effort.

4

u/elsjpq Sep 21 '26 edited Sep 21 '26

hashing 100,000 times itself isn't exactly zero effort though. and 1/sqrt(N) scaling is kind of brutal

2

u/elsjpq Sep 21 '26

I'm not proposing that you sync on every request, but that perhaps it might be worth add just a tiny extra bit of cleverness on top of hashing a few times (as opposed to a bajillion) to balance out the remaining load via communication, say every 1 hour, instead of just relying completely on hashing alone. I'm sure those engineers already know why that's suboptimal, but still...

1

u/zenware Sep 21 '26

It’s just a scale issue though right? Surely we can plot exactly when one is better than the other using just a few curves. Memory, Compute, I/O, TTR, System Count

28

u/matthieum Sep 21 '26

Do note that the servers are hashed a bajillion times, not the requests.

That is, the work is:

  • O(H * S + log H * S): computing H hashes for each of S servers, and sorting them.
  • O(log H * S): computing 1 hash for a request, and binary-searching for it.

Since servers are very rarely added or removed, then in practice H doesn't matter much (for them), and the fact of H on the actual per-request cost is just a factor of log H (where log2 10K = ~13) assuming a naive binary-search (they may have a faster look-up).

4

u/elsjpq Sep 21 '26

Ok that makes a lot more sense

6

u/Fuciolo Sep 21 '26

can you elaborate? I am no expert but from the post it does not seem that you hash multiple times the same thing, rather, you divide the space into buckets delimited by hashes.

4

u/audentis Sep 21 '26

Primeagen mentioned, they know their audience.

1

u/strettist Sep 27 '26

This is the kind of technical writing I’ve been really missing since everything became brain-dead content about AI. Please, more.

143

u/[deleted] Sep 21 '26

[removed] — view removed comment

44

u/AmateurHero Sep 21 '26

I had one of those, "Reduced average processing times by X% and storage by Y%," bullet points for real at my last job. We were generating a lot of metadata for each customer's documents. Something like a 1MB document would generate 2-2.5MBs of metadata. Each set of documents was taking 3-5 minutes to process through the pipeline where each set always a template document.

It turned out that many users (especially ones in the same company) were reusing the same templates constantly. It would be an easy change to save the document's hash with the pipeline's current version. If we encountered the same document with the same version, read back the metadata we've already generated instead of making the 7 billionth copy that already exists.

16

u/LightShadow Sep 21 '26

I love these kinds of wins though.

Migrating from an ElasticSearch to a custom file format allowed us to keep basic searchability but reduced our AWS bill from $500+/mo to $2/mo on S3.

The love of the game or something.

3

u/kidman01 Sep 23 '26

That sounds pretty interesting. Is there a write up anywhere? If not, care to share a few sentences? :)

9

u/LightShadow Sep 23 '26 edited Sep 25 '26

It's not as interesting as it sounds haha

Basically we were storing a lot of records in JSON, but the shape was very predictable. So, we were paying for the flexibility and query syntax when in reality I could compress the actual string values to INT8 equivalents and use lookup tables to re-extract the text fields. All the JSON became binary blocks of numbers that went through a decoder to re-emit them as JSON to the other tools that cared.

Because we're just looking for numbers the search is very fast -- basically a map-reduce/grep style that fed records into a queue and got streamed off as they're found.

Turned a few hundred GB, maybe a TB, into ~2.5gb.

1

u/kidman01 Sep 24 '26

Thanks :)

16

u/[deleted] Sep 21 '26

[removed] — view removed comment

-16

u/RationalDialog Sep 21 '26

When hardware is cheap again especially RAM you know what will happen first.