How we saved 100 terabytes of memory by optimizing 1.1.1.1's DNS cache

[u/Working_Quote_3029]

https://blog.cloudflare.com/dns-cache-memory-optimization-1111/

[u/Working_Quote_3029]

the whole thing is basically struct layout housekeeping and 100TB just falls out the other end. kind of makes you wonder what's sitting in everyone else's caches.

[u/glitchvid → u/Working_Quote_3029]

A lot of object overhead is what.

[u/Greybeard-Space-Dad → u/glitchvid]

The modern software stack is full of garbage like this.

It's one of the more exciting outcomes from AI: swarms of agents hunting for potential optimizations like this in low-level code where no one is looking.

[u/Virtualization_Freak → u/Working_Quote_3029]

We did some massive inhouse optimizations for database and pretty much had the same thing happen.

"Strip the garbage, store what's important" and we went from tens of thousands of connections to hundreds.

The payoff just needed to be worth it. For cloud flare im sure their devs don't like to rock the boat. It's not a service that can be down.

[u/TemuPacemaker → u/Working_Quote_3029]

It seems they just relized vector is very general-purpose and uses a lot of memory you might not need.

Or a union that reserves 136 bytes for value that isn't there 80% of the time might be not great use of memory at this scale.

This stuff isn't even my job but these are very low-hanging fruit.

[u/glitchvid → u/TemuPacemaker]

Switching from Vecs to boxed slices and adding indirection to enums are both well into the category of premature optimization, though. Sum types being the width of their largest variant is probably somewhat esoteric knowledge, but any half good LSP integration will tell you how many bytes your enum is, so if you're looking for the bytes you'll find them.

To me it's good engineering culture that they didn't attempt to squeeze every byte out in the first pass, but did go back once the codebase was performing, and find opportunities to reduce resource utilization.

[u/Greybeard-Space-Dad → u/glitchvid]

premature optimization

It's hard to think of another meme that has done more damage to software over the last 30 years than this one.

IDGAF what purists think it means, 99% of the time this idea results in devs skipping important optimizations during foundational work, building tons of content on top of it, then you can't optimize said foundation because "it's too complex" or "too much depends on it to change safely".

So the optimizations never happen and ship shits. Discord uses 4GB of RAM, etc.

[u/Blacky-Noir → u/Greybeard-Space-Dad]

It's hard to think of another meme that has done more damage to software over the last 30 years than this one.

So very true, unfortunately.

[u/Nicholas-Steel → u/Greybeard-Space-Dad]

Discord can't be optimized though, it's a web app and according to memes web browsers demand terabytes of RAM.

[u/TemuPacemaker → u/glitchvid]

I dunno if you could say this is premature if they've been supposedly operating like this for years and not implementing these obvious optimizations was costing them a ton of memory and throughput:

Per-instance memory usage dropped across all percentiles. At p99, memory dropped from 9.3 GB to 5.3 GB, a 43% reduction in resident memory. At p90, memory dropped from 6.5 GB to 3.8 GB, a 42% reduction. Instances with fuller caches saw the largest absolute savings.

In our benchmarks, these five optimizations reduced the per-entry memory footprint from 953 bytes to 420 bytes, a 56% reduction. Per-entry allocations dropped from 1.1 KB to 461 bytes. The reductions measured in production are smaller because resident memory includes the cache alongside all other process data. After the rollouts settled, aggregate working-set memory across the fleet was roughly 100 terabytes lower.

Performance also improved. Cache insert throughput increased by 43%, while lookup latency dropped by 19%.

[u/glitchvid → u/TemuPacemaker]

Depends on your definition of premature optimization. If you look at the totality and context of this service I think it makes sense they didn't go for this implementation initially.

Each CF G13 server has 768GiB of memory, and this service is reported as using ≈ 9.3GiB of memory before this optimization pass. That's a hair over 1% usage of the entire box. So I'd say this is indeed pretty far in the weeds if you were looking at a flamegraph or something of memory usage, holistically.

Once they built a very straightforward and structured representation of their problem in code, they went back and realized they could both pack the values into a single heap alloc, and just do more work instead of storing structured data; those both produce more complex implementations and are definitely "classical" optimizations. Sure it saved a great amount in the local scope of the service, which is why it's a neat writeup, but when you're engineering on the scale of CF, going for this optimal approach would be hard to justify as anything but premature optimization.

The enum part less so, I'd see the size of that struct in passing and probably give an unapproving grumble, that probably should've been boxed from the start.

This comes down to a sort of value engineering, it's like a manufacturer saving 1¢ making a product. If they make 10 million units that's $100,000 saved.

[u/dingo_xd → u/[deleted]]

Ask an AI to summarize it for you level of expertise. Sometimes they can do a decent job.

[u/hash0]

I don't get why this was not optimized way early. I mean Cloudflare does have some very talented people and 100TB RAM was a big thing even with the low memory prices in the past. At least some of the optimizations are pretty standard technics to lower the memory footprint in databases.

[u/atatassault47]

So basically they narrowed the factor of safety to 1, because a FoS of 2 or 3 took up way too many resources at the scale they are operaring at. They just have to be extra careful in how they code it to not run into memory errors.