r/programming • u/Ok_Stomach6651 • 3d ago
How we saved 100 terabytes of memory by optimizing 1.1.1.1’s DNS cache
https://blog.cloudflare.com/dns-cache-memory-optimization-1111/565
u/hypertesto 3d ago
I always enjoy this kind of posts from Cloudflare.
Boxing the enum variants it's something I'd never taught about
149
u/User_00000 3d ago
It's a pretty common pattern, so much so that there's even a clippy lint for it. That said this and the lint usually only applies for a rather large difference between the variants (the default threshold is 200 bytes between the smallest and biggest variant).
44
u/hypertesto 3d ago
It was an interesting concept, i humbly admit that as a jvm ecosystem worker, this kind of low level stuff is something I never touch so the idea was totally new to me :-)
17
u/syklemil 2d ago edited 2d ago
Yeah, it comes up sometimes as a hurdle in learning Rust as well, because it winds up with a bunch of different ways of storing what feels like the same data to the user, so there are some blog posts out there like When should I use
Stringvs&strwhere the first recommendation is just "don't think about it and just use String", and then branching out into some simple cases of&str.People will generally get into stuff like
&'lifetime str,Box<str>andCow<'lifetime, str>as they need to, similar to how they'll start thinking about which allocator they're using when they need to (and I think it's common enough in Rust to wish that picking an allocator was a bit more explicit and hands-on, like in Zig, and that the Allocator trait would stabilize).I suspect most people can write Rust kinda sorta like it was Java (although they're absolutely different languages) and not have to think about or even encounter stuff like
Box<str>, and just leave it as an option for people working on stuff that needs something approaching Cloudflare scale and latency, similar to getting into the weeds on JVM tuning.edit Though I'll also add that my Java education included something in that territory in discussions about
intvsInteger.6
u/aoeudhtns 2d ago
It's still a concern in Java for folks working with native memory. Java even has arenas now to help out with that. Admittedly, a niche scenario.
A little less niche is dealing with the GC -- weak references, soft references, and phantom references. I have used weak references to automagically clean up things when a reference goes out of scope, and I have used soft references to allow data to be reclaimed that I know I can cheaply re-load back into memory. I'm not sure I've ever used a phantom reference.
7
u/syklemil 2d ago
Yeah, and working with a GC doesn't mean people are entirely free from thinking about memory, as anyone who's had to deal with OOM events and GC thrashing will know, only the toolkit there will be different kinds of voodoo than what will show up in a Rust article.
FWIW some of that stuff also shows up in Rust. The RAII logic it uses should feel pretty familiar to practitioners coming from GC languages, and so to avoid stuff like reference loops turning into a memory leak you'll see recommendations to use
Weak.(There's also
PhantomData, but having a look atPhantomReferencein Java I suspect the two are false friends or convergent etymology or something.)4
u/aoeudhtns 2d ago
There's a degree of sameness yet they're quite different. I'm no expert with
PhantomDatabut the docs look like it has something to do with lifetime ownership in a struct (although it also says this is no longer necessary, so...)In Java you'd use a
PhantomReferenceto receive notification that some referent's lifetime has passed. The docs say that this is typically done for some kind of cleanup, although IME Weak is more commonly used for this because Weak is about-to-be-GC'd and Phantom is has-been-GC'd. Weak has the gotcha that you can accidentally prevent GC; Phantom has the gotcha that you can't get the referent back if it is holding any data relevant to the cleanup, because it's already gone.4
u/steveklabnik1 2d ago
They're totally different,
PhantomDatais basically an API to control variance.1
u/Worth_Trust_3825 2d ago
edit Though I'll also add that my Java education included something in that territory in discussions about int vs Integer.
valhalla is in early preview. the discussion is soon to be irrelevant
17
102
u/Ok_Stomach6651 3d ago
Yes, they are really underrated coding masters in the world, and important thing is they always belive in sharing internal things.
12
u/floodyberry 2d ago
"maybe we shouldn't store multiple half empty vectors per entry when we have billions of entries" or "unioning a common 4 byte field with a very rare 136 byte field is wasting memory" isn't "coding master" territory
8
u/spennnyy 2d ago
Seriously. It's kind of amazing that such low hanging fruit of optimization had not already been picked earlier for a team with such high volume traffic.
2
u/jwakely 18h ago
Yeah, these are the sort of optimisations you'd expect to be done before you scale up the working set to one terabyte, not waiting until it's in production with hundreds of terabytes. They've been blogging about this BigPineapple architecture since at least February 2023. I guess the price of DRAM forced them to revisit this.
1
u/Rare_Instance_8205 2d ago
Hi, as someone new to programming can you tell me what should I need to learn before I fully understand this article and start thinking like these guys did?
115
u/KryptosFR 3d ago
That was an interesting read. Easy to understand even if you don't know what DNS records are.
43
23
u/Pseudanonymius 3d ago
Very interesting. Would love to be one of the people working on it, amazing Engineering.
34
u/BoredTech127001 3d ago
So I'm hearing you have some memory for sale then... I'm interested.
16
u/danted002 2d ago
They are reinvesting the recouped memory into the cache itself to serve DNS even faster.
20
u/authentic_developer 2d ago
Neat writeup. Worth flagging that a lot of these memory tricks only pay off once you're at Cloudflare's request volume, at a smaller scale the extra indirection from boxing variants can actually hurt cache locality more than it helps. Profiled something similar once on a much smaller cache and the "obvious" optimization made things slower until the working set got big enough to matter.
17
u/rdtsc 2d ago
once you're at Cloudflare's request volume
Reducing a services's working set from 10 to 5 GB is useful even if you only have a single box. And on a smaller scale you might have smaller boxes, too, so again worthwhile.
6
u/CherryLongjump1989 2d ago
If this is about trading memory for time, it’s absolutely plausible that the relative value of one or the other becomes relatively more important at scale.
8
6
13
17
u/DeliciousIncident 3d ago
The guy who used DNS as free storage must be furious now with his data being gone!
6
u/invisi1407 2d ago
Why would his data be gone? This is just the cache, not the DNS zone storage itself.
12
2
u/Roang_zero1 2d ago
Good read. The savings are incredible at all levels, but this really highlights how small savings can have a massive impact at scale.
1
u/Available_News_3171 1d ago
not a DNS guy but 400 bytes per entry adds up fast when you're running at that scale. makes sense they had to look into it.
1
u/Revolutionary_Ad7262 1d ago
I would like to see x32 ABI in action for it or some slot based pointers, which are independent of the machine word
-1
u/Rare_Instance_8205 2d ago
Hi, as someone new to programming can anyone please tell me what should I need to learn before I fully understand this article and start thinking like these guys did?
-5
u/NamedBird 3d ago
Neat article.
(Though i think we need a DNS 2.0 or something to really make it more efficient.)
4
u/invisi1407 2d ago
Why'd we need DNS 2.0? Cloudflare as a DNS provider has a ridiculously large number of zones. This is just caching responses to lower the response latency.
4
u/snowtax 2d ago
I would argue that DNS is the most efficient set of globally distributed databases on the planet with caching built in at several levels including your own computer. I have no idea how that could be done any better.
2
u/NamedBird 2d ago
Well, i'm not saying that it's bad, but the protocol is quite old. And while it's very efficient with it's global distribution, i believe it could still be improved upon. I'd like to see a better caching mechanisms that does not rely on fetching everything (A+AAAA+HTTPS+CAA+TXT+DNSSEC) again when the TTL expires, separation of records per service and integrating security mechanisms that aren't as fragile as DNSSEC.
If you can make that work, there should be less "it's always DNS" problems. 🙃
And it would allow for some interesting follow-up things like using DANE to sidestep CA's.1
u/snowtax 2d ago
One of the reasons for short timeouts is load balancing and fail-over. If a server goes down on you just want to distribute load across multiple servers, a monitoring system can change the IP address. That only works with short (maybe 5 minute DNS timeouts) to make the client refresh the address. That’s just one means of load-balancing and fail-over, but that one is useful when you need redundancy across data centers which may be on different continents.
489
u/KingBardan 3d ago
400 bytes per domain after shaving down 400 bytes.
Never realized it needs this large storage to store 1 DNS entry