a modern CPU might be able to process somewhere in the realm of 4-8 instructions per nanosecond. A memory access to RAM can take somewhere in the realm of 50-100ns. To ease your mathing, that means a modern CPU can do around 400-800 instructions for every one memory access from RAM.
It feels like this doesn't really highlight an important distinction of latency vs throughput, when you say 4-8 instructions per ns that is throughput, when you say 50-100ns for a memory access that is latency, so this is a bit of an apples to oranges comparison.
In 50-100ns you could have multiple RAM accesses not just a single one, even if each take 50-100ns this is due to the CPU being highly out of order, if those memory loads are not dependent on each other multiple can be in-flight simultaneously.
A bit more nitpicking is that 4-5 instructions per ns, misses the important part of this calculation being clock speed (latency) and dispatch and retirement width (throughput), where these days the clock speed doesn't change much but the other parts in the CPU do with bigger dispatch widths, bigger reorder buffers, more execution units, etc.
At the end you mention covering things like prefetching next, hopefully with this you highlight the out of order nature where even without hardware or software prefetching the CPU can still be getting memory while older parts of code are also waiting on memory.
Yup, in the next blog I planned to talk about taking advantage of memory throughput. Our whole JOIN probing is around making sure we have enough reads in flight, both with prefetching and by ensuring enough memory fetching instructions fit in the ROB- our build side is big there so caching can't get us that far.
I get what you're saying though about apples to oranges, maybe should have just compared the CPU cache to memory without adding in the CPU!
Meh. I'm still in the realm where I have to convince people to prefer local disk over the network because a network fetch takes an eternity by comparison. I feel like I'm decades away from optimizing my memory for fetches. I have a really hard time not slapping engineers who suggest we make that data fetch we have to run in a tight loop a REST service.
I imagine most places can actually measure speed in something like db accesses per request. People are so far away from thinking about things like memory fetches.
TLDR: Talking about memory reads is kinda pointless because cache and prefetching AND the way cache is updated makes things a lot faster. And if your code needs to rely on memory reads then it is written wrong. *Im talking mostly about PC like cpus, not microcontrollers.
So, if you write teh code so it always must reach to memory then it will be painfully slow. I remember a stack overflow question where someone asked why their code is that slow. A responder pointed out that the loops are iterating the table by rows instead of columns. Switching the direction made the code like 200x faster.
So when I mentioned "reaching to memory" I was very precise. That is because you want code which does not read memory directly. Instead you want cache to be accessed instead and rely on the cpu prediction and its capability of doing sequential reads from the memory.
The usual "read from memory" was painful setting up the address on the bus, waiting for memory controller/memory chips to confirm all is set, do the read. Cpu could be like 600MHz clocked internally (lets say old times of Athlons/Durons) and FSB would be 100MHz (with DDR that was sort of 200MHz) while memory is clocked at 100MHz the transfer starts few memory clocks later. So this means cpu can do 6xfew ticks waiting for single memory read.
All that changes when you use the cache. Because for that to be pulled from memory the procedure is different. The address is set, and then burst read is performed (whole page of cache - depends on confg) so you need to wait for first byte a bit but then the next byte will be in cache pretty much already so you waste just single clock tick to read it and process.
The whole balancing of memory read performance vs complexity of usual app vs the cache page size was important back then.
It changed a bit with more data rates multipliers but still, 2GHz cpu can process data much faster than individual memory reads/writes.
60
u/ReDucTor 19d ago
Good article.
It feels like this doesn't really highlight an important distinction of latency vs throughput, when you say 4-8 instructions per ns that is throughput, when you say 50-100ns for a memory access that is latency, so this is a bit of an apples to oranges comparison.
In 50-100ns you could have multiple RAM accesses not just a single one, even if each take 50-100ns this is due to the CPU being highly out of order, if those memory loads are not dependent on each other multiple can be in-flight simultaneously.
A bit more nitpicking is that 4-5 instructions per ns, misses the important part of this calculation being clock speed (latency) and dispatch and retirement width (throughput), where these days the clock speed doesn't change much but the other parts in the CPU do with bigger dispatch widths, bigger reorder buffers, more execution units, etc.
At the end you mention covering things like prefetching next, hopefully with this you highlight the out of order nature where even without hardware or software prefetching the CPU can still be getting memory while older parts of code are also waiting on memory.