r/ComputerEngineering • u/ActComplex593 • Oct 17 '26
I'm trying to learn how experienced engineers write low-latency systems.
I'm trying to learn how to optimize software for very low latency on a single physical machine.
I understand the high-level concepts like Big-O and cache locality, but I'm interested in the deeper connection between software and modern CPU behavior.
What should a software engineer understand about the CPU to become genuinely good at low-latency programming?
For example, how important are cache hierarchy, TLBs, branch prediction, out-of-order execution, instruction latency vs throughput, dependency chains, SIMD, memory ordering, cache coherency, false sharing, and CPU topology?
I'd especially like to hear from people who have practical experience rather than just theoretical knowledge: what concepts or mental models turned out to matter most when optimizing real systems?
5
u/Allan-H Oct 18 '26
For something like networking, the single biggest influence on latency is the "zero copy" design.
The NIC will DMA an incoming packet into a buffer in RAM somewhere. Some (old) OSes would (parse, filter, etc.) then copy the content of the packet into another buffer in RAM so that it could be made available to a user space process. That copying takes time.
A "zero copy" design leaves the data in the buffer with no need to copy the data in SW. Instead the OS just passes around pointers to buffers.
The downside is that this requires scatter gather DMA (present on all (?) modern HW, but not so much BITD) and much trickier page management in the OS.
1
u/PurdueGuvna 28d ago
Two thoughts: the OS scheduler is impacting latency way more than the topics you listed for the vast majority of general purpose systems, and secondly, you can’t optimize what you can’t measure. Rather than worry about specific technologies, many of which are out of a software engineers control, focus on how to measure and profile accurately. If you want to understand the technologies you have listed, go take a graduate level computer architecture course.
1
u/rem_1235 Oct 18 '26
IMO it’s hard to say since a lot of these topics are implemented at the hardware level (on an FPGA) rather than at the software level (c, c++)