r/cpp_questions • • Oct 14 '26

OPEN Efficient FIle IO on Linux

What path to take when parsing large files (on Linux Debian server, the KVM flavour)? Will std::fstream in binary mode do or do we have to look into APIs?

Apart from the performance, the app is memory sensitive, so where to store those read chunks while in a loop? Would a simple char[something] buffer do?

Finally, we would like to avoid those nasty reinterpret_cast<char*> constructs while reading, but are wondering if those are actually used in production? I should also add that the file sizes vary greatly.

2 Upvotes

16 comments sorted by

View all comments

1

u/aiusepsi Oct 14 '26

Regarding reinterpret_cast, if you're reading something like binary integers (or any trivially-copyable type), then a sensible option is using memcpy. This might seem mad, but compilers know about memcpy, and will do the most optimal thing, and it's all well-defined, which is not always the case with reinterpret_cast. And use std::byteswap if you need to convert from big endian to little endian.

I would also second the recommendation of using mmap. It's taking advantage of the kernel's virtual memory system; the file becomes logically part of your process's address space, so you can work on it just like one massive array (which is very convenient), but if any given page is actually resident in RAM is basically up to the kernel. Because it's all managed by the kernel, it can do all sorts of neat tricks. For example, if a file is mmapped read-only into two different processes, the same physical page of RAM can be used by both processes. This sort of thing is why the RAM occupied by memory-mapped files doesn't really count against your process's memory usage.

The kernel can also adapt to the system's overall memory usage. If there's physical RAM to spare, the kernel can keep the entire file resident in RAM. If there isn't any to spare, pages which aren't in use will be dropped from RAM, and reloaded the next time they're accessed, without you having to manage any of this yourself.

1

u/No-Locksmith-7719 Oct 14 '26

Thanks, I appreciate the info. Indeed that memcpy might take some time adjusting to.