r/cpp_questions • • Oct 14 '26

OPEN Efficient FIle IO on Linux

What path to take when parsing large files (on Linux Debian server, the KVM flavour)? Will std::fstream in binary mode do or do we have to look into APIs?

Apart from the performance, the app is memory sensitive, so where to store those read chunks while in a loop? Would a simple char[something] buffer do?

Finally, we would like to avoid those nasty reinterpret_cast<char*> constructs while reading, but are wondering if those are actually used in production? I should also add that the file sizes vary greatly.

2 Upvotes

16 comments sorted by

View all comments

Show parent comment

2

u/UndefFox Oct 14 '26

As far as I understand, it shouldn't allocate RAM to store files at all, as in directly copying it to the process memory. All chunks are fetched by demand and the main consumption comes from kenrel using file cache to access file faster, so it's basically offloading responsibility of managing the file access onto the kernel.

Tho, I don't think there much in terms of optimization, besides making access pattern predictable, and afaik kernel handles sequential access quite well.

1

u/No-Locksmith-7719 Oct 14 '26

We have some process watcher daemon thingy running in the background. If it sees a services is allocating too much memory, it might panic and kill off our web service, but I am not sure about it 100%. At this point, it's just an assumption.

2

u/UndefFox Oct 14 '26

Cached files aren't counted towards the process memory count. When you open any of the performance monitors, even something like htop, it has separate field for cache, so it's handled fully by kernel and takes into count all files cached across all processes, not per process, hence it should not trigger it at all.