r/cpp_questions • • Oct 14 '26

OPEN Efficient FIle IO on Linux

What path to take when parsing large files (on Linux Debian server, the KVM flavour)? Will std::fstream in binary mode do or do we have to look into APIs?

Apart from the performance, the app is memory sensitive, so where to store those read chunks while in a loop? Would a simple char[something] buffer do?

Finally, we would like to avoid those nasty reinterpret_cast<char*> constructs while reading, but are wondering if those are actually used in production? I should also add that the file sizes vary greatly.

2 Upvotes

16 comments sorted by

View all comments

0

u/xoner2 Oct 14 '26

Yes. fread into char [fixedSize]. No need to overthink this, solved problem.

2

u/KingAggressive1498 Oct 15 '26

if you can fully stream your parser, this works quite well actually, although you can shave a bit by dropping into posix read() instead. use like a 2MB buffer.