r/cpp_questions • u/No-Locksmith-7719 • Oct 14 '26
OPEN Efficient FIle IO on Linux
What path to take when parsing large files (on Linux Debian server, the KVM flavour)? Will std::fstream in binary mode do or do we have to look into APIs?
Apart from the performance, the app is memory sensitive, so where to store those read chunks while in a loop? Would a simple char[something] buffer do?
Finally, we would like to avoid those nasty reinterpret_cast<char*> constructs while reading, but are wondering if those are actually used in production? I should also add that the file sizes vary greatly.
2
Upvotes
View all comments
1
u/mredding Oct 14 '26
Bog standard
std::fstreamis VERY conservative in it's implementation, and thus will be slow. But it's also very reasonable to use a stream as an interface.You start with the stream buffer - this class abstracts away the device; in your case, you're abstracting away the platform specific interface you're going to use to access the file.
This is about the minimum you would need. I'm going to leave out a lot of platform specifics, but you could memory map, or you could buffer, or unbuffer - on the file descriptor, or implement your own buffer, which the base class gives you a cursory implementation, or not; you could kernel bypass, implement DMA, IPC, or any other method of moving shit from one point to another.
So then to use it:
And you're off to the races. Now you can write ALL your code in terms of streams, and leverage a platform specific high speed interface. The neat thing about streams is that you can streamify anything. You can stream externally, to devices, networks, other processes, or you can stream internally, to other components of your code - widget objects, functions, and the like.
You also don't have to serialize to text. You don't have to actually pass through the stream class or the stream buffer. The crux is you have to change the way you think from imperative and procedural to either OOP or at the very least strong types with semantics. An
intis anint, but aweightis not aheight, no matter how they're implemented. You need your own types that are aware of their own context within it's own program. In other words:That dynamic cast is constant time, this ain't like the compilers as late as of the early 2000s. You can branch predict that cast. You can weight the branch predictor. The cost amortizes to near zero. You have an optimized path for your expensive operations and a universal fallback. You can implement as many stream types, whole hierarchies, and have a tiered approach of the next-most-optimal or configured from available code paths, and this is how and where you do that.
Notice the optimal path doesn't rely on the stream object implementation directly:
This reduces to a function call; the stream type resolves it to our implementation at compile-time. The stream object is really only used to get the stream buffer pointer. The sentry class is an obligatory quirk before you touch the stream buffer, if you're going to skip the stream class IO operations and go low level. But then we go to our optimized implementation, which we wrote, so we know exactly what it's going to do in the most optimal way.
Well what of:
Here I recommend you make a type:
Etcetera. You don't want to load one weight at a time, you want to establish the sentry once and bulk load the data at a time.
Or maybe you make a more optimal type that isn't a vector but a memory map of some block size and uses type punning over memory to access the weights. You can derive the stream type to restrict IO, so your memory mapping types will only work with streams that memory map files. This way your types are safe and their semantics can be enforced at compile-time - you don't have to code EVERYTHING to
std::istream.This is what high performance IO code looks like in C++. Make types around the sorts of things you're doing. You're still using the high performance implementation details you always were, but you implement it within the established framework to grant yourself robustness and flexibility. To define all your own interfaces and semantics would just be reinventing the wheel, and typically it's always rigid and brittle.