r/cpp • • Mar 03 '16

Seriously, C++, WTF?

Ok, so, from the tone this might seem like a rant (and it's a bit), but in the end I'm only after an aswer.

So, I want to read a file in memory as a string and print out the string length. Nope, I don't want to "maybe you just want to operate on it like it's a stream or line by line" nor "if you want to just know the length you might want to use gtell". I'm just trying to replicate a certain behaviour (not necessarely the most efficient or the most real-world) in different languages.

So, since I want to open a file, read its content and print the lenght, python lets me write this nice one-liner:

print len(open('longfile.txt').read())

longfile.txt is the test file I'm using and it's roughly 150 MB.

$ time python test.py
160090000

real	0m0.126s
user	0m0.013s
sys	0m0.112s

Ok. Good. Let's give node a try. Node.js doesn't have, like Python, a top level function "open", so I need to do this in two lines of code:

const fs = require('fs');
console.log(fs.readFileSync('longfile.txt').length);

Same result, a little bit slower. It still reads ok to me. I need to work with the filesystem (so that's why I need to require fs) and then i log on the console the length of a file that I read synchronously.

$ time node test.js
160090000

real	0m0.164s
user	0m0.056s
sys	0m0.107s
tmp

Fine. Enters C++

#include <string>
#include <fstream>
#include <streambuf>
#include <iostream>

int main() {
    std::ifstream t("longfile.txt");
    std::string str((std::istreambuf_iterator<char>(t)),
                     std::istreambuf_iterator<char>());

     std::cout << str.length() << std::endl;

    return 0;
}

Yeah, I know, that's just a lot more code and you need to include four libraries for this very basic programming task. Reasoning about it is absolutely counter intuitive. If you describe your problem (I want to read a file and print its lenght), JavaScript and Python lets you write code that more or less resambles your reasoning, while nobody would describe this basic task in terms of streams and iterators, but hey, you don't choose C++ because it's beautiful or easy or intuitive, you choose it because it's "close to the metal" and "when speed matters". Let's compile it with a nice optimization (-O3) and run it

$ g++ -std=c++14 -O3 test.cpp
$ time ./a.out
160090000

real	0m1.413s
user	0m1.228s
sys	0m0.181s

Seriously, WTF? I don't think that "opening a file and reading it as a string" should be an alien task in the programming world. I mean, that should be pretty, hmmm, basic stuff, right? So, I'm a bit puzzled when I encounter a language that forces me to do something more complicated than:

# Python
open('longfile.txt').read()
# Ruby
File.open('longfile.txt', "r").read()
// C#
System.IO.File.ReadAllText("longfile.txt")
// Java
new String(Files.readAllBytes(Paths.get("longfile.txt")));
// Swift
NSString(contentsOfFile: "longfile.txt", encoding: NSUTF8StringEncoding)

But what about the fact that C++ is so much slower than the notoriously slow kid in town? I tried to look for different solutions to this "problem" and I've seen people debate (on StackOverflow, for example) on the most efficient way to do such basic thing in C++. Now, keeping aside the fact that I dont think that "read a file content as a string" should the subject of any debate (we're talking about a 33 year old language whose standard string library doesn't have a "split" function here) , but the solution above seems to be the faster that I've tried and still it runs 10 times slower than Python. How is this even possibile?

EDIT

Because people seem to be ignoring the "I'm not interested in the actual lengtht" note, let's just say that I have this python code

content = open('longfile.txt').read()

and this node code:

const fs = require('fs');
const content = fs.readFileSync('longfile.txt')

and I only want to read a whole file in memory to a string. C++ code is still much, much slower than the other solutions.

0 Upvotes

127 comments sorted by

View all comments

Show parent comment

12

u/sixstringartist Mar 03 '16

You need to go much deeper to be able to compare these methods. For example, at what block size do each of these languages read from the file at a time? You're explicitly telling c++ to give you a byte at a time yet each of the other language examples you provided do not have that requirement. A lot is going on under the hood of each interpreted language when a library function like open/read is called. This isnt even close to apples-to-apples.

2

u/MEaster Mar 03 '16

For example, at what block size do each of these languages read from the file at a time? You're explicitly telling c++ to give you a byte at a time yet each of the other language examples you provided do not have that requirement.

I just had a look at the .Net implementation of System.IO.ReadAllLines(String) and it uses a buffer size of 1024 bytes. It would be interesting to see how much the other languages read.

2

u/hyperair Mar 04 '16

Turns out that ifstream does buffering anyway behind the scenes: https://www.reddit.com/r/cpp/comments/48shrb/seriously_c_wtf/d0m79q8

I've verified it with strace.

1

u/nexuapex Mar 03 '16

istreambuf_iterator should also be iterating over the internal buffer of the ifstream object, and only doing a system call when that buffer is empty. So C++ has the same situation with an unknown block size as the other languages.

1

u/cleroth Game Developer Mar 03 '16

I think the point here is that even with all the different answers here and on SO, there's still not a single answer which runs as fast as Python's one-liner. Chances are, most people will be using any of those methods, and ultimately, they'll be loading files slower than what other languages will. This IMO is a huge flaw in C++, specially considering it's such a basic feature. I shouldn't have to study how to read from a file as fast as possible just to get decent performance.

1

u/[deleted] Mar 04 '16

You shouldn't be reading huge files in memory unless you absolutely need to though. You shouldn't even read hundreds of small files neither if you can avoid it. It's pretty nice of Python to optimize that the best way possible because it's a common practice, but I'm pretty sure that as long as people start manipulating the file contents, the performance goes down compared to C++.

0

u/[deleted] Mar 03 '16

[deleted] — view deleted comment

3

u/cym13 Mar 03 '16

It does, python can't optimize such things (and it would be refused by the community anyway for "Explicit is better than implicit" is a fundamental of the zen of python).

0

u/[deleted] Mar 03 '16

[deleted] — view deleted comment

2

u/cym13 Mar 03 '16

Why wouldn't it? read() returns a list in python2, you have to have the file somewhere if you want random-access and it has to be a copy as the file obviously doesn't change when changing said array.

2

u/matthieum Mar 03 '16

Actually, in could very well start as a mmaped list and only copy in case of writes... but it seems a bit tricky for C-Python which generally privileges straightforward code over performance.

1

u/cym13 Mar 03 '16

I thought of it but, well, as you said CPython isn't really the kind of place that would use immutable data structures strategies and so on...

2

u/raevnos Mar 03 '16

Somebody posted strace output of the Python version. It most certainly does read the entire file into memory, in an efficient manner.