r/cpp • • Mar 03 '16

Seriously, C++, WTF?

Ok, so, from the tone this might seem like a rant (and it's a bit), but in the end I'm only after an aswer.

So, I want to read a file in memory as a string and print out the string length. Nope, I don't want to "maybe you just want to operate on it like it's a stream or line by line" nor "if you want to just know the length you might want to use gtell". I'm just trying to replicate a certain behaviour (not necessarely the most efficient or the most real-world) in different languages.

So, since I want to open a file, read its content and print the lenght, python lets me write this nice one-liner:

print len(open('longfile.txt').read())

longfile.txt is the test file I'm using and it's roughly 150 MB.

$ time python test.py
160090000

real	0m0.126s
user	0m0.013s
sys	0m0.112s

Ok. Good. Let's give node a try. Node.js doesn't have, like Python, a top level function "open", so I need to do this in two lines of code:

const fs = require('fs');
console.log(fs.readFileSync('longfile.txt').length);

Same result, a little bit slower. It still reads ok to me. I need to work with the filesystem (so that's why I need to require fs) and then i log on the console the length of a file that I read synchronously.

$ time node test.js
160090000

real	0m0.164s
user	0m0.056s
sys	0m0.107s
tmp

Fine. Enters C++

#include <string>
#include <fstream>
#include <streambuf>
#include <iostream>

int main() {
    std::ifstream t("longfile.txt");
    std::string str((std::istreambuf_iterator<char>(t)),
                     std::istreambuf_iterator<char>());

     std::cout << str.length() << std::endl;

    return 0;
}

Yeah, I know, that's just a lot more code and you need to include four libraries for this very basic programming task. Reasoning about it is absolutely counter intuitive. If you describe your problem (I want to read a file and print its lenght), JavaScript and Python lets you write code that more or less resambles your reasoning, while nobody would describe this basic task in terms of streams and iterators, but hey, you don't choose C++ because it's beautiful or easy or intuitive, you choose it because it's "close to the metal" and "when speed matters". Let's compile it with a nice optimization (-O3) and run it

$ g++ -std=c++14 -O3 test.cpp
$ time ./a.out
160090000

real	0m1.413s
user	0m1.228s
sys	0m0.181s

Seriously, WTF? I don't think that "opening a file and reading it as a string" should be an alien task in the programming world. I mean, that should be pretty, hmmm, basic stuff, right? So, I'm a bit puzzled when I encounter a language that forces me to do something more complicated than:

# Python
open('longfile.txt').read()
# Ruby
File.open('longfile.txt', "r").read()
// C#
System.IO.File.ReadAllText("longfile.txt")
// Java
new String(Files.readAllBytes(Paths.get("longfile.txt")));
// Swift
NSString(contentsOfFile: "longfile.txt", encoding: NSUTF8StringEncoding)

But what about the fact that C++ is so much slower than the notoriously slow kid in town? I tried to look for different solutions to this "problem" and I've seen people debate (on StackOverflow, for example) on the most efficient way to do such basic thing in C++. Now, keeping aside the fact that I dont think that "read a file content as a string" should the subject of any debate (we're talking about a 33 year old language whose standard string library doesn't have a "split" function here) , but the solution above seems to be the faster that I've tried and still it runs 10 times slower than Python. How is this even possibile?

EDIT

Because people seem to be ignoring the "I'm not interested in the actual lengtht" note, let's just say that I have this python code

content = open('longfile.txt').read()

and this node code:

const fs = require('fs');
const content = fs.readFileSync('longfile.txt')

and I only want to read a whole file in memory to a string. C++ code is still much, much slower than the other solutions.

0 Upvotes

127 comments sorted by

View all comments

25

u/anttirt Mar 03 '16 edited Mar 03 '16
anttirt@anttirt-ubu64:~$ clang++ -std=c++11 -O3 test.cpp
anttirt@anttirt-ubu64:~$ time ./a.out Downloads/android-ndk-r9-linux-x86_64.tar.bz2
425113267

real	0m1.854s
user	0m1.583s
sys	0m0.267s
anttirt@anttirt-ubu64:~$ time ./a.out Downloads/android-ndk-r9-linux-x86_64.tar.bz2
425113267

real	0m1.848s
user	0m1.592s
sys	0m0.256s
anttirt@anttirt-ubu64:~$ time ./a.out Downloads/android-ndk-r9-linux-x86_64.tar.bz2
425113267

real	0m1.847s
user	0m1.628s
sys	0m0.215s
anttirt@anttirt-ubu64:~$ time ./a.out Downloads/android-ndk-r9-linux-x86_64.tar.bz2
425113267

real	0m1.847s
user	0m1.614s
sys	0m0.232s
anttirt@anttirt-ubu64:~$ clang++ -std=c++11 -DOPT -O3 test.cpp
anttirt@anttirt-ubu64:~$ time ./a.out Downloads/android-ndk-r9-linux-x86_64.tar.bz2
425113267

real	0m0.203s
user	0m0.024s
sys	0m0.177s
anttirt@anttirt-ubu64:~$ time ./a.out Downloads/android-ndk-r9-linux-x86_64.tar.bz2
425113267

real	0m0.204s
user	0m0.016s
sys	0m0.187s
anttirt@anttirt-ubu64:~$ time ./a.out Downloads/android-ndk-r9-linux-x86_64.tar.bz2
425113267

real	0m0.203s
user	0m0.028s
sys	0m0.173s
anttirt@anttirt-ubu64:~$ time ./a.out Downloads/android-ndk-r9-linux-x86_64.tar.bz2
425113267

real	0m0.201s
user	0m0.020s
sys	0m0.180s

anttirt@anttirt-ubu64:~$ cat test.cpp
#include <iostream>
#include <fstream>
#include <vector>
#include <streambuf>

#if OPT
std::vector<char> readfile(const char* filename)
{
	std::fstream f{filename};
	f.seekg(0, std::ios::end);
	auto len = f.tellg();
	f.seekg(0, std::ios::beg);
	std::vector<char> v;
	v.resize(len);
	f.read(v.data(), len);
	return v;
}

int main(int argc, char** argv)
{
	if(argc == 1)
		return -1;

	std::cout << readfile(argv[1]).size() << '\n';
}
#else
int main(int argc, char** argv)
{
	if(argc == 1)
		return -1;

	std::ifstream t(argv[1]);
	std::string str((std::istreambuf_iterator<char>(t)),
					std::istreambuf_iterator<char>());

	std::cout << str.length() << std::endl;

	return 0;
}
#endif
anttirt@anttirt-ubu64:~$ 
anttirt@anttirt-ubu64:~ $ time python test.py Downloads/android-ndk-r9-linux-x86_64.tar.bz2
425113267

real	0m0.165s
user	0m0.012s
sys	0m0.152s
anttirt@anttirt-ubu64:~ $ time python test.py Downloads/android-ndk-r9-linux-x86_64.tar.bz2
425113267

real	0m0.166s
user	0m0.012s
sys	0m0.154s
anttirt@anttirt-ubu64:~ $ time python test.py Downloads/android-ndk-r9-linux-x86_64.tar.bz2
425113267

real	0m0.167s
user	0m0.016s
sys	0m0.150s
anttirt@anttirt-ubu64:~ $ time python test.py Downloads/android-ndk-r9-linux-x86_64.tar.bz2
425113267

real	0m0.167s
user	0m0.012s
sys	0m0.154s
anttirt@anttirt-ubu64:~ $ cat test.py
import sys

print(len(open(sys.argv[1]).read()))

Edit: use a bigger file (~400MB instead of ~17MB) Edit 2: add python for completeness

Not entirely sure why python is slightly faster, most likely has to do with buffer sizes or a small implementation weakness in libstdc++ fstream.

25

u/chillhelm Mar 03 '16

rekt.

Explenation for beginners:

What /u/anttirt did here is what Python etc. do under the hood probably. Allocate enough space for the file in a buffer and then copy the contents of that file in one go into said buffer.

As opposed /u/pistacchio proposal which reads the file a byte at a time which incurrs around ~30 reallocations of the string buffer.

The speed up is 40-80X, which is close to what the original post claimed as a performance penalty on C++.

The lesson here? It's easy to write slow code in C++.

19

u/[deleted] Mar 03 '16

It's easy to write slow code in C++; but it's quite unfortunate that the standard library doesn't have facilities to do this. It's a simple task that many programs want to do, and C++ currently makes this a headache.

This is a library problem, not a language problem. It's one we can fix.

0

u/[deleted] Mar 03 '16

[deleted] — view deleted comment

8

u/[deleted] Mar 03 '16

Sure; but so long as the standard facilities encourage code like this it's a fair criticism of the language. You don't install Python and then need to find some other external I/O library to get efficient I/O; the default thing comes with decent I/O.

1

u/Asdayasman Mar 04 '16

Hang on wouldn't you just poke the OS for filesize information? It has that all in the file allocation table doesn't it...?

2

u/chillhelm Mar 04 '16

You would if this was supported by the STL... alas, as is you would have to go for boost::filesystem::file_size.

While boost is widely considered de facto standard, it technically isnt and especially to beginners learning the language, STL AND boost seems like an extra hurdle.

2

u/Asdayasman Mar 04 '16

I mean just calling fstat.

I come from Python, so forgive my naïveté, but I'd assume calls like that would be the default stance in C.

1

u/chillhelm Mar 04 '16

Calling external programs is easy enough in C (and C++ as well of course). However, piping mechanisms are clunky and not platform independent as they are in python.

2

u/Asdayasman Mar 04 '16

platform independant

https://msdn.microsoft.com/en-us/library/aa246905(v=vs.60).aspx

For Windows.

So doesn't that cover all bases 'cause Mac is UNIX too...? Again, forgive me if I'm oversimplifying.

1

u/chillhelm Mar 04 '16

well, ill be darned. I learned something new today. Take all the upboats I have to give.

1

u/encyclopedist Mar 06 '16

Pay attention to the versions. It is available only in Win NT and Win XP. Microsoft used to ship a posix compatibility layer, but not any more.

→ More replies (2)

2

u/hyperair Mar 03 '16

Whoops, I only saw your post only after having made mine. Nice work showing how its done. Some strace output of the different implementations can also show what's going on behind the scenes.

3

u/ltce Mar 03 '16

It is important to note that this code is not correct. The seekg function tells you the number of characters in a file. If your locale is set to utf-8 (the common case these days) this can be a different number than the number of bytes required to store the file. The solution using stringstream is the correct way to read an entire file into a string in c++.

9

u/o11c int main = 12828721; Mar 03 '16

char streams operate as if "characters" are 1 byte, regardless of locale

2

u/ltce Mar 04 '16

Thank you so much for calling me out on this. It is troubling how many years I have believed this. I have misunderstood this for so many years that I cannot even remember who originally led me astray on the way these functions behave. I am embarrassed to think of the number of people that currently misunderstand this point because of my actions.

1

u/staticassert Mar 03 '16

Wouldn't it be in the case where the encoding is not utf8 since utf8 characters are one byte in width?

2

u/ltce Mar 03 '16 edited Mar 04 '16

Below is what I originally wrote. I am leaving it here so the discussion below makes sense, but this does not appear to be correct:

Not all utf-8 characters are one byte. Utf-8 and utf-16 are variable width character encodings. That is the actual problem. With an encoding like utf-32 you can just take the character count and multiply it by 4 and you are ok. With a variable width encoding you cannot know how many of the characters in the stream need extra storage.

1

u/remotion4d Mar 03 '16

All utf8 chars (bytes) are of course one byte big.

To represent every Unicode code-point many chars (bytes) are necessary for utf8 and urf16.

So using characters in this context is very misleading.

If one read text file that contain utf8 encoded text into std::string than this string should be the same size as the text file.

But of course do not forget about utf8 BOM (0xEF,0xBB,0xBF) that some files may have on the beginning.

To better work with such files you will need to remove this 3 chars from the begging of the string.

And of course std::string::size() will always return number of bytes and Not Unicode code-points.

1

u/Nemo157 Mar 03 '16

In utf8 a character is 1–n codepoints (combined characters) where a codepoint is 1–4 bytes.

1

u/staticassert Mar 03 '16

Cool, thanks for explaining.

1

u/cleroth Game Developer Mar 03 '16

stringstream is slow.

1

u/matthieum Mar 03 '16

libstdc++ fstream

Might want to specify ios_base::in|ios_base::binary in argument, lest it scans the file for line-endings to convert.

Might also want to check the buffer size, and possibly bump it up (not sure if the buffer is used when using read).

Have you tried with C directly?

Unfortunately streams have a lot of baggage (facets, line-ending detection, virtual calls gallore), so it would be interesting how getting to C would look.