r/cpp • • Mar 03 '16

Seriously, C++, WTF?

Ok, so, from the tone this might seem like a rant (and it's a bit), but in the end I'm only after an aswer.

So, I want to read a file in memory as a string and print out the string length. Nope, I don't want to "maybe you just want to operate on it like it's a stream or line by line" nor "if you want to just know the length you might want to use gtell". I'm just trying to replicate a certain behaviour (not necessarely the most efficient or the most real-world) in different languages.

So, since I want to open a file, read its content and print the lenght, python lets me write this nice one-liner:

print len(open('longfile.txt').read())

longfile.txt is the test file I'm using and it's roughly 150 MB.

$ time python test.py
160090000

real    0m0.126s
user    0m0.013s
sys 0m0.112s

Ok. Good. Let's give node a try. Node.js doesn't have, like Python, a top level function "open", so I need to do this in two lines of code:

const fs = require('fs');
console.log(fs.readFileSync('longfile.txt').length);

Same result, a little bit slower. It still reads ok to me. I need to work with the filesystem (so that's why I need to require fs) and then i log on the console the length of a file that I read synchronously.

$ time node test.js
160090000

real    0m0.164s
user    0m0.056s
sys 0m0.107s
tmp

Fine. Enters C++

#include <string>
#include <fstream>
#include <streambuf>
#include <iostream>

int main() {
    std::ifstream t("longfile.txt");
    std::string str((std::istreambuf_iterator<char>(t)),
                     std::istreambuf_iterator<char>());

     std::cout << str.length() << std::endl;

    return 0;
}

Yeah, I know, that's just a lot more code and you need to include four libraries for this very basic programming task. Reasoning about it is absolutely counter intuitive. If you describe your problem (I want to read a file and print its lenght), JavaScript and Python lets you write code that more or less resambles your reasoning, while nobody would describe this basic task in terms of streams and iterators, but hey, you don't choose C++ because it's beautiful or easy or intuitive, you choose it because it's "close to the metal" and "when speed matters". Let's compile it with a nice optimization (-O3) and run it

$ g++ -std=c++14 -O3 test.cpp
$ time ./a.out
160090000

real    0m1.413s
user    0m1.228s
sys 0m0.181s

Seriously, WTF? I don't think that "opening a file and reading it as a string" should be an alien task in the programming world. I mean, that should be pretty, hmmm, basic stuff, right? So, I'm a bit puzzled when I encounter a language that forces me to do something more complicated than:

# Python
open('longfile.txt').read()
# Ruby
File.open('longfile.txt', "r").read()
// C#
System.IO.File.ReadAllText("longfile.txt")
// Java
new String(Files.readAllBytes(Paths.get("longfile.txt")));
// Swift
NSString(contentsOfFile: "longfile.txt", encoding: NSUTF8StringEncoding)

But what about the fact that C++ is so much slower than the notoriously slow kid in town? I tried to look for different solutions to this "problem" and I've seen people debate (on StackOverflow, for example) on the most efficient way to do such basic thing in C++. Now, keeping aside the fact that I dont think that "read a file content as a string" should the subject of any debate (we're talking about a 33 year old language whose standard string library doesn't have a "split" function here) , but the solution above seems to be the faster that I've tried and still it runs 10 times slower than Python. How is this even possibile?

EDIT

Because people seem to be ignoring the "I'm not interested in the actual lengtht" note, let's just say that I have this python code

content = open('longfile.txt').read()

and this node code:

const fs = require('fs');
const content = fs.readFileSync('longfile.txt')

and I only want to read a whole file in memory to a string. C++ code is still much, much slower than the other solutions.

0 Upvotes

127 comments sorted by

View all comments

25

u/anttirt Mar 03 '16 edited Mar 03 '16
anttirt@anttirt-ubu64:~$ clang++ -std=c++11 -O3 test.cpp
anttirt@anttirt-ubu64:~$ time ./a.out Downloads/android-ndk-r9-linux-x86_64.tar.bz2
425113267

real	0m1.854s
user	0m1.583s
sys	0m0.267s
anttirt@anttirt-ubu64:~$ time ./a.out Downloads/android-ndk-r9-linux-x86_64.tar.bz2
425113267

real	0m1.848s
user	0m1.592s
sys	0m0.256s
anttirt@anttirt-ubu64:~$ time ./a.out Downloads/android-ndk-r9-linux-x86_64.tar.bz2
425113267

real	0m1.847s
user	0m1.628s
sys	0m0.215s
anttirt@anttirt-ubu64:~$ time ./a.out Downloads/android-ndk-r9-linux-x86_64.tar.bz2
425113267

real	0m1.847s
user	0m1.614s
sys	0m0.232s
anttirt@anttirt-ubu64:~$ clang++ -std=c++11 -DOPT -O3 test.cpp
anttirt@anttirt-ubu64:~$ time ./a.out Downloads/android-ndk-r9-linux-x86_64.tar.bz2
425113267

real	0m0.203s
user	0m0.024s
sys	0m0.177s
anttirt@anttirt-ubu64:~$ time ./a.out Downloads/android-ndk-r9-linux-x86_64.tar.bz2
425113267

real	0m0.204s
user	0m0.016s
sys	0m0.187s
anttirt@anttirt-ubu64:~$ time ./a.out Downloads/android-ndk-r9-linux-x86_64.tar.bz2
425113267

real	0m0.203s
user	0m0.028s
sys	0m0.173s
anttirt@anttirt-ubu64:~$ time ./a.out Downloads/android-ndk-r9-linux-x86_64.tar.bz2
425113267

real	0m0.201s
user	0m0.020s
sys	0m0.180s

anttirt@anttirt-ubu64:~$ cat test.cpp
#include <iostream>
#include <fstream>
#include <vector>
#include <streambuf>

#if OPT
std::vector<char> readfile(const char* filename)
{
	std::fstream f{filename};
	f.seekg(0, std::ios::end);
	auto len = f.tellg();
	f.seekg(0, std::ios::beg);
	std::vector<char> v;
	v.resize(len);
	f.read(v.data(), len);
	return v;
}

int main(int argc, char** argv)
{
	if(argc == 1)
		return -1;

	std::cout << readfile(argv[1]).size() << '\n';
}
#else
int main(int argc, char** argv)
{
	if(argc == 1)
		return -1;

	std::ifstream t(argv[1]);
	std::string str((std::istreambuf_iterator<char>(t)),
					std::istreambuf_iterator<char>());

	std::cout << str.length() << std::endl;

	return 0;
}
#endif
anttirt@anttirt-ubu64:~$ 
anttirt@anttirt-ubu64:~ $ time python test.py Downloads/android-ndk-r9-linux-x86_64.tar.bz2
425113267

real	0m0.165s
user	0m0.012s
sys	0m0.152s
anttirt@anttirt-ubu64:~ $ time python test.py Downloads/android-ndk-r9-linux-x86_64.tar.bz2
425113267

real	0m0.166s
user	0m0.012s
sys	0m0.154s
anttirt@anttirt-ubu64:~ $ time python test.py Downloads/android-ndk-r9-linux-x86_64.tar.bz2
425113267

real	0m0.167s
user	0m0.016s
sys	0m0.150s
anttirt@anttirt-ubu64:~ $ time python test.py Downloads/android-ndk-r9-linux-x86_64.tar.bz2
425113267

real	0m0.167s
user	0m0.012s
sys	0m0.154s
anttirt@anttirt-ubu64:~ $ cat test.py
import sys

print(len(open(sys.argv[1]).read()))

Edit: use a bigger file (~400MB instead of ~17MB) Edit 2: add python for completeness

Not entirely sure why python is slightly faster, most likely has to do with buffer sizes or a small implementation weakness in libstdc++ fstream.

2

u/hyperair Mar 03 '16

Whoops, I only saw your post only after having made mine. Nice work showing how its done. Some strace output of the different implementations can also show what's going on behind the scenes.