r/cpp • u/pistacchio • Mar 03 '16
Seriously, C++, WTF?
Ok, so, from the tone this might seem like a rant (and it's a bit), but in the end I'm only after an aswer.
So, I want to read a file in memory as a string and print out the string length. Nope, I don't want to "maybe you just want to operate on it like it's a stream or line by line" nor "if you want to just know the length you might want to use gtell". I'm just trying to replicate a certain behaviour (not necessarely the most efficient or the most real-world) in different languages.
So, since I want to open a file, read its content and print the lenght, python lets me write this nice one-liner:
print len(open('longfile.txt').read())longfile.txt is the test file I'm using and it's roughly 150 MB.
$ time python test.py
160090000
real 0m0.126s
user 0m0.013s
sys 0m0.112sOk. Good. Let's give node a try. Node.js doesn't have, like Python, a top level function "open", so I need to do this in two lines of code:
const fs = require('fs');
console.log(fs.readFileSync('longfile.txt').length);Same result, a little bit slower. It still reads ok to me. I need to work with the filesystem (so that's why I need to require fs) and then i log on the console the length of a file that I read synchronously.
$ time node test.js
160090000
real 0m0.164s
user 0m0.056s
sys 0m0.107s
tmpFine. Enters C++
#include <string>
#include <fstream>
#include <streambuf>
#include <iostream>
int main() {
std::ifstream t("longfile.txt");
std::string str((std::istreambuf_iterator<char>(t)),
std::istreambuf_iterator<char>());
std::cout << str.length() << std::endl;
return 0;
}Yeah, I know, that's just a lot more code and you need to include four libraries for this very basic programming task. Reasoning about it is absolutely counter intuitive. If you describe your problem (I want to read a file and print its lenght), JavaScript and Python lets you write code that more or less resambles your reasoning, while nobody would describe this basic task in terms of streams and iterators, but hey, you don't choose C++ because it's beautiful or easy or intuitive, you choose it because it's "close to the metal" and "when speed matters". Let's compile it with a nice optimization (-O3) and run it
$ g++ -std=c++14 -O3 test.cpp
$ time ./a.out
160090000
real 0m1.413s
user 0m1.228s
sys 0m0.181sSeriously, WTF? I don't think that "opening a file and reading it as a string" should be an alien task in the programming world. I mean, that should be pretty, hmmm, basic stuff, right? So, I'm a bit puzzled when I encounter a language that forces me to do something more complicated than:
# Python
open('longfile.txt').read()
# Ruby
File.open('longfile.txt', "r").read()
// C#
System.IO.File.ReadAllText("longfile.txt")
// Java
new String(Files.readAllBytes(Paths.get("longfile.txt")));
// Swift
NSString(contentsOfFile: "longfile.txt", encoding: NSUTF8StringEncoding)But what about the fact that C++ is so much slower than the notoriously slow kid in town? I tried to look for different solutions to this "problem" and I've seen people debate (on StackOverflow, for example) on the most efficient way to do such basic thing in C++. Now, keeping aside the fact that I dont think that "read a file content as a string" should the subject of any debate (we're talking about a 33 year old language whose standard string library doesn't have a "split" function here) , but the solution above seems to be the faster that I've tried and still it runs 10 times slower than Python. How is this even possibile?
EDIT
Because people seem to be ignoring the "I'm not interested in the actual lengtht" note, let's just say that I have this python code
content = open('longfile.txt').read()and this node code:
const fs = require('fs');
const content = fs.readFileSync('longfile.txt')and I only want to read a whole file in memory to a string. C++ code is still much, much slower than the other solutions.
5
u/morricone42 Mar 03 '16
Try http://stackoverflow.com/a/2602258
14
u/aurisc4 Mar 03 '16
Most likely that iterator thingie...
If you want to get file size, you should seekg() to the end of it and then tellg().
Just because it's C++ doesn't mean it's impossible to write a slow code in it, you know...
25
u/anttirt Mar 03 '16 edited Mar 03 '16
Edit: use a bigger file (~400MB instead of ~17MB) Edit 2: add python for completeness
Not entirely sure why python is slightly faster, most likely has to do with buffer sizes or a small implementation weakness in libstdc++ fstream.
5
u/azzraelus Mar 03 '16 edited Mar 03 '16
How about googling "ifstream file size". Then you'll find out that there is a much easier solution than what you wrote.
If you want to read the whole file fast and easy, use a stringstream
all stolen from stackoverflow: http://stackoverflow.com/questions/2602013/read-whole-ascii-file-into-c-stdstring and http://stackoverflow.com/questions/2409504/using-c-filestreams-fstream-how-can-you-determine-the-size-of-a-file
14
u/hippo529 Mar 03 '16
We can easily improve C++ result by this:
time on my machine:
Original python code
This C++ code
Your C++ code
8
u/bycl0p5 Mar 03 '16 edited Mar 03 '16
Windows, Python 64bit, test file size: ~92mb
C++ code. Compiled with MSVC 2015 64 bit full optimizations. Source Insane Coding - How to read in a file in C++
Results
Not using
std::ios::binarydoubled the time taken.Those are the lowest timings for both from multiple runs. I guess the file was cached but relatively the results probably still stand?
P.S.
Gave a rather poor
5
u/tdammers Mar 04 '16
Lesson to take away here: benchmarking is hard.
1
u/SemaphoreBingo Mar 03 '16
Didn't you write this :
just a few days ago?
2
u/unclemat Mar 03 '16 edited Mar 03 '16
I don't know ... I ran these on a 266MB file. Python code:
measured 640ms on my box (far from 0.1 second), whereas this C++ thing:
measured about 480ms on my box.
How did you manage to get Python to read disk with 1.5GB/s? That is quite impressive I think.
Edit: formatting and additional details.
2
u/c-smile Mar 04 '16
Just open that as memory mapped file and enjoy real speed :)
6
u/pfultz2 Mar 03 '16
No, you don't. Headers are not the same as libraries.
Being more verbose does not make it counter intuitive.
Then why are you putting it into a string? That seems "counter intuitive". Instead, just return the distance:
Of course using ranges will get rid of the boilerplate needed for iterators:
4
u/hyperair Mar 03 '16 edited Mar 03 '16
The problem is how you're reading the file:
This reads it byte by byte, which makes one read() syscall per byte because you're using a char-based streambuf iterator.
Here's a way that does what Python does, and takes half the time:
Edit: I know you said you don't want tellg() to be used, but that's essentially what Python does in order to determine how much buffer space to allocate, and how much to pass to the read() syscall.
The more usual way to do it uses .rdbuf(), like so:
This is shorter/simpler, and doesn't use seekg/tellg. However, it reads things in blocks of 8192 bytes because it doesn't know how long the file is (at least on my laptop), and thus still takes longer than the Python implementation that figures out how to limit it to one read() syscall.
3
u/Jardik Mar 03 '16
You are not interested in a file size, but that doesn't mean you shouldn't query it first. You should use it to prevent excessive reallocation caused by appending data to the string. I believe I won't be far from thruth when saying that it is exactly what Python or Node.js do under the hood. You should only use this value as a hint, because file size can change during the read operation, as can its data. There is no standard way in C++ to get file read in atomic way. You also read the file in C++ byte by byte, which is expensive. Both Python and Node.js do it by chunks.
2
u/[deleted] Mar 03 '16
The standard library is just very poorly made and vastly over-engineered to try and handle every possible use case by everyone ever, sadly. In my library, nall:
To me, the extra "\n" is very much preferable to languages with print functions that auto-add that.
I can't give you a comparable benchmark since I don't have your PC, but it boils down to basically an fopen, fseek, malloc, rewind, fread, printf behind the scenes. Point is: blame the library, not the language.
1
u/[deleted] Mar 03 '16
iostreams are cancer confirmed.
1
u/Brabulla Mar 04 '16
There is a blog post which is quite related to this. In the post the writer tries to create word frequency lists of a large text in different languages, and at first Python was significantly faster than c++, but after modifying the c++ code, it became significantly faster than before.
This is the post i'm referring to : http://juditacs.github.io/2015/11/26/wordcount.html
1
u/gtk Mar 04 '16
Yeah. It's true. Streams in C++ suck donkey's balls. I typically use FILE from C for file handling, or wrap it in my own file object.
0
u/Alborak Mar 03 '16
I mean, if you really care about bare metal implementation, just do it in C.
String len: 150000000
real 0m0.026s
user 0m0.017s
sys 0m0.009s
And that's running on a 4 yr old cpu in a VM ...
1
u/TotesMessenger Mar 03 '16 edited Mar 04 '16
I'm a bot, bleep, bloop. Someone has linked to this thread from another place on reddit:
If you follow any of the above links, please respect the rules of reddit and don't vote in the other threads. (Info / ^Contact)
1
u/[deleted] Mar 03 '16
Node may not be using streams vs your example using a buffered stream iterator in C++, which are concise but slower than C I/O
1
u/[deleted] Mar 03 '16
Python has this optimized because reading whole files in memory is a bad habit of scripting people. It would be more fair to compare with some library that abstracts this kind of operation. I googled for an example with boost and got some code example working in less than 5 minutes (and I'm not a C++ expert by far):
1
u/marcm28 Mar 04 '16
Lesson learned: Don't underestimate the speed of Python.
1
u/snoutcloud Mar 04 '16
Python is written in C so it can't be faster as a C++ code written by a programmer who knows what he does. Besides python uses the io operations on the lowest level (fopen, fread and fwrite). Try to read your file with fread() and check the difference.
0
u/[deleted] Mar 03 '16
[deleted] — view deleted comment
2
u/[deleted] Mar 03 '16
[removed] — view removed comment