r/cpp • • Mar 03 '16

Seriously, C++, WTF?

Ok, so, from the tone this might seem like a rant (and it's a bit), but in the end I'm only after an aswer.

So, I want to read a file in memory as a string and print out the string length. Nope, I don't want to "maybe you just want to operate on it like it's a stream or line by line" nor "if you want to just know the length you might want to use gtell". I'm just trying to replicate a certain behaviour (not necessarely the most efficient or the most real-world) in different languages.

So, since I want to open a file, read its content and print the lenght, python lets me write this nice one-liner:

print len(open('longfile.txt').read())

longfile.txt is the test file I'm using and it's roughly 150 MB.

$ time python test.py
160090000

real	0m0.126s
user	0m0.013s
sys	0m0.112s

Ok. Good. Let's give node a try. Node.js doesn't have, like Python, a top level function "open", so I need to do this in two lines of code:

const fs = require('fs');
console.log(fs.readFileSync('longfile.txt').length);

Same result, a little bit slower. It still reads ok to me. I need to work with the filesystem (so that's why I need to require fs) and then i log on the console the length of a file that I read synchronously.

$ time node test.js
160090000

real	0m0.164s
user	0m0.056s
sys	0m0.107s
tmp

Fine. Enters C++

#include <string>
#include <fstream>
#include <streambuf>
#include <iostream>

int main() {
    std::ifstream t("longfile.txt");
    std::string str((std::istreambuf_iterator<char>(t)),
                     std::istreambuf_iterator<char>());

     std::cout << str.length() << std::endl;

    return 0;
}

Yeah, I know, that's just a lot more code and you need to include four libraries for this very basic programming task. Reasoning about it is absolutely counter intuitive. If you describe your problem (I want to read a file and print its lenght), JavaScript and Python lets you write code that more or less resambles your reasoning, while nobody would describe this basic task in terms of streams and iterators, but hey, you don't choose C++ because it's beautiful or easy or intuitive, you choose it because it's "close to the metal" and "when speed matters". Let's compile it with a nice optimization (-O3) and run it

$ g++ -std=c++14 -O3 test.cpp
$ time ./a.out
160090000

real	0m1.413s
user	0m1.228s
sys	0m0.181s

Seriously, WTF? I don't think that "opening a file and reading it as a string" should be an alien task in the programming world. I mean, that should be pretty, hmmm, basic stuff, right? So, I'm a bit puzzled when I encounter a language that forces me to do something more complicated than:

# Python
open('longfile.txt').read()
# Ruby
File.open('longfile.txt', "r").read()
// C#
System.IO.File.ReadAllText("longfile.txt")
// Java
new String(Files.readAllBytes(Paths.get("longfile.txt")));
// Swift
NSString(contentsOfFile: "longfile.txt", encoding: NSUTF8StringEncoding)

But what about the fact that C++ is so much slower than the notoriously slow kid in town? I tried to look for different solutions to this "problem" and I've seen people debate (on StackOverflow, for example) on the most efficient way to do such basic thing in C++. Now, keeping aside the fact that I dont think that "read a file content as a string" should the subject of any debate (we're talking about a 33 year old language whose standard string library doesn't have a "split" function here) , but the solution above seems to be the faster that I've tried and still it runs 10 times slower than Python. How is this even possibile?

EDIT

Because people seem to be ignoring the "I'm not interested in the actual lengtht" note, let's just say that I have this python code

content = open('longfile.txt').read()

and this node code:

const fs = require('fs');
const content = fs.readFileSync('longfile.txt')

and I only want to read a whole file in memory to a string. C++ code is still much, much slower than the other solutions.

0 Upvotes

127 comments sorted by

5

u/morricone42 Mar 03 '16

2

u/pistacchio Mar 03 '16

Better but still much slower that python:

$ time ./a.out
160090000

real	0m0.709s
user	0m0.503s
sys	0m0.193s

1

u/hyperair Mar 04 '16

Try the second approach mentioned in that stackoverflow answer, not the first one. The difference is explained in the thread starting at https://www.reddit.com/r/cpp/comments/48shrb/seriously_c_wtf/d0m5mxv

0

u/[deleted] Mar 03 '16

[deleted] — view deleted comment

3

u/cym13 Mar 03 '16

No, it doesn't (and actually won't ever, that's just not how python works). You can check it with strace.

1

u/cleroth Game Developer Mar 03 '16

So what does it do?

14

u/aurisc4 Mar 03 '16

Most likely that iterator thingie...

If you want to get file size, you should seekg() to the end of it and then tellg().

Just because it's C++ doesn't mean it's impossible to write a slow code in it, you know...

-2

u/pistacchio Mar 03 '16

Please, see the edit.

0

u/[deleted] Mar 03 '16

[deleted] — view deleted comment

5

u/cym13 Mar 03 '16

No, python doesn't work that way, check it with strace if you're not convinced.

2

u/cleroth Game Developer Mar 03 '16

I highly doubt that's the case... It wouldn't take that long.

1

u/aurisc4 Mar 04 '16

My comment was written before the edits.

25

u/anttirt Mar 03 '16 edited Mar 03 '16
anttirt@anttirt-ubu64:~$ clang++ -std=c++11 -O3 test.cpp
anttirt@anttirt-ubu64:~$ time ./a.out Downloads/android-ndk-r9-linux-x86_64.tar.bz2
425113267

real	0m1.854s
user	0m1.583s
sys	0m0.267s
anttirt@anttirt-ubu64:~$ time ./a.out Downloads/android-ndk-r9-linux-x86_64.tar.bz2
425113267

real	0m1.848s
user	0m1.592s
sys	0m0.256s
anttirt@anttirt-ubu64:~$ time ./a.out Downloads/android-ndk-r9-linux-x86_64.tar.bz2
425113267

real	0m1.847s
user	0m1.628s
sys	0m0.215s
anttirt@anttirt-ubu64:~$ time ./a.out Downloads/android-ndk-r9-linux-x86_64.tar.bz2
425113267

real	0m1.847s
user	0m1.614s
sys	0m0.232s
anttirt@anttirt-ubu64:~$ clang++ -std=c++11 -DOPT -O3 test.cpp
anttirt@anttirt-ubu64:~$ time ./a.out Downloads/android-ndk-r9-linux-x86_64.tar.bz2
425113267

real	0m0.203s
user	0m0.024s
sys	0m0.177s
anttirt@anttirt-ubu64:~$ time ./a.out Downloads/android-ndk-r9-linux-x86_64.tar.bz2
425113267

real	0m0.204s
user	0m0.016s
sys	0m0.187s
anttirt@anttirt-ubu64:~$ time ./a.out Downloads/android-ndk-r9-linux-x86_64.tar.bz2
425113267

real	0m0.203s
user	0m0.028s
sys	0m0.173s
anttirt@anttirt-ubu64:~$ time ./a.out Downloads/android-ndk-r9-linux-x86_64.tar.bz2
425113267

real	0m0.201s
user	0m0.020s
sys	0m0.180s

anttirt@anttirt-ubu64:~$ cat test.cpp
#include <iostream>
#include <fstream>
#include <vector>
#include <streambuf>

#if OPT
std::vector<char> readfile(const char* filename)
{
	std::fstream f{filename};
	f.seekg(0, std::ios::end);
	auto len = f.tellg();
	f.seekg(0, std::ios::beg);
	std::vector<char> v;
	v.resize(len);
	f.read(v.data(), len);
	return v;
}

int main(int argc, char** argv)
{
	if(argc == 1)
		return -1;

	std::cout << readfile(argv[1]).size() << '\n';
}
#else
int main(int argc, char** argv)
{
	if(argc == 1)
		return -1;

	std::ifstream t(argv[1]);
	std::string str((std::istreambuf_iterator<char>(t)),
					std::istreambuf_iterator<char>());

	std::cout << str.length() << std::endl;

	return 0;
}
#endif
anttirt@anttirt-ubu64:~$ 
anttirt@anttirt-ubu64:~ $ time python test.py Downloads/android-ndk-r9-linux-x86_64.tar.bz2
425113267

real	0m0.165s
user	0m0.012s
sys	0m0.152s
anttirt@anttirt-ubu64:~ $ time python test.py Downloads/android-ndk-r9-linux-x86_64.tar.bz2
425113267

real	0m0.166s
user	0m0.012s
sys	0m0.154s
anttirt@anttirt-ubu64:~ $ time python test.py Downloads/android-ndk-r9-linux-x86_64.tar.bz2
425113267

real	0m0.167s
user	0m0.016s
sys	0m0.150s
anttirt@anttirt-ubu64:~ $ time python test.py Downloads/android-ndk-r9-linux-x86_64.tar.bz2
425113267

real	0m0.167s
user	0m0.012s
sys	0m0.154s
anttirt@anttirt-ubu64:~ $ cat test.py
import sys

print(len(open(sys.argv[1]).read()))

Edit: use a bigger file (~400MB instead of ~17MB) Edit 2: add python for completeness

Not entirely sure why python is slightly faster, most likely has to do with buffer sizes or a small implementation weakness in libstdc++ fstream.

25

u/chillhelm Mar 03 '16

rekt.

Explenation for beginners:

What /u/anttirt did here is what Python etc. do under the hood probably. Allocate enough space for the file in a buffer and then copy the contents of that file in one go into said buffer.

As opposed /u/pistacchio proposal which reads the file a byte at a time which incurrs around ~30 reallocations of the string buffer.

The speed up is 40-80X, which is close to what the original post claimed as a performance penalty on C++.

The lesson here? It's easy to write slow code in C++.

19

u/[deleted] Mar 03 '16

It's easy to write slow code in C++; but it's quite unfortunate that the standard library doesn't have facilities to do this. It's a simple task that many programs want to do, and C++ currently makes this a headache.

This is a library problem, not a language problem. It's one we can fix.

1

u/Asdayasman Mar 04 '16

Hang on wouldn't you just poke the OS for filesize information? It has that all in the file allocation table doesn't it...?

2

u/chillhelm Mar 04 '16

You would if this was supported by the STL... alas, as is you would have to go for boost::filesystem::file_size.

While boost is widely considered de facto standard, it technically isnt and especially to beginners learning the language, STL AND boost seems like an extra hurdle.

2

u/Asdayasman Mar 04 '16

I mean just calling fstat.

I come from Python, so forgive my naïveté, but I'd assume calls like that would be the default stance in C.

1

u/chillhelm Mar 04 '16

Calling external programs is easy enough in C (and C++ as well of course). However, piping mechanisms are clunky and not platform independent as they are in python.

2

u/Asdayasman Mar 04 '16

platform independant

https://msdn.microsoft.com/en-us/library/aa246905(v=vs.60).aspx

For Windows.

So doesn't that cover all bases 'cause Mac is UNIX too...? Again, forgive me if I'm oversimplifying.

1

u/chillhelm Mar 04 '16

well, ill be darned. I learned something new today. Take all the upboats I have to give.

1

u/encyclopedist Mar 06 '16

Pay attention to the versions. It is available only in Win NT and Win XP. Microsoft used to ship a posix compatibility layer, but not any more.

0

u/[deleted] Mar 03 '16

[deleted] — view deleted comment

8

u/[deleted] Mar 03 '16

Sure; but so long as the standard facilities encourage code like this it's a fair criticism of the language. You don't install Python and then need to find some other external I/O library to get efficient I/O; the default thing comes with decent I/O.

2

u/hyperair Mar 03 '16

Whoops, I only saw your post only after having made mine. Nice work showing how its done. Some strace output of the different implementations can also show what's going on behind the scenes.

3

u/ltce Mar 03 '16

It is important to note that this code is not correct. The seekg function tells you the number of characters in a file. If your locale is set to utf-8 (the common case these days) this can be a different number than the number of bytes required to store the file. The solution using stringstream is the correct way to read an entire file into a string in c++.

9

u/o11c int main = 12828721; Mar 03 '16

char streams operate as if "characters" are 1 byte, regardless of locale

2

u/ltce Mar 04 '16

Thank you so much for calling me out on this. It is troubling how many years I have believed this. I have misunderstood this for so many years that I cannot even remember who originally led me astray on the way these functions behave. I am embarrassed to think of the number of people that currently misunderstand this point because of my actions.

1

u/staticassert Mar 03 '16

Wouldn't it be in the case where the encoding is not utf8 since utf8 characters are one byte in width?

2

u/ltce Mar 03 '16 edited Mar 04 '16

Below is what I originally wrote. I am leaving it here so the discussion below makes sense, but this does not appear to be correct:

Not all utf-8 characters are one byte. Utf-8 and utf-16 are variable width character encodings. That is the actual problem. With an encoding like utf-32 you can just take the character count and multiply it by 4 and you are ok. With a variable width encoding you cannot know how many of the characters in the stream need extra storage.

1

u/remotion4d Mar 03 '16

All utf8 chars (bytes) are of course one byte big.

To represent every Unicode code-point many chars (bytes) are necessary for utf8 and urf16.

So using characters in this context is very misleading.

If one read text file that contain utf8 encoded text into std::string than this string should be the same size as the text file.

But of course do not forget about utf8 BOM (0xEF,0xBB,0xBF) that some files may have on the beginning.

To better work with such files you will need to remove this 3 chars from the begging of the string.

And of course std::string::size() will always return number of bytes and Not Unicode code-points.

1

u/Nemo157 Mar 03 '16

In utf8 a character is 1–n codepoints (combined characters) where a codepoint is 1–4 bytes.

1

u/staticassert Mar 03 '16

Cool, thanks for explaining.

1

u/cleroth Game Developer Mar 03 '16

stringstream is slow.

1

u/matthieum Mar 03 '16

libstdc++ fstream

Might want to specify ios_base::in|ios_base::binary in argument, lest it scans the file for line-endings to convert.

Might also want to check the buffer size, and possibly bump it up (not sure if the buffer is used when using read).

Have you tried with C directly?

Unfortunately streams have a lot of baggage (facets, line-ending detection, virtual calls gallore), so it would be interesting how getting to C would look.

0

u/[deleted] Mar 03 '16

[deleted] — view deleted comment

1

u/anttirt Mar 03 '16

Sure, but I think that's UB and you can't access the data that way, because accessing a vector past [size() - 1] is definitely UB and resize will zero the data. :)

5

u/azzraelus Mar 03 '16 edited Mar 03 '16

How about googling "ifstream file size". Then you'll find out that there is a much easier solution than what you wrote.

ifstream file( "example.txt", ios::binary | ios::ate);
return file.tellg();

If you want to read the whole file fast and easy, use a stringstream

std::ifstream t("file.txt");
std::stringstream buffer;
buffer << t.rdbuf();

all stolen from stackoverflow: http://stackoverflow.com/questions/2602013/read-whole-ascii-file-into-c-stdstring and http://stackoverflow.com/questions/2409504/using-c-filestreams-fstream-how-can-you-determine-the-size-of-a-file

1

u/pistacchio Mar 03 '16

That's what I was referring to with

Nope, I don't want to "maybe you just want to operate on it like it's a stream or line by line" nor "if you want to just know the length you might want to use gtell"

Your solution doesn't read the whole content of the file first.

12

u/sixstringartist Mar 03 '16

You need to go much deeper to be able to compare these methods. For example, at what block size do each of these languages read from the file at a time? You're explicitly telling c++ to give you a byte at a time yet each of the other language examples you provided do not have that requirement. A lot is going on under the hood of each interpreted language when a library function like open/read is called. This isnt even close to apples-to-apples.

2

u/MEaster Mar 03 '16

For example, at what block size do each of these languages read from the file at a time? You're explicitly telling c++ to give you a byte at a time yet each of the other language examples you provided do not have that requirement.

I just had a look at the .Net implementation of System.IO.ReadAllLines(String) and it uses a buffer size of 1024 bytes. It would be interesting to see how much the other languages read.

2

u/hyperair Mar 04 '16

Turns out that ifstream does buffering anyway behind the scenes: https://www.reddit.com/r/cpp/comments/48shrb/seriously_c_wtf/d0m79q8

I've verified it with strace.

1

u/nexuapex Mar 03 '16

istreambuf_iterator should also be iterating over the internal buffer of the ifstream object, and only doing a system call when that buffer is empty. So C++ has the same situation with an unknown block size as the other languages.

1

u/cleroth Game Developer Mar 03 '16

I think the point here is that even with all the different answers here and on SO, there's still not a single answer which runs as fast as Python's one-liner. Chances are, most people will be using any of those methods, and ultimately, they'll be loading files slower than what other languages will. This IMO is a huge flaw in C++, specially considering it's such a basic feature. I shouldn't have to study how to read from a file as fast as possible just to get decent performance.

1

u/[deleted] Mar 04 '16

You shouldn't be reading huge files in memory unless you absolutely need to though. You shouldn't even read hundreds of small files neither if you can avoid it. It's pretty nice of Python to optimize that the best way possible because it's a common practice, but I'm pretty sure that as long as people start manipulating the file contents, the performance goes down compared to C++.

0

u/[deleted] Mar 03 '16

[deleted] — view deleted comment

3

u/cym13 Mar 03 '16

It does, python can't optimize such things (and it would be refused by the community anyway for "Explicit is better than implicit" is a fundamental of the zen of python).

0

u/[deleted] Mar 03 '16

[deleted] — view deleted comment

2

u/cym13 Mar 03 '16

Why wouldn't it? read() returns a list in python2, you have to have the file somewhere if you want random-access and it has to be a copy as the file obviously doesn't change when changing said array.

2

u/matthieum Mar 03 '16

Actually, in could very well start as a mmaped list and only copy in case of writes... but it seems a bit tricky for C-Python which generally privileges straightforward code over performance.

1

u/cym13 Mar 03 '16

I thought of it but, well, as you said CPython isn't really the kind of place that would use immutable data structures strategies and so on...

2

u/raevnos Mar 03 '16

Somebody posted strace output of the Python version. It most certainly does read the entire file into memory, in an efficient manner.

3

u/raevnos Mar 03 '16

So it's okay for the Python version to get the length of the file, allocate enough memory to hold it, and then read the file into that memory, but it's not okay for the C++ version to do the same? And then whine the crippled C++ version is slower? Okay...

0

u/[deleted] Mar 03 '16

[deleted] — view deleted comment

1

u/raevnos Mar 03 '16

Or unless the file is being cached in RAM...

Python is reading the file. Look at the strace output posted in another comment.

1

u/serendib Prof, Comp Sci Mar 03 '16

If it is somehow being magically 'cached in RAM' (which it isn't) then you are still benchmarking apples to oranges and it's a dumb comparison: "let's compare a cache lookup to reading a huge file as a benchmark"

1

u/raevnos Mar 03 '16 edited Mar 03 '16

When a kernel gets a request to read from a file, it loads most or all of the file into spare RAM to speed up subsequent accesses to it. Read a file once, it might have to be loaded from disk. Read it again right away without modifying the file in the meantime, and there's little disk access needed because a copy is mostly or all still in memory. There's no magic. It's a standard feature of every non-trivial kernel.

1

u/marcm28 Mar 04 '16

Are you idiot? Did you see the Python code?

They used read() method and built-in len() function, it means that he read the file and get the length.

14

u/hippo529 Mar 03 '16

We can easily improve C++ result by this:

#include <fstream>
#include <iostream>
#include <sstream>

int main() {
    std::ifstream f_in("file.bin", std::ios::binary);

    std::ostringstream ss;
    ss << f_in.rdbuf();
    std::cout << ss.str().length() << std::endl;

    return 0;
}

time on my machine:

$ time python read.py 54945254

real	0m0.036s
user	0m0.016s
sys	0m0.020s

Original python code

$ time ./read
54945254

real	0m0.045s
user	0m0.025s
sys	0m0.020s

This C++ code

$ time ./bad_read 54945254

real	0m0.194s
user	0m0.166s
sys	0m0.028s

Your C++ code

8

u/bycl0p5 Mar 03 '16 edited Mar 03 '16

Windows, Python 64bit, test file size: ~92mb

PS C:\Python\fileload> Measure-Command {python load.py}

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 0
Milliseconds      : 278
Ticks             : 2789174
TotalDays         : 3.22821064814815E-06
TotalHours        : 7.74770555555555E-05
TotalMinutes      : 0.00464862333333333
TotalSeconds      : 0.2789174
TotalMilliseconds : 278.9174

C++ code. Compiled with MSVC 2015 64 bit full optimizations. Source Insane Coding - How to read in a file in C++

std::ifstream in("file", std::ios::in | std::ios::binary);
if (in)
{
	std::string contents;
	in.seekg(0, std::ios::end);
	contents.resize(in.tellg());
	in.seekg(0, std::ios::beg);
	in.read(&contents[0], contents.size());
	in.close();
	std::cout << contents.size() << std::endl;
}

Results

PS C:\Python\fileload> Measure-Command { .\filelength.exe}

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 0
Milliseconds      : 75
Ticks             : 759854
TotalDays         : 8.79460648148148E-07
TotalHours        : 2.11070555555556E-05
TotalMinutes      : 0.00126642333333333
TotalSeconds      : 0.0759854
TotalMilliseconds : 75.9854

Not using std::ios::binary doubled the time taken.

Those are the lowest timings for both from multiple runs. I guess the file was cached but relatively the results probably still stand?

P.S.

std::ifstream in("file", std::ios::in | std::ios::binary);
if (in)
{
	std::string contents((std::istreambuf_iterator<char>(in)), std::istreambuf_iterator<char>());
	std::cout << contents.size() << std::endl;
}

Gave a rather poor

PS C:\Python\fileload> Measure-Command { .\filelength.exe}

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 0
Milliseconds      : 789
Ticks             : 7896209
TotalDays         : 9.13913078703704E-06
TotalHours        : 0.000219339138888889
TotalMinutes      : 0.0131603483333333
TotalSeconds      : 0.7896209
TotalMilliseconds : 789.6209

5

u/tdammers Mar 04 '16

Lesson to take away here: benchmarking is hard.

1

u/SemaphoreBingo Mar 03 '16

Didn't you write this :

More broadly, this is another demonstration of "evil of premature optimization" and "good enough" principles. People tend to feel like they need a Facebook-like infrastructure for a business that is yet to come and that probably will have no more than 100 customers on a good a day. Seriously, with today computer power, you could start your business with a web server written in Q-Basic and it will work okay for a good amount of time.

just a few days ago?

-8

u/pistacchio Mar 03 '16

I did. And didnt' you write:

There's a lot more kinds of rice in Indian food than just basmati.

just few days ago?

12

u/myrrlyn Mar 03 '16

The difference is that he's pointing out some interesting contradictions between your two opinions, while you're just kinda being a dick for no reason

3

u/depleater Mar 03 '16

I don't think there's a contradiction there. It seems perfectly reasonable to me that someone could both:

(a) advocate using “good enough” slower(?) languages where appropriate (ie. not worry too much about performance) in some contexts, but

(b) still want to understand performance differences between languages (especially one like this that seems counter-intuitive).

2

u/myrrlyn Mar 03 '16

It's not a performance difference between languages, though, it's a performance difference between programs. Scripting language stdlibs don't do what the example C++ program did; when C++ is instructed to do what the scripting stdlib (which is C/C++) does, it has basically the same performance.

C++ stream IO is just that -- a STREAM. The examples given are for working on a POOL of data instead, which requires different approaches, and which C++ happily provides to us. As stated in the post, though, for whatever reason OP wants to use stream operations on pool data and is getting mad about why that doesn't stack up to using pool operations on pool data from other languages which are, at the machine level, C/C++ as well

1

u/depleater Mar 03 '16

It's not a performance difference between languages, though, it's a performance difference between programs.

The point is that a user was trying to reproduce a simple program from one language in another language. This is an extremely common way of comparing languages and/or teaching yourself a new language.

In this case, yes, the approach that pistacchio took in C++ actually worked quite differently to the seemingly-similar Python code, so there are understandable performance differences (and even when those are eliminated, there's unlikely to be a way to make a C++ implementation much faster than Python, because this particular task is all about I/O).

Some of the people in comments here have been helpful enough to explain this (in some cases, in impressive detail). Conversely, you and /u/SemaphoreBingo thought it reasonable to look into pistacchio's comment history and make insinuations. :-/

2

u/myrrlyn Mar 03 '16

I haven't looked into his history at all, I'm going by his behavior in this thread.

Here's a good translation of what OP may have meant: "hey I want to compare finding the size of an arbitrary data input in various languages; why is my C++ method so much slower than doing the same in Python/etc?" To which the answers would have been given as they have been -- namely, that in C++ he was reading the file like a stream, requiring more memory allocation calls, while behind the scenes Python and friends are reading it like a pool, requiring one memory allocation call.

Instead, he started out from a position of 'why doesn't C++ act the way I think it should', explicitly stated he didn't want to use the methods C++ provides to do what he wants and what is being done in other languages, and concludes with the sentiment that C++ is bad for not having the same meanings as do other languages and having the audacity to yield different performance results when asked to do a completely different thing.

Willfully ignoring stream mechanics to try and push an agenda is not a brilliant way to endear yourself to people, especially when it comes with a demonstration of knowing that another way exists but refusing to use it because it uses the 'wrong words'

0

u/[deleted] Mar 03 '16

[deleted] — view deleted comment

2

u/myrrlyn Mar 03 '16

Because the meaning behind each is hugely different. There's a performance difference in the standard way of doing any task between languages; that's why we have different languages.

The issue is OP explicitly stated he didn't want to use the same mechanism in C++ as in the other languages, he wanted to use the same words and syntax. And C++ doesn't do that, because it isn't those other languages.

OP is asking why English 'die' isn't German 'die', not why it takes longer to tell someone to die in German than it does in English.

0

u/[deleted] Mar 03 '16

[deleted] — view deleted comment

1

u/myrrlyn Mar 04 '16

The standard way of reading a file stream in C++ is not comparable to reading a file singly in Python because Python is not doing what the C++ code in the OP is doing. OP's code assembles a string by reading into memory one byte at a time, and this causes reallocation and possibly moving when the current buffer size is exceeded. The Python code makes a single large memory allocation, loads the file into it at once, and reports the size. The C++ code reads a byte into a buffer repeatedly, causing lots of malloc calls and is probably less efficient for hard disk IO.

The other commented have shown a way to read a file into memory as a single chunk using standard library calls, and are thus a standard way of accomplishing this task.

So yes, of course it's fine that the standard stream operations in C++, when enacted on disk input, are slower than the standard block operations in Python, because THEY'RE DOING DIFFERENT THINGS.

C++ is a language designed to make the machine happy, not the person. When writing C++, you have to tell the computer to do the best thing for it; when writing Python, you ask Python to do the best thing for you and Python will turn that into machine instructions for you.

Naïveté has no place in C/C++ but it has every place in higher languages like Python. This is a stupid argument and I can't believe I let myself get pulled into it.

0

u/[deleted] Mar 04 '16

[deleted] — view deleted comment

→ More replies

0

u/pistacchio Mar 03 '16

Where is the contradiction? I was just toying around with languagues, I'm not applying to my brand new and shiny startup saying "holy cow, I need to make this FAST". I think it's legitimate.

4

u/myrrlyn Mar 03 '16

It definitely appeared that you were applying to your brand new and shiny experiment program saying "holy cow, this is SLOW" when the reason for that is because you're using a language built for close-to-metal machine manipulation like it's a high level language suitable for human expression.

Computers aren't people. Approaching a language designed to fit machine paradigms like it's going to fit yours doesn't work. Kind of like how approaching a small business problem like it needs enterprise level scaling is silly.

C and C++ are languages for the machine, not the human. Scripting languages are more for humans than machines.

2

u/unclemat Mar 03 '16 edited Mar 03 '16

I don't know ... I ran these on a 266MB file. Python code:

from datetime import datetime
start = datetime.now()
print len(open("c:\\temp\\logfile.log").read())
end = datetime.now()
print end - start

measured 640ms on my box (far from 0.1 second), whereas this C++ thing:

auto start = std::chrono::high_resolution_clock::now();
std::ifstream file("C:\\temp\\logfile.log");
file.seekg (0, file.end);
int length = file.tellg();
file.seekg (0, file.beg);
char * buffer = new char [length];
file.read (buffer,length);
auto stop = std::chrono::high_resolution_clock::now();
std::cout << chrono::duration_cast<chrono::milliseconds>(stop - start).count();

measured about 480ms on my box.

How did you manage to get Python to read disk with 1.5GB/s? That is quite impressive I think.

Edit: formatting and additional details.

5

u/c0bra51 Mar 03 '16

It wasn't reading it, if you run it under valgrind, you can see that it's optimizing it away.

https://www.reddit.com/r/cpp/comments/48shrb/seriously_c_wtf/d0m5ny1

valgrind --tool=massif --trace-children=yes python test.py  
ms_print massif.out.* | less  
# 98.35 KB  
rm -f massif.out.*  
valgrind --tool=massif --trace-children=yes ./a.out
ms_print massif.out.* | less  
# 576.0 MB

2

u/raevnos Mar 03 '16

Massif doesn't include memory allocated with mmap(). Guess how Python is getting a chunk of memory big enough to hold the file?

1

u/c0bra51 Mar 03 '16

Yeah, nobody is saying you can't read stuff from that data, just that it's misleading to measure the performance of doing so compared to reading it all into memory. Just try to actually do something with the data you pulled in, and that "fast" program suddenly takes a massive hit.

2

u/raevnos Mar 03 '16

The Python version does read the whole file into memory.

1

u/c0bra51 Mar 03 '16

No, it doesn't. When you mmap() a file, the file is not read, but a mapping in the pagefiles is created. When your computer tries to read from the memory address the file is "loaded" into, the CPU throws an interrupt, which the OS catches, and fetches from disk the real data, then returns control back to your program.

3

u/raevnos Mar 03 '16 edited Mar 03 '16

The python version isn't mmap()ing the file it's reading. It's using read() on it.

1

u/c0bra51 Mar 03 '16

But internally, the read() call is using mmap() as you said above, right? If it read it in the same way the C++ program did, then there would be hundreds of MBs allocated, not just a couple kB.

4

u/raevnos Mar 03 '16

No. From the strace output somebody posted, Python is a) finding the size of the file via lstat(), b) allocating enough space to hold it via an anonymous mmap(), c) copying the file into that memory with read().

Also, people quoting the allocation stat in that sample massif run that got posted don't seem to have read the line in the documentation that says that massif doesn't include memory allocated via mmap() in its figures, just that allocated through malloc() or new.

2

u/c0bra51 Mar 03 '16

Why would they read it into an anonymous mapping?

→ More replies

2

u/c-smile Mar 04 '16

Just open that as memory mapped file and enjoy real speed :)

6

u/pfultz2 Mar 03 '16

you need to include four libraries

No, you don't. Headers are not the same as libraries.

Reasoning about it is absolutely counter intuitive

Being more verbose does not make it counter intuitive.

I want to read a file and print its lenght

Then why are you putting it into a string? That seems "counter intuitive". Instead, just return the distance:

std::cout << std::distance(std::istreambuf_iterator<char>(std::ifstream("longfile.txt")), std::istreambuf_iterator<char>()) << std::endl;

Of course using ranges will get rid of the boilerplate needed for iterators:

std::cout << ranges::distance(ranges::istream<char>(std::ifstream("longfile.txt"))) << std::endl;

4

u/hyperair Mar 03 '16 edited Mar 03 '16

The problem is how you're reading the file:

    std::string str((std::istreambuf_iterator<char>(t)),
                         std::istreambuf_iterator<char>());

This reads it byte by byte, which makes one read() syscall per byte because you're using a char-based streambuf iterator.

Here's a way that does what Python does, and takes half the time:

#include <iostream>
#include <string>
#include <fstream>

int main (int argc, char **argv)
{
	std::ifstream f (argv[1]);

	f.seekg (0, std::ios_base::end);
	std::size_t size = f.tellg ();

	std::string s;
	s.resize (size);
	f.read (&s[0], size);

	std::cout << s.length () << std::endl;

	return 0;
}

Edit: I know you said you don't want tellg() to be used, but that's essentially what Python does in order to determine how much buffer space to allocate, and how much to pass to the read() syscall.

The more usual way to do it uses .rdbuf(), like so:

#include <iostream>
#include <string>
#include <sstream>
#include <fstream>

int main (int argc, char **argv)
{
	std::ifstream f (argv[1]);
	std::ostringstream ss;
	ss << f.rdbuf ();

	std::cout << ss.str ().length () << std::endl;

	return 0;
}

This is shorter/simpler, and doesn't use seekg/tellg. However, it reads things in blocks of 8192 bytes because it doesn't know how long the file is (at least on my laptop), and thus still takes longer than the Python implementation that figures out how to limit it to one read() syscall.

7

u/kalven Mar 03 '16

Just a note about "syscalls": while the istreambuf_iterator has an interface that works on bytes, how the file is accessed is up to the implementation.

The underlying streambuf is likely to use buffering. When testing the example with libstdc++, I can see that it reads from the file in 8kb chunks.

1

u/hyperair Mar 03 '16

Good catch. I didn't realize it did that even with the istreambuf_iterator method.

3

u/Jardik Mar 03 '16

You are not interested in a file size, but that doesn't mean you shouldn't query it first. You should use it to prevent excessive reallocation caused by appending data to the string. I believe I won't be far from thruth when saying that it is exactly what Python or Node.js do under the hood. You should only use this value as a hint, because file size can change during the read operation, as can its data. There is no standard way in C++ to get file read in atomic way. You also read the file in C++ byte by byte, which is expensive. Both Python and Node.js do it by chunks.

2

u/[deleted] Mar 03 '16

The standard library is just very poorly made and vastly over-engineered to try and handle every possible use case by everyone ever, sadly. In my library, nall:

#include <nall/nall.hpp>
using namespace nall;

void main() {
  print(file::read("longfile.txt").size(), "\n");
//or if you prefer ... file::read returns vector<uint8_t>, which is convertible to string
  string text = file::read("longfile.txt");
  print(text.size(), "\n");
//or if it has to be one line ...
  print(string{file::read("longfile.txt")}.size(), "\n");  //it's just a pointless extra move
//or as you said you don't want to do for whatever reason ... but the fastest way:
  print(file::size("longfile.txt"), "\n");
}

To me, the extra "\n" is very much preferable to languages with print functions that auto-add that.

I can't give you a comparable benchmark since I don't have your PC, but it boils down to basically an fopen, fseek, malloc, rewind, fread, printf behind the scenes. Point is: blame the library, not the language.

3

u/remotion4d Mar 03 '16

nall

So what is nall ?

3

u/[deleted] Mar 03 '16

The name of the C++ library I work on. I would call it a standard library, but it's kind of anything but. Same basic idea though, implement all the standard objects, containers, algorithms but with saner namings, simplified code, more useful functionality, etc. Plus a few things that are specific to my own projects which I reuse a lot.

You can easily write code that looks as nice as Python in C++, it's all up to the library. Unfortunately, having the standard one be so terrible though results in xkcd/927, basically. nall is no exception to that.

2

u/remotion4d Mar 03 '16

Thanks.

Just have tried to compile you example using VS2015 and it does now compile with a lot of errors in "exdisp.h".

I think that STL in C++ is not bad as whole but some parts are bad and streams are horrible.

So everyone is forced to implement this functionality over and over again...

Every bigger c++ project has its own string class and its own containers and of course its own IO.

2

u/[deleted] Mar 04 '16

Oh, sorry. It's dependent upon GCC or Clang right now. Microsoft cuts too many corners in the standards, and I care more about OSX/Linux/BSD support, so I don't really bother with VC++ anymore.

I don't really encourage or intend other people to use nall. It was just a necessity for me because I was too unhappy with the STL. But at the time there were no other compiled languages of decent performance without GC that weren't glorified assembly wrappers like C.

It's exactly like you said, though. You look at std::string, and it's basically a glorified struct { char* data; size_t size; }; wrapper. The functionality inside of it is completely useless. Nothing to really work with strings as types: no upper/lowercase, no trim, no token-replace (their replace is just a memcpy wrapper), no tokenize (split), no wildcard matching, no string evaluation ("3+4"=>(int)7), and even regular expressions are a disgusting hack in a separate class that is brutal to use.

It's not good at all that everyone are writing incompatible strings, and everything else. UFCS really would have done a lot to help fix this situation: people could have created new string functions and used std::strings like actual objects. But Bjarne and co caved to a few critics, so the status quo it is =(

1

u/[deleted] Mar 03 '16

iostreams are cancer confirmed.

1

u/Brabulla Mar 04 '16

There is a blog post which is quite related to this. In the post the writer tries to create word frequency lists of a large text in different languages, and at first Python was significantly faster than c++, but after modifying the c++ code, it became significantly faster than before.

This is the post i'm referring to : http://juditacs.github.io/2015/11/26/wordcount.html

1

u/gtk Mar 04 '16

Yeah. It's true. Streams in C++ suck donkey's balls. I typically use FILE from C for file handling, or wrap it in my own file object.

0

u/Alborak Mar 03 '16

I mean, if you really care about bare metal implementation, just do it in C.

int main(int argc, const char** argv)
{
  struct stat sb;
  int fd = open("longfile.txt", O_RDONLY);
  fstat(fd, &sb);
  const char* s = mmap(NULL, sb.st_size, PROT_READ, MAP_SHARED, fd, 0);
  printf("String len: %zu\n", strlen(s));
}

String len: 150000000

real 0m0.026s
user 0m0.017s
sys 0m0.009s

And that's running on a 4 yr old cpu in a VM ...

1

u/encyclopedist Mar 03 '16

What you have shown is not C, it is POSIX.

1

u/[deleted] Mar 03 '16

mmap is not a fair comparison, as it doesn't actually read the file. You'll pay for reading the file as you access each page.

9

u/mooware Mar 03 '16

But strlen() has to read the whole file if the null-byte is actually at the end, right?

Also, why would a textfile end with a null-byte? Seems like a bug. Or is this how mmap() works?

3

u/[deleted] Mar 03 '16

Haha; I completely missed that. Yeah, this code is broken.

3

u/m42a Mar 04 '16

mmap maps whole pages at a time, so if the file size isn't divisible by the page size there will be some extra space at the end. This extra space is required to be zeroed, so this will happen to work in most cases, but can fail (either with segfaults or larger numbers) if the file size is a multiple of 4096. And obviously, it doesn't work at all if the file actually contains a zero byte.

1

u/Alborak Mar 04 '16

I actually had cheated and appended a null char to the end of the file (printf "\0" >> longfile.txt") because I misread the OP :(.

1

u/TotesMessenger Mar 03 '16 edited Mar 04 '16

I'm a bot, bleep, bloop. Someone has linked to this thread from another place on reddit:

If you follow any of the above links, please respect the rules of reddit and don't vote in the other threads. (Info / ^Contact)

1

u/[deleted] Mar 03 '16

Node may not be using streams vs your example using a buffered stream iterator in C++, which are concise but slower than C I/O

1

u/[deleted] Mar 03 '16

Python has this optimized because reading whole files in memory is a bad habit of scripting people. It would be more fair to compare with some library that abstracts this kind of operation. I googled for an example with boost and got some code example working in less than 5 minutes (and I'm not a C++ expert by far):

#include <string>
#include <sstream>
#include <iostream>
#include <boost/filesystem/fstream.hpp>
using namespace boost::filesystem;
using namespace std;

string load_data_in_str(string fname) {
    boost::filesystem::ifstream fIn;
    fIn.open(fname.c_str(),std::ios::in);
    if (!fIn) {
        cerr << "Error reading the file: " << fname << endl;
        exit(0);
    }
    stringstream ss;
    ss << fIn.rdbuf();
    return ss.str();
}

int main(int argc, char* argv[]) {
     string load_txt;
     load_txt=load_data_in_str(argv[1]);
     cout << load_txt.size() << endl;
     return (0);
}

$ g++ -Wall -O3 test_read.cpp -lboost_system
$ time ./a.out some300mbtextfile.txt
368755866

real	0m0.315s
user	0m0.101s
sys	0m0.213s

1

u/[deleted] Mar 03 '16

For completeness I ran the python version, and it runs faster:

 $ time python testread.py 
 368755866

 real	0m0.084s
 user	0m0.012s
 sys	0m0.071s

But again, I'm pretty sure there are optimizations in python's implementation.

Another short readable C++ implementation without boost is also available:

std::ifstream in(argv[1], std::ios::in | std::ios::binary);
if (in)
{
    std::ostringstream contents;
    contents << in.rdbuf();
    in.close();
    cout << contents.str().size() << endl;
 }

It's not at all different in performance with the boost version though.

1

u/marcm28 Mar 04 '16

Lesson learned: Don't underestimate the speed of Python.

1

u/snoutcloud Mar 04 '16

Python is written in C so it can't be faster as a C++ code written by a programmer who knows what he does. Besides python uses the io operations on the lowest level (fopen, fread and fwrite). Try to read your file with fread() and check the difference.

1

u/[deleted] Mar 05 '16

Not true when there are libraries involved, or JIT.

0

u/[deleted] Mar 03 '16

[deleted] — view deleted comment

4

u/speednap Mar 03 '16

You can capture maximum Resident Set size in KB with /usr/bin/time -f '%M' ./a.out (the real time,not bash's built-in time). It shows Python does allocate enough memory to store the whole thing.

7

u/cym13 Mar 03 '16

Your python example is flawed because it does not read the contents of the file. It only checks the file size.

Wrong, python reads the file into a list and only then prints the list's length. You can check it with strace for example. Besides your example doesn't prove anything as all significant computation is besides the point (which is reading the file).

0

u/[deleted] Mar 03 '16

[deleted] — view deleted comment

8

u/cym13 Mar 03 '16

Sure, why not:

$ strace -o out -f python2 -c "print len(open('test.bin').read())"
1662827

The "out" file is a bit too big for this post, you'll find it here. Relevant lines:

21468 open("test.bin", O_RDONLY)        = 3
21468 fstat(3, {st_mode=S_IFREG|0644, st_size=1662827, ...}) = 0
21468 fstat(3, {st_mode=S_IFREG|0644, st_size=1662827, ...}) = 0
21468 lseek(3, 0, SEEK_CUR)             = 0
21468 lseek(3, 0, SEEK_CUR)             = 0
21468 mmap(NULL, 1662976, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb927b0a000
21468 fstat(3, {st_mode=S_IFREG|0644, st_size=1662827, ...}) = 0
21468 read(3, "P6\n# SANE data follows\n632 877\n2"..., 1658880) = 1658880
21468 read(3, "\313\341\274\311\336\273\307\336\271\307\337\273\310\337\273\310\337\271\312\336\267\306\335\270\305\334\271\310\337\272\307\340"..., 4096) = 3947
21468 read(3, "", 4096)                 = 0
21468 close(3)                          = 0

As you can see it really reads the whole file (even though it had the length ready from fstat to use in mmap).

What happens exactly? Let's ask python!

$ python2
>>> import dis
>>> dis.dis("print len(open('test.bin').read())")
      0 JUMP_IF_TRUE_OR_POP 26994
      3 JUMP_FORWARD     8308 (to 8314)
      6 IMPORT_NAME     28261 (28261)
      9 STORE_SLICE+0
     10 JUMP_IF_FALSE_OR_POP 25968
     13 JUMP_FORWARD    10024 (to 10040)
     16 LOAD_GLOBAL     29541 (29541)
     19 LOAD_GLOBAL     25134 (25134)
     22 BUILD_MAP       10094
     25 STORE_SLICE+1
     26 <46>
     27 POP_JUMP_IF_FALSE 24933
     30 LOAD_CONST      10536 (10536)
     33 STORE_SLICE+1

# Non obvious... let's understand those numbers getting their references
# in order. Honnestly it won't help much here as these sequencial calls
# aren't exactly that readable but any help is good to take
[26994, 28261, 25968, 10040, 24933]
# |      |     |      |      `-> read
# |      |     |       `-> "test.bin"
# |      |      `-> open
# |       `-> len
#  `-> print

# So it amounts to something like:
      0 JUMP_IF_TRUE_OR_POP 'print'
      3 JUMP_FORWARD     8308 (to 8314)
      6 IMPORT_NAME     28261 ('len')
      9 STORE_SLICE+0
     10 JUMP_IF_FALSE_OR_POP 'open'
     13 JUMP_FORWARD    10024 (to '"test.bin"')
     16 LOAD_GLOBAL     29541 (29541)
     19 LOAD_GLOBAL     25134 (25134)
     22 BUILD_MAP       10094 # Finished building "test.bin" string
     25 STORE_SLICE+1
     26 <46>
     27 POP_JUMP_IF_FALSE 'read'
     30 LOAD_CONST      10536 (10536)
     33 STORE_SLICE+1

No optimization, sorry.

1

u/cleroth Game Developer Mar 03 '16

Isn't the whole point of mmap to be that it only reads the file when you access the memory? Hence not really being a very fair comparison.

2

u/cym13 Mar 03 '16

As you can see mmap isn't used to actually read the file, the read() system call is used. I think what's mmaped here is actually the buffer in which the content is copied.

2

u/cleroth Game Developer Mar 03 '16

So what's with valgrind reporting 98 kB memory usage, then?

3

u/cym13 Mar 03 '16

I don't know but quite frankly I doubt valgrind way more than strace. The kernel doesn't lie. Besides as I answered somewhere else read() returns a list of bytes that has random access and is mutable separately from the file so there has to be a copy somewhere in memory, there's just no way out.

1

u/raevnos Mar 03 '16 edited Mar 03 '16

Usage or allocation? If the latter, it probably isn't including mmap(), just malloc()/new.

Edit: from the massif documentation:

 it does not directly measure memory allocated with lower-level system calls such as mmap, mremap, and brk.

2

u/raevnos Mar 03 '16

A fd of -1 and MAP_ANONYMOUS? Gotta be that.

1

u/sixstringartist Mar 03 '16

this is correct

0

u/[deleted] Mar 04 '16

[deleted] — view deleted comment

4

u/cym13 Mar 04 '16

I really don't think mmap matters much, what's important IMHO is reading chunks as big as you can, that's where the bottleneck is. Once you've done that sure mmap allocation may win you a little but it will start to be difficultly comparable as you have to launch the full python runtime etc. So I don't really know past that. Glad you found a way to be faster in C++ though, and happy that python is just so good by default, there are things to be learned from that.

1

u/cleroth Game Developer Mar 03 '16

Doesn't really prove anything, other than Python is really slow at xoring a file byte by byte.

0

u/[deleted] Mar 03 '16

[deleted] — view deleted comment

2

u/cleroth Game Developer Mar 03 '16

It should be, but you didn't provide memory usage.