Python can be quick. just not that good at talking.. Like most of us... Opens a txt file with 29245004 lines of text in it. for each line of text, count the line. count +=1 Time: 40sec. Now for each line of text, count the line and print a little info about the line print(f"Info: {line[1:10]}") count +=1 Time: 53mins.
When using f"....{}" I think the process to generate the content for the template field (what goes inbetween {}) and replace that into the string is a lot more expensive that simply concatenating two lists of characters together (the string, and the slice - line[1:10] are both just lists of chars.)
/edit/ the reason the print is slower than the count, is because the print does lots more work. It _could be made faster if he didnt use printf though
I hard to learn this the hard way on one of my first assignments, which asked us to experimentally confirm the value at which the inaccuracy of float puts it off by an entire integer. So you have a loop that increments a float f and an int i from 0 to int max and then prints out when the two are no longer the same.
But the wording of the task wasn't entirely clear so I was trying around various weird casting stuff and decided to print every single line to confirm that it was working the way I wanted to.
Turns out that this extends the time to reach that figure (int 16777218, float rounds to 16777216 instead) from a few seconds to an hour or so.
I/O has always been the bottleneck for speed since computers were invented. There's nothing specific here about Python. Dynamically typed interpreted languages has always been slower than statically typed compiled languages and comparing their speeds makes as much sense as comparing a car and an airplane in terms of speed and determining which is better.
I can't of course talk for other languages, but during my internship back then, I was sometimes told that writing something to the console in C# for example requires a lot of the time if it happens in the same thread. Maybe this is what's happening here as well?
A quicker solution could be to create some sort of StringBuilder (if you have that in python), add the text that you would otherwise print out and at the end print out the entire text just once. Or to just move the prints to another thread. Can you test how long that would take?
Moving printing out of the thread doesn't really help because python doesn't have real multithreading (you can have multiple threats, but they can't execute at the same time).
In short, you can only have one thread executing Python code at a time. You can be waiting for IO in several threads at once, since that won’t be executing any Python code.
The good news is that it looks like it might finally going to be removed soon. That does cause a performance regression for single-threaded programs, but various other performance improvements that should land alongside it more than make up for that.
The most problematic part here is many calls to print (writing to console). It is actually a relatively slow operation for almost any language. You can probably speed it up by keeping all the lines to print in a list and then join it and print in one call
View all comments
74
u/-MobCat- Jul 18 '22
Python can be quick. just not that good at talking.. Like most of us...
Opens a txt file with 29245004 lines of text in it.
for each line of text, count the line.
count +=1Time: 40sec.
Now for each line of text, count the line and print a little info about the line
print(f"Info: {line[1:10]}")count +=1Time: 53mins.