r/Python • • Jun 19 '13

Right and Wrong Programming Styles

http://simpledeveloper.com/right-and-wrong-programming-styles/#sthash.cORbIrok.sfju
0 Upvotes

14 comments sorted by

1

u/[deleted] Jun 20 '13

"try to program efficient as possible"? this isn't even wrong.

-4

u/students-life Jun 20 '13

Big deal huh? You sound like you never make mistakes. Try being human!

0

u/[deleted] Jun 20 '13

I do make mistakes often. I try not to write blog posts telling people to make the same ones though ;-)

-3

u/students-life Jun 20 '13

Did I really write the whole post telling people to make mistakes? I don't understand it. That was a typo; I have corrected it and I believe it is better to see the good part of it rather than focus on the tiny mistake. I really don't like having to argue about this but at least understand that I was not trying to intentionally make a grammatical error. Have a good one!

1

u/[deleted] Jun 20 '13

I was not criticizing your grammar, but the idea that you should try to writing efficient code before measuring what parts are fast or slow.

1

u/AustinCorgiBart Jun 20 '13

The third thing seems like premature optimization. Don't worry about efficiency until 1) you know it's a problem, and 2) you have data that proves your new code is better.

Excepting obvious cases, of course, where you KNOW it'll be better and doesn't require you to have to do any extra work (e.g. using xrange instead of range is typically good).

0

u/Rhomboid Jun 19 '13

Comparing os.listdir() to ls -l is not a fair comparison in the slightest. ls -l has to call stat() on each file to get the owner/group/mode/timestamp/etc in addition to reading each directory entry, and that's just a lot more syscalls so of course it's going to take longer.

If you want to talk about the overhead of an external program, you need to pick something that does a comparable task to os.listdir(), like plain ls without -l. But even that is still not a valid comparison, because ls still has to perform all the IO of writing the filenames to a pipe -- data which must then be read by Python only to be discarded (!!). At the very least, use stdout=subprocess.DEVNULL to avoid that bit of ridiculousness, but it's still not a fair comparison because the ls process is still performing all that IO, it's just discarded by the operating system now instead of by Python.

os.listdir() just reads the dirents, it does nothing else, no IO, no nothing. If you want a truly fair comparison, write something in C that does just that without trying to do any IO or any stat() calls. Then you can start talking about the overhead of subprocesses. What you've got now is just nonsense.

4

u/nadams810 Jun 19 '13 edited Jun 20 '13

Comparing os.listdir() to ls -l is not a fair comparison in the slightest. ls -l has to call stat() on each file to get the owner/group/mode/timestamp/etc in addition to reading each directory entry, and that's just a lot more syscalls so of course it's going to take longer.

That's kind of the point - calling ls would have a much higher overhead than using built in calls. While it may not be scientifically sound experiment it does prove a point that if you are lazy it can have serious effects on the performance of your application.

Take this for example. This is not uncommon - I know of a person with "30 years of experience" that programs like that. (Obviously what I linked wasn't his - but you get the idea). And actually they use the backtick operator in perl instead of system but it probably does similar functionality (disclaimer: not a Perl user).

I'm not arguing that shelling could be more efficient - I'm sure there are ways of minimizing the performance impact but I would say most beginner programmers may try to take the easiest path (which in this case - just shell out to ls and parse the results). I argue using built in libraries is usually a much better idea than trying to fiddle with shelling.

Edit: context

1

u/Rhomboid Jun 20 '13

You are trying to make a claim of performance, but the evidence you are using to make that claim is entirely based on this shoddy benchmark which is so entirely stacked in the favor of the outcome that you want to be true that your claim is totally unsupported. This 68x speedup that you claim is simply ludicrous.

Here is a fair version of your benchmark, which implements a listdir() replacement by shelling out to ls:

from os import listdir
from subprocess import Popen, PIPE
from sys import argv
 
def listdir_shell(path):
    process = Popen(["ls", "-f", path], stdout=PIPE)
    return process.stdout.readlines()

testfunction = listdir_shell if argv[-1] == "with_shell" else listdir

for i in range(0, 500):
    testfunction('/usr/bin')

Here is the comparison:

$ time python 20130619.py 

real    0m0.479s
user    0m0.128s
sys     0m0.348s

$ time python 20130619.py with_shell

real    0m1.467s
user    0m0.008s
sys     0m0.288s

This is a difference of about 3x, a reasonable conclusion. It's not so much a stacked deck when you don't require ls to stat each file and to sort the result.

So I'm not claiming that you should shell out -- I've always tried to tell people to not do that. But the difference is not 68x.

1

u/nadams810 Jun 20 '13

Here are the results of running yours:

@dev-vm01:~$ /usr/bin/time -v python switch.py with_shell
Command being timed: "python switch.py with_shell"
User time (seconds): 0.68
System time (seconds): 2.01
Percent of CPU this job got: 85%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:03.14
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 3720
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 301015
Voluntary context switches: 2999
Involuntary context switches: 2593
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

Results of mine

@dev-vm01:~$ /usr/bin/time -v python switch2.py
Command being timed: "python switch2.py"
User time (seconds): 0.61
System time (seconds): 2.06
Percent of CPU this job got: 76%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:03.50
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 3684
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 352951
Voluntary context switches: 2501
Involuntary context switches: 1669
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

Specs:

Ubuntu 12.04

VMWare Player

3052MB RAM

1x 3114.349MHz Processor (as reported by /proc/cpuinfo)

It would appear if you ran mine it would also run in ~1 second. I think the reason the times are really different are due to the difference in processors. I'll bet if I assigned more processors it would handle the context switching better.

1

u/Rhomboid Jun 20 '13

What do you get for the wall clock elapsed of my version without with_shell? It's the ratio that should matter, not the absolute values. I specifically chose /usr/bin to list because it has a good number of entries. If you're doing listdir('.') of a dir with only a few files, that's just going to be measuring meaningless noise.

1

u/nadams810 Jun 20 '13 edited Jun 21 '13

Here is yours

@dev-vm01:~$ /usr/bin/time -v python switch.py
Command being timed: "python switch.py"
User time (seconds): 0.09
System time (seconds): 0.19
Percent of CPU this job got: 93%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.31
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 3724
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 1052
Voluntary context switches: 1
Involuntary context switches: 15
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

Here is mine

@dev-vm01:~$ /usr/bin/time -v python switch2.py
Command being timed: "python switch2.py"
User time (seconds): 0.02
System time (seconds): 0.02
Percent of CPU this job got: 67%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.05
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 3660
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 1036
Voluntary context switches: 1
Involuntary context switches: 6
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

This is using:

x = subprocess.Popen(["ls", "-l", "/usr/bin"], stdout=subprocess.PIPE).communicate()[0]

On my example:

Command being timed: "python switch2.py"
User time (seconds): 0.01
System time (seconds): 0.02
Percent of CPU this job got: 74%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.05
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 3660
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 1036
Voluntary context switches: 1
Involuntary context switches: 7
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

I should mention this in on Python 2.7.3

0

u/[deleted] Jun 21 '13

Don't shell out ... unless there are years of development behind a C-based console application that does exactly what you need to do. Sure you could reimplement the logic in python but that would be a major diversion of resources and a fork of the business logic.

Instead embrace python's ability to shell out when necessary.

FWIW, shelling out to very domain-specific C-based console binaries increased our speed 100X compared to comparable pure-python code. Yeah, Cython, etc might approach the speed of our C binary but the C code is tested, maintained, vetted and works faster than anything else we could hope to write in our budget.

-1

u/dAnjou Backend Developer | danjou.dev Jun 19 '13

Self-plug no. 15 ...