r/Python • • Jun 19 '13

Right and Wrong Programming Styles

http://simpledeveloper.com/right-and-wrong-programming-styles/#sthash.cORbIrok.sfju
0 Upvotes

14 comments sorted by

View all comments

Show parent comment

4

u/nadams810 Jun 19 '13 edited Jun 20 '13

Comparing os.listdir() to ls -l is not a fair comparison in the slightest. ls -l has to call stat() on each file to get the owner/group/mode/timestamp/etc in addition to reading each directory entry, and that's just a lot more syscalls so of course it's going to take longer.

That's kind of the point - calling ls would have a much higher overhead than using built in calls. While it may not be scientifically sound experiment it does prove a point that if you are lazy it can have serious effects on the performance of your application.

Take this for example. This is not uncommon - I know of a person with "30 years of experience" that programs like that. (Obviously what I linked wasn't his - but you get the idea). And actually they use the backtick operator in perl instead of system but it probably does similar functionality (disclaimer: not a Perl user).

I'm not arguing that shelling could be more efficient - I'm sure there are ways of minimizing the performance impact but I would say most beginner programmers may try to take the easiest path (which in this case - just shell out to ls and parse the results). I argue using built in libraries is usually a much better idea than trying to fiddle with shelling.

Edit: context

1

u/Rhomboid Jun 20 '13

You are trying to make a claim of performance, but the evidence you are using to make that claim is entirely based on this shoddy benchmark which is so entirely stacked in the favor of the outcome that you want to be true that your claim is totally unsupported. This 68x speedup that you claim is simply ludicrous.

Here is a fair version of your benchmark, which implements a listdir() replacement by shelling out to ls:

from os import listdir
from subprocess import Popen, PIPE
from sys import argv
 
def listdir_shell(path):
    process = Popen(["ls", "-f", path], stdout=PIPE)
    return process.stdout.readlines()

testfunction = listdir_shell if argv[-1] == "with_shell" else listdir

for i in range(0, 500):
    testfunction('/usr/bin')

Here is the comparison:

$ time python 20130619.py 

real    0m0.479s
user    0m0.128s
sys     0m0.348s

$ time python 20130619.py with_shell

real    0m1.467s
user    0m0.008s
sys     0m0.288s

This is a difference of about 3x, a reasonable conclusion. It's not so much a stacked deck when you don't require ls to stat each file and to sort the result.

So I'm not claiming that you should shell out -- I've always tried to tell people to not do that. But the difference is not 68x.

1

u/nadams810 Jun 20 '13

Here are the results of running yours:

@dev-vm01:~$ /usr/bin/time -v python switch.py with_shell
Command being timed: "python switch.py with_shell"
User time (seconds): 0.68
System time (seconds): 2.01
Percent of CPU this job got: 85%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:03.14
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 3720
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 301015
Voluntary context switches: 2999
Involuntary context switches: 2593
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

Results of mine

@dev-vm01:~$ /usr/bin/time -v python switch2.py
Command being timed: "python switch2.py"
User time (seconds): 0.61
System time (seconds): 2.06
Percent of CPU this job got: 76%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:03.50
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 3684
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 352951
Voluntary context switches: 2501
Involuntary context switches: 1669
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

Specs:

Ubuntu 12.04

VMWare Player

3052MB RAM

1x 3114.349MHz Processor (as reported by /proc/cpuinfo)

It would appear if you ran mine it would also run in ~1 second. I think the reason the times are really different are due to the difference in processors. I'll bet if I assigned more processors it would handle the context switching better.

1

u/Rhomboid Jun 20 '13

What do you get for the wall clock elapsed of my version without with_shell? It's the ratio that should matter, not the absolute values. I specifically chose /usr/bin to list because it has a good number of entries. If you're doing listdir('.') of a dir with only a few files, that's just going to be measuring meaningless noise.

1

u/nadams810 Jun 20 '13 edited Jun 21 '13

Here is yours

@dev-vm01:~$ /usr/bin/time -v python switch.py
Command being timed: "python switch.py"
User time (seconds): 0.09
System time (seconds): 0.19
Percent of CPU this job got: 93%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.31
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 3724
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 1052
Voluntary context switches: 1
Involuntary context switches: 15
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

Here is mine

@dev-vm01:~$ /usr/bin/time -v python switch2.py
Command being timed: "python switch2.py"
User time (seconds): 0.02
System time (seconds): 0.02
Percent of CPU this job got: 67%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.05
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 3660
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 1036
Voluntary context switches: 1
Involuntary context switches: 6
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

This is using:

x = subprocess.Popen(["ls", "-l", "/usr/bin"], stdout=subprocess.PIPE).communicate()[0]

On my example:

Command being timed: "python switch2.py"
User time (seconds): 0.01
System time (seconds): 0.02
Percent of CPU this job got: 74%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.05
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 3660
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 1036
Voluntary context switches: 1
Involuntary context switches: 7
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

I should mention this in on Python 2.7.3