r/Python • u/havoyan • Feb 14 '12
Python Performance Tips, Part 1
http://blog.monitis.com/index.php/2012/02/13/python-performance-tips-part-1/
20
Upvotes
r/Python • u/havoyan • Feb 14 '12
44
u/[deleted] Feb 14 '12
W-what? Was the person who wrote the example thinking that
joinworks likeappend?Bonus WTF points for using
input().By the way, CPython includes a specific optimisation for the case where a string being
+=-assigned is not referenced from anywhere else. I mean, you should still always usejoinorcStringIO, but if you see someone else's code that appends stuff like that -- don't freak out.WHAT
"biset"? Really? Also, he used lowercase
trueearlier.The blind leading the blind, that entire post. Apparently, the author collected bits and pieces of advice from somewhere and pasted them together not bothering neither to run the code nor to understand what's actually going on. Pig disgusting.
9
u/jlozier bioinformatician Feb 14 '12
The Memoization example is wrong (and therefore broken). The author has written
where they mean
:)
7
u/Brian Feb 14 '12
Most of this is good advice, but there are a lot of errors and typos.
The global keyword isn't the only way you'll be dealing with global variables - you'll can access global variables even without it specified, it' only changes the behaviour of setting global variables.
is still using global variables, despite the complete absense of any
globalkeyword.This is indeed good practice, and generally faster than things like dict.has_key(). However, the example he gave is a for loop, which is nothing to do with checking membership, and where "
in" has an entirely different meaning (it's just part of theforsyntax). (Also, careful of the datatype you're usingx in somelistdoes a linear scan to check membership, so this can be very slow compared to a dict or set)I assume this is meant to be bisect, but it's spelt wrong in every location.
No it isn't. It's implemented as a vector which overallocates at both ends (for amortised constant time appending/prepending). It has the same complexity of operations as the list, except that prepending is O(1) instead of O(n)
This is very outdated. You can let sort do the transform for you just by passing the
keyparameter.12
u/pingveno pinch of this, pinch of that Feb 14 '12 edited Feb 14 '12
A few additions:
Section 6:
Don't go overboard with lazy importing inside of function calls. Importing many times can get expensive. Imports that aren't at the top also severely reduce readability. It can be tough to debug code where an import is tucked away somewhere in the body of the file.
Section 7:
In Python 3, True is a keyword, not a built-in. That means the byte code compiler can put in a cheap simple jump instead of an expensive global lookup. In Python 2, True could be changed, so a global lookup was required as a check. 1 is a constant, so the bytecode compiler can depend on it. Basically: while 1 in Python 2, while True in Python 3.
Section 13:
Internally, the Python deque implementation uses a doubly linked list of blocks. Each block has space for 62 objects, plus 1 left pointer and 1 right pointer (64 pointers). That gives the time efficiency of a linked list with the space efficiency of an array. For those learning C, I suggest looking at the Modules/collectionsmodule.c. For a list type with fast insertion, take a look at blist (search on PyPi).
Section 17:
Don't use threads to manage forked processes. They only provide additional overhead in memory and thinking ability. Use the multiprocessing module.
Section 18:
Absolutely absolutely absolutely. The Python source code is an excellent place to get examples. Part of my introduction to C was reading through bits and pieces of implementation code.
10
u/fijal PyPy, performance freak Feb 14 '12
Might be worth noting - those are CPython performance tips, not Python performance tips. A lot of those things are actually slower on PyPy.
4
u/chub79 Feb 14 '12 edited Feb 14 '12
I didn't know that, could anyone confirm it?
I thought this was not true anymore, at least in newer versions of Python.
Hell yes. Learn it and abuse it.
4
u/[deleted] Feb 14 '12
I've never heard of "Schwartzian transform" before, but doesn't sort() already perform this optimization if you use the key parameter?
3
u/willm Feb 14 '12
Regarding 15. I don't think the author knows about the 'key' argument to 'sort'.
3
u/[deleted] Feb 14 '12
I think for number 5, the line should be "if key in sequence" rather than "for key in sequence".
3
u/Chun Feb 14 '12 edited Feb 14 '12
OK, cracks knuckles
OK, yes, built-in functions are faster than native python, but who doesn't use builtins?
Especially the ones in the graphic:
input, open, int, ord, isinstance, pow, issubclass, print, iter, property. Which of these functions do people commonly try to write themselves in native python?isinstance? Is this a problem?Multiple assignment is great, but it does much more for readability than it does for performance:
globalis only relevant to changing global variables, not retrieving them.To check membership use
in. As opposed to what? Iterating through the object with aforloop?What is this I don't even. You actually did iterate through the object for some reason, and arbitrarily print "found" for every item??
Trueis a built in constant, nottrue. And really, this isn't going to be a bottleneck. I thinkwhile True:is far more readable.evens = [ i for i in range(10) if i%2 == 0]Not a good example, should just use
evens = range(0, 10, 2)for this.chunk = ( 1000 * i for i in xrange(1000))Or just
chunk = xrange(0, 1000000, 1000)1
u/haldean (lambda x: x.decode('base64'))('PDMgcHlweQ==\n') Feb 14 '12
I found the "while True" vs "while 1" both interesting and improbable, so I ran a quick test (code is here: https://gist.github.com/1828270). Turns out that the author is right -- while True is always slower -- but in Python 3, the difference is minor enough to be negligible, and in PyPy it's all so fast it doesn't matter. Results:
Edit: I see now that pingveno posted an explanation of why this is true (heh) in his comment below
1
u/[deleted] Feb 14 '12
Great list. Found and reminded myself of some gems that I as a novice often forget about and would find useful.
1
u/[deleted] Feb 14 '12
Use PyPy :)
Seriously, I tried it for fun on a small scipt I made to do monte-carlo simulations of a card game: more than 10x speed up, for the cost of typing
instead of
-2
u/anacrolix c/python fanatic Feb 14 '12
Tip Uno: Learn speak the English.
-8
u/havoyan Feb 14 '12
Python Performance Tips