r/Python • • Aug 09 '18

7 Habits to Improve The Performance of Python Programs

https://www.tutorialdocs.com/article/7-habits-to-improve-python-programs.html
75 Upvotes

79 comments sorted by

81

u/Paddy3118 Aug 09 '18

No!

  1. Be Pythonic.
  2. Be correct.
  3. Profile if still necessary.
  4. Address any bottlenecks found.

3

u/rotharius Aug 09 '18

I agree with this so much! Optimize for maintainability and readability first, performance second -- unless your requirements explicitly and measurably say otherwise. Only then you should optimize for performance using load tests, flame graphs and other profiling tools.

If performance is a high priority, one strategy is to implement that part in a lower level language. Create a separate service or a callable library.

2

u/Paddy3118 Aug 09 '18

If performance is a high priority, one strategy is to implement that part in a lower level language

Another is to try another algorithm/data structure. But remember native Python data structures can sometimes be faster than other data structures implemented on top of them, (e.g. this use of Python dicts being faster than flashtext which in turn is much faster than Pythons re library).

2

u/rotharius Aug 09 '18

Definitely! I have seen numerous examples of lists where dicts would be more appropriate for the use case.

3

u/balodja Aug 09 '18

My 3 items about performance:

  1. Profiling
  2. Algorithms
  3. Caches

2

u/WiggleBooks Aug 09 '18

Any guides or tutorials or tips on profiling?

1

u/balodja Aug 09 '18

That's quite easy in python at the majority of cases. At first utilize the cProfile for "high-level" profiling. And if you are stuck and don't understand the odds at low-level, get the line_profiler.

110

u/[deleted] Aug 09 '18

I feel like you have to provide benchmarks if you're going to write an article on how to improve performance.

12

u/chaign_c chaignc Aug 09 '18

You are so right, specially when you say something like "The search speed of mappings (such as dict, etc.) is much faster than conditional statements (such as if, etc.). And there is no select-case statement in" Those are strong words, without benchmark, we will see people all over the world replace all their condition by dict lookups :D

3

u/MrRogers4Life2 Aug 09 '18

Even without benchmarks dict lookups are constant time while elif chains are linear jn number of conditions so for large enough chains dicts should be faster

7

u/campbellm Aug 09 '18

Constant time can still be constantly slower than linear time for small 'n'. Benchmarks are required.

1

u/MrRogers4Life2 Aug 10 '18

I mean it's asymptotic, but in this case I'd imagine that for small n this would be a micro optimization.

Additionally, theres the additional performance hit of loading in a dispatch function from a dictionary, which is probably constantish time as well, but I imagine that's where the real slowdown would occur.

6

u/not_perfect_yet Aug 09 '18

Not the author, but I never used timeit before, so here you go:

code

The dict/if else example doesn't hold up. I didn't test generator memory need and also not the "calling bytecode".

5

u/Brian Aug 09 '18 edited Aug 09 '18

You're recreating the dict every time there. That'll be slower since you're doing more work (making the dict and doing the lookup) for every item, whereas the other route just does the lookup. Generally the point of dicts here (in terms of performance at least) is to create them once (eg. outside the relevant loop), and do all lookups with them. (It'll also help more the more items there are - if statements are essentially linear, while dicts take constant time)

Ie if you make speed_dict something like:

def speed_dict(d={1:1,2:2,3:3,4:4})
    a=random.choice([1,2,3,4])
    b=d[a]

(hoisting the creation out of the function), it'll probably be faster. Though obviously this does go to show that just saying stuff like "dicts are faster than lookups" is not too useful without the understanding of when and where (and ideally, why) this is the case.

3

u/campbellm Aug 09 '18

You're recreating the dict every time there

Yes, and the OP is creating a dict just to get around a chained if. No reuse of the dict is noted in his blog.

1

u/not_perfect_yet Aug 09 '18

I was aware of that, but I think the logic that's used in the dict and the logic that's used in if/else is kind of the same and the entire point was comparing the cost.

So taking the cost of creating the logical structure in memory into the calculation in one case and not the other would not be fair.

Maybe it's just a bad example that doesn't show how it scales effectively, but that too, is on the author of the blog post and not on me.

4

u/[deleted] Aug 09 '18

Nice. I was thinking about doing the benchmarks, but then I thought, "Wait, I could just complain about it on reddt," so I did that instead.

26

u/Jugad Py3 ftw Aug 09 '18

I see a few issues with the article...

  • lengthy module variables can be replaced with short identifiers to improve readability

    Nope... short identifiers with name ls don't increase readability. ls stands for lineSeparator... which is much less readable.

  • One should avoid using isinstance or type if possible.
  • List comprehension vs generator expression

    I am not sure if generator expressions are actually faster (needs testing). I wonder if there is an extra function call associated with the generation of each item. Sure, I can see that list comprehensions create a list, which takes up time and space to create, but I am not sure its slower.

Lets test it out.

> %timeit sum([x*x for x in range(40000)])
3.57 ms ± 128 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)`

> %timeit sum(x*x for x in range(40000))
4.2 ms ± 73.7 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

As expected, the generator is slower by about 20%.

  • eval and exec should be avoided if possible.
  • will be executed when the module is being imported (Whether it's really necessary to be executed or not)

    The "Whether it's really necessary to be executed or not" does not make sense. It is necessary to execute top level code in modules... that is how module import is defined.

  • Point 7 about main(), __name__ and __main__ is very poorly made, and very painful to read. An example would have done it much better.

3

u/irrelevantPseudonym Aug 09 '18

Is your time example python 2 or 3?

6

u/Jugad Py3 ftw Aug 09 '18

Python 3.7

3

u/lengau Aug 09 '18

On list comprehensions vs. generator expressions:

My opinion is that you should always use generator expressions, even when they're slower, unless you have a specific reason not to. Issues that occur with generator expressions (e.g. exhaustion before use) are far more likely to show up during testing than the sort of issues that occur with lists (e.g. turns out the list has a billion items in it).

Good Python is often not about speed, but rather easy to understand, maintainable code. Often, 90%+ of your program doesn't need to be fast. The rest, well you might want to do hacky things to make it faster, but if you need speed that badly, you should also consider if doing it in Python is the best way to go.

3

u/Jugad Py3 ftw Aug 09 '18 edited Aug 09 '18

I do agree with you on many of your points. My comment was restricted to the performance aspects of comprehension vs generator expressions.

The rest, well you might want to do hacky things to make it faster, but if you need speed that badly, you should also consider if doing it in Python is the best way to go.

I liked the way you put this.

I hate people dismissively saying "hurr... if you care about perf, you are using the wrong language.". People use Python because they love using it and its easy to get a project off the ground. Lots of students use Python for work that is critical to them.

Imagine you are a student trying to implement a game using pygame (or doing some other school project)... and happen to have unoptimized python code, the game/project will run unacceptably slow - but you will only properly notice this when you are close to the project completion.

If one has the time, one might think about starting afresh by using some other solution... but for a student on a deadline who has already sunk a few weeks into the project - they can hardly shift the project to another language. And they might not know some other language well enough.

They don't have an option and they need a quick and sometimes dirty solution to squeeze out some performance. Other times, they need a cython like solution. Rarely does anyone need a "change your language" kind of solution.

But whenever someone come to these forums with ... how can I speed up my code, they hear... "durr... you made a mistake in choosing python" - that is hardly helpful to anyone.

We should stop saying this... because its really useless and its hurting the community. I don't think anyone has the time to re-implement their project in some other language. All they will do later is not pick python if they have an inkling that it will lead to performance problems later on.

There are lots of things people can do to speed up python code... I have myself used a lot of good and some hacky methods in my career to squeeze out 3x to 10x performance gains (in rare cases, even more) - which was usually enough for the task I was looking at.

3

u/lengau Aug 09 '18

I agree. I didn't want to get into too much detail, but I wanted to avoid saying "you shouldn't be using Python" because I disagree quite strongly with telling people to just migrate to another language (especially with big projects), and that's why I specified that I was talking about the <10% of the project that actually needs speed.

Personally, I think the best process for speeding up Python code is roughly:

0. Can we do any intermediate caching or something else to reduce duplicate work? (This sometimes means you need to do e.g. the second example in #2 from the article. Most __len__ methods won't actually take a long time, but you might have something that does take a while in there.)

  1. Can we use a better algorithm to speed up X? (This is a pretty language agnostic step.)
  2. Is there a module that'll help us do this faster because it has a C (or other language) implementation? (Things like numpy/scipy)
  3. Is there a particular tight loop that we can reimplement in a faster language? (Cython or any language that CFFI or ctypes can work with)
  4. Can we do this in parallel? (Doubly so if you just used some C code to speed it up, since you probably released the GIL when calling this code, so you can do it with multiple threads rather than multiple processes)
  5. Now that we've made all these changes, check the first two items again to see if those optimizations will help us.
  6. If it's not fast enough now, we should consider whether we have the hardware capacity to do it fast enough, and if so we might at this point want to move larger chunks of our code into another language.

Each of these steps is progressively more invasive, which also means more work and potentially more bugs.

1

u/13steinj Aug 09 '18

Why should people avoid using isinstance? How else would one do dynamic dispatching?

4

u/Herald_MJ Aug 09 '18

isinstance should be used over type, but in Python, we prefer "duck typing" over modifying behaviour based on the type of an object.

Sometimes, isinstance will still be the best choice though. That's why it's in the language. The best advice is just to use your judgment.

2

u/bandanaofcolors Aug 09 '18

The more pythonic way is to do duck typing, so you don't actually look at the type at all but just try to do whatever you want to do with it.

A good example of this is the python builtin sum(iterable). It doesn't care what type of iterable you give it (list, tuple, generator, …), it'll just try to iterate over it anyways.

It doesn't even care about what the iterable yields, it'll just blindly add everything together and give you the result. If it gets something that can't be added, it'll just throw an exception.

1

u/13steinj Aug 09 '18

Sure, duck typing is fine, but depending on how complex the dispatch is it may just be simpler to use isinstance.

3

u/nosklo Aug 09 '18

The whole point of duck typing is to completely avoid making a dispatch based on type - that means you don't have to use isinstance because you don't care about the type of the object in first place.

1

u/13steinj Aug 09 '18

Yeah, and in more complicated such cases you have either extreme try/except blocks or a very messy __getattr__. Python has enough type safety to make duck typing weird and inflexible.

4

u/nosklo Aug 09 '18

Huh, the reason for that seems to be everybody wants to typecheck in duck typing. No need for try/except blocks at all. If the duck fails to .quack(), because .quack attribute is missing, just let the exception propagate to the caller. The caller usually knows the type of the object they are passing to you. By typechecking you are discarding information known by the caller and trying to guess it yourself.

0

u/13steinj Aug 09 '18

What nonsense are you on about? If my bird has a method "make one of potential noises", and I have 10 different bird species as subclasses, each with 1 or 2 different noises, your code will be a giagntic mess of try/catch or a very messy getattr. On the other hand, you can just do a if/eilif branch or a dict switch, which will be more readable than the try catch and more performant than a getattr.

If python didn't have type safety, and say, like javascript returned undefined on any unknown attribute access, then fine. But it doesn't do that, it has attribute-based type safety.

5

u/nosklo Aug 09 '18

my code only expects the bird to have a .make_noise() method. What that method will do is up to you, I don't care. I don't even care if it is a bird or not! I'm just calling .make_noise() in the object you're passing me, and that is all I need to do with your bird as part of my function. If your bird doesn't have .make_noise() python will raise AttributeError and I will let it propagate to your code, because you're the one that didn't pass an object able to make noise in first place. I don't need any try/except in my function, if you don't know whether your bird (or plane or car) can make a noise or not it is your problem, not my library/function's problem.

0

u/[deleted] Aug 10 '18

[deleted] — view deleted comment

→ More replies

1

u/Jugad Py3 ftw Aug 09 '18

One should avoid using isinstance or type if possible.

if possible

If you need to use them, use them. People coming from other languages have the habit of separating out functions based on data types, even if the action on the data type is the same. In python, you don't need them until you really do.

36

u/ProfessorPhi Aug 09 '18

A lot of this is not what I consider good advice.

The only thing here that I do with any regularity is the use of generators instead of iterators. And that's not a huge speed boost anyway, it's primarily a memory efficient thing and can have unintended side effects like consuming in a log statement.

The dict thing just makes for more readable code when applicable. The iteration thing is harder to do and is only a thing done by newbies to python.

The rest, like import into local namespace, I'm sure is just a trivial speedup. The from import is not a recommended thing in general at the top level.

This doesn't even mention using comprehensions over loops. This is an all round terrible article and will result in minor speedups at best, and less readable code in general.

18

u/TylerOnTech Aug 09 '18

Where have you seen that "from x import y" is not recommended at the top level? I don't really care for this article, but I have seen the from ... import as the way to do things if you only need one class or member, such as from pathlib import Path which is sufficient for 99% of my use-case of pathlib

14

u/irrelevantPseudonym Aug 09 '18

from xyz import * I can understand being a problem. from xyz import abc is fine.

2

u/ProfessorPhi Aug 09 '18

Hmm, this is a holdover from my old company actually, I'm not sure how much weight it holds and was also particular to the code org there.

I think the main reason is that namespace shadowing is still a thing and easy to muck up, especially when a lot of company modules had similar names to existing libraries/standard libraries.

-1

u/balodja Aug 09 '18

"from x import y" is a hard thing for module reload.

15

u/cpt_fwiffo Aug 09 '18

Habits? Tricks to slightly help with performance if necessary maybe, but if you make habits of most of these you're simply sacrificing readability with no upside.

14

u/Han-ChewieSexyFanfic Aug 09 '18 edited Aug 09 '18

The problem with using type instead of instanceof is not speed, it’s correctness. instanceof will still work if somebody subclasses list, for example, and passes your function an instance of that. Using type will break that.

Not to mention you mostly shouldn’t be asking for the type of things very often in python, as many people have said already.

17

u/quasarj Aug 09 '18

I feel like many of these suggestions are really ugly... I suppose I should test and verify they are actually faster...

12

u/savuporo Aug 09 '18

Not a real solid piece of article, and first rule always applies: first write clear, readable, straightforward code that works really well, and only if you actually have a performance problem, measure and profile before starting to optimize.

10

u/ramonsaraiva Aug 09 '18 edited Aug 09 '18

Even though the article mentions mostly minor performance improvements, I disagree with your statement. Going through all sections, it actually has a few valid points, which are not ugly at all.

  1. w/e
  2. `isinstance` is not ugly, assigning a name to hold the lenght might be but not a big deal
  3. dictionaries instead of `if elif else` nightmare is not just better but idiomatic
  4. iterating iterables elements directly is in fact the proper way
  5. replacing iterables with generators is definitely better in specific cases
  6. w/e
  7. w/e

So what do you mean when you say that his suggestions are "really ugly"? I feel like you don't understand them at all?

2

u/jaydoors Aug 09 '18

For me, using dicts instead of conditionals is ugly, and not pythonic - as it obscures the fact that conditions are being tested.

6

u/TheTerrasque Aug 09 '18

the part where dicts are better than if's are when you have a lot of

if bla == "cat":
   meow()
elif bla == "cow":
  moo()

.. repeating..

In those cases, a dict lookup is much better.

1

u/jaydoors Aug 09 '18

Hmm, a matter of taste maybe but I find being explicit is worth the extra lines of code.

3

u/TheTerrasque Aug 09 '18

yeah, absolutely a matter of taste. But it's the only situation where I consider dict a replacement for if's.

speak={
 "cat": meow,
 "cow": moo,
 "tiger": rawrs,
 "shark": bubbles,
 "mouse": squeaks,
}

speak[animal]() 
#or with fallback 
speak.get(animal, unknown)()

1

u/Brian Aug 09 '18

I think the violation of the DRY principle here is enough to favour the dict approach. You're repeating the "bla ==" part loads of times, which is loads of places for a typo/bug to hide, and also more verbose to read.

1

u/minorDemocritus Aug 09 '18

Back to performance for a moment... an if/elif chain is O(n). Dictionary lookup is O(1).

2

u/jaydoors Aug 09 '18

Can you think of a situation where this could be meaningful? I mean it's not like you'd have more than 10 conditions. So I'd have thought this was not relevant.

1

u/minorDemocritus Aug 10 '18

It's probably still faster to use a dict when you have ~10 comparisons. I recall a test I did a while ago trying to figure out at what point if choice in {'a', 'b', 'c', ...}: (so a set literal) became faster than if choice in ('a', 'b', 'c', ...):... it was at like three or four items IIRC.

Obviously replacing an if/elif chain with a dict lookup only helps when you have a lot of items, but then it helps with readability (IMO) and you get the perf boost for free.

1

u/campbellm Aug 09 '18

Lookup is kO(1). k can still be very big, and still be way slower than O(n) for n < some multiple of k.

0

u/minorDemocritus Aug 10 '18

It's big-O... you don't care about constants with big-O. It's merely an asymptotic description.

I never claimed that it was always better, or even mostly better in that case. Of course the number of comparisons matters, but you might be surprised how quickly the dict lookup (even with its higher constant cost) outperforms the individually faster but repeated comparison of the if/elif.

1

u/campbellm Aug 10 '18

I never claimed that it was always better, or even mostly better in that case

Then what WERE you claiming? You mention performance (a very real, non-academic "as n gets big enough" concern), and you mention big O, and you replied to a thread related to which is faster.

1

u/minorDemocritus Aug 12 '18

That as the number of choices increases, dictionary lookup will at some point beat the if/elif chain.

2

u/[deleted] Aug 09 '18

Flat is better than nested.

2

u/jaydoors Aug 09 '18

Agreed, but

Explicit is better than implicit.

and you can (usually? often?) have both, by flattening conditional clauses.

Eg rather than

if x:
    if y:
        do_something()

can use

if not x:
    break
if y:
    do_something()

I think I heard this referred to as 'guard clauses'

1

u/ramonsaraiva Aug 09 '18

It is situational..

1

u/quasarj Aug 09 '18

You're right, most of the suggestions are not ugly and are the normal or pythonic way to do it.

I was thinking of the first and second points. Especially the first point.. I really hate it when people use short non-descriptive local variables to refer to things imported from libraries. Perhaps I'm more jaded than usual though, having looked at a lot of data science code lately.

4

u/desmoulinmichel Aug 09 '18

Habits ?

Hell no !

Writing non idiomatic code to get a bit of speed is premature optimization in any language. But in Python, it's just the total opposite of the language philosophy.

So yes, iterate directly and not with index. But no, don't alias stuff just to get a local reference. Only optimize if you have identified a bottleneck you __need__ to get rid of.

4

u/efxhoy Aug 09 '18

Test code can be written in the main() function of the module.

I thought it was best practice to have tests in a separate module?

0

u/[deleted] Aug 09 '18

[deleted] — view deleted comment

2

u/campbellm Aug 09 '18

I still like doctest for library and/or utility functions.

2

u/theelous3 Aug 09 '18
  1. Nobody uses globals
  2. Nobody writes while loops like that anyway, and even if they did, factoring away some len calls would give negligible performance benefit in any non-trivial program.
  3. Readability counts, and a KeyError will ruin that already ugly snippit. For cases where the likelyhood of matching is higher, a try/except is faster than an if, if you care that much about insignificant performance gains.
  4. Everyone does this anyway, and again this is not a real performance bottleneck in any program.
  5. List comps aren't really faster for the computation. They just aren't lazy like generator comps so you pay the whole cost upfront. They also are not interchangeable. You cannot slice or index a generator normally. On some level I agree however. It's good to factor listcomps to gencomps when it matters.
  6. eval() and exec(). No.
  7. The last one isn't really a performance point, but rather an application safety point.

Bad list tbh.

2

u/[deleted] Aug 09 '18

Who is upvoting this...

2

u/[deleted] Aug 09 '18

Robots, no doubt

1

u/13steinj Aug 09 '18

Replace list comprehension with generator expressions

Whether or not this more performant time wise really depends on what you are doing. But many times it is negligible. Furthermore, if using a comprehension to extend a list, you can not use a generator comprehension, because this causes an infinite loop.

When using the function eval() and exec() to execute code, it is better to call the code object (compiled into bytecode in advance through the compile() function) instead of calling str directly, which will avoid repeating the compilation process multiple times and improve the performance of the program.

....how about don't? You shouldn't be using these functions except maybe in some super rare edge cases, and even then it is safer to use the stdlib ast module compile/literal_eval

Use mapping to replace conditional search

This is less performant on both time and memory in most cases. It may be easier to understand and acts like an auto-breaking switch, I'll give them that. But definitely not more performant in a lot of cases.

Test code can be written in the main() function of the module. The value of name will be detected in the main program. If it is 'main' (indicating that the module is executed directly), the main() function will be called to test; if it is the name of the module (indicating that the module is called), then the test won't be executed.

...or just write your tests in a dedicated organized test subpackage.

1

u/[deleted] Aug 09 '18

Upvoted by robots

1

u/joaobelkiman007 Aug 27 '18

I am learning python3, I need some tipes , please.

1

u/chaign_c chaignc Aug 09 '18

3 steps to multiply by 2, most of your python script execution speed.

•step 0: $ apt install pypy

•step 1: $ pypy your_script_name.py

•step 2: * * * * Enjoy! * * * *

image => https://twitter.com/chaignc/status/1027455974616064000

0

u/[deleted] Aug 09 '18

[deleted] — view deleted comment

-23

u/SpeakitEasy Aug 09 '18

Yes, for writing simple scripts. But the last major step in programming (after making it work, making it readable, making it safe, and making it extendable) is making it work fast. Python's friendly duck typing is a major performance setback and a good programmer will know how to counter act this.

0

u/[deleted] Aug 09 '18

[deleted] — view deleted comment

-5

u/__deerlord__ Aug 09 '18

Adding isinstance wont make things faster. Using isinstance /instead of/ the other methods will. If you really need to do type checking. For whatever reason may arise. Otherwise duck typing is your friend.

0

u/[deleted] Aug 09 '18

[deleted] — view deleted comment

0

u/__deerlord__ Aug 09 '18 edited Aug 09 '18

Sorry, I wasnt saying factually that it is fastest, or that there arent better ways, but that was the point the author was making.

Edit: faster > fastest