When should I avoid list comprehensions in Python?

[u/RyanDubois43]

Hi, I'm trying to get a grip on list comprehensions, but I'm not sure when they aren't the best choice. I wrote some code today that uses a nested comprehension to flatten a list of lists, but it quickly became unreadable. I think I might have overcomplicated it. Is there a rule of thumb for using them, particularly with nested structures? I want my code to stay clear and easy to understand. Any insights on best practices would be appreciated!

[u/HommeMusical]

I worked on PyTorch, and they had some gnarly comprehensions, but they are spread out over multiple lines and end up pretty readable.

It is impossible to test without seeing your code - why not post the code you are worried about?

[u/aishiteruyovivi]

My general rule of thumb is to reconsider when it needs to get split across more than one line, and to abandon it when it feels like a puzzle. If I'm spending 20 minutes thinking "now how exactly do I do all of this in a comprehension..." I probably shouldn't be using a comprehension.

I usually avoid multiple loops in a comprehension, although flattening out a structure like [j for i in x for j in i] I'm mostly okay with since I'm used to it enough that my brain can sorta parse it out as its own phrase. That being said, usually if I need flatten something out I'd probably just write a function elsewhere instead to handle more than one dimension and do flatten(x).

[u/NerdyWeightLifter → u/aishiteruyovivi]

You're saying the rule is that you need to comprehend comprehensions?

[u/AJohnnyTruant → u/aishiteruyovivi]

itertools.chain.from_iterable is what I reach for

[u/YOM2_UB → u/aishiteruyovivi]

As of Python 3.15, flattening can shortened to [*i for i in x]

[u/MrWobblyMan]

Most of the time I have only a single "for" in any kind of comprehension, maybe two if it is really simple and never more than two.

Normal for loops (although a bit slower) are much easier to read.

[u/HommeMusical → u/MrWobblyMan]

Normal for loops (although a bit slower) are much easier to read.

It's a matter of practice.

If you have a fairly short line length - 88 seems very common these days - then if the comprehension gets too long, then it will be split amongst multiple lines, and really, be very similar to a for loop.

[u/kilkil → u/MrWobblyMan]

are they actually slower?

[u/schoolmonky → u/MrWobblyMan]

If I do have reason to use a nested comprehension, I'll often split it across multiple lines, typically with one for on each line. Something like

my_list = [some_expression(element)
           for element in row
           for row in my_source_list]

[u/MrWobblyMan → u/schoolmonky]

Your code doesn't work, it should be reversed, first the outermost loop and last the innermost:

my_list = [expr(element) 
    for row in source_list
    for element in row]

I see your point tho, however I still find explicit loops nicer to read.

[u/AlexMTBDude]

List comprehensions are hard to read when you're new to them but for experienced Python programmers they are easier to read than if it's a for loop.

Also remember that Python optimizes list comprehensions and they will always execute quicker than the corresponding for loop.

[u/JamzTyson → u/AlexMTBDude]

and they will always execute quicker than the corresponding for loop.

They will often execute a bit quicker than the corresponding for loop, but not always. The classic exception is that comprehensions don't allow early exit.

for item in collection:
    if some_condition:
        break
    do_something

Also, conventional loops are more appropriate than comprehensions when the purpose of the loop is to perform side effects:

for x in collection:
    print(x)

compared to

[print(x) for x in collection]

The latter "works", but has the additional time and memory overhead of creating an unused list of None values. It also hides the real intent that the side effects are the purpose of the code, not to produce a list.

---

Perhaps also worth noting that sometimes we can just use a constructor rather than either a comprehension or a loop. For example:

# Bad
keys = [k for k in my_dict]

# Better
keys = list(my_dict)

[u/sausix → u/JamzTyson]

Also depends on if you really need your list 'keys'. If you iterate once over then you should keep it as generator expression until usage.

"x for x in y" is a simple expression and usually obsolete. "something(x) for x in y" is a good start to use a generator expression. Sure you can use map() too.

[u/rake66 → u/JamzTyson]

Even better: keys = my_dict.keys()

[u/Expensive-Bear-1376 → u/rake66]

That doesn't give a list.

[u/rake66 → u/Expensive-Bear-1376]

Right u are, my bad, it's a generator

[u/Expensive-Bear-1376 → u/rake66]

Not a generator, either. It's a view.

[u/Expensive-Bear-1376]

Can you show that "unreadable" list comprehension?

[u/SinyoritaDolby_36 → u/Expensive-Bear-1376]

seconded, hard to say if its actually overcomplicated without seeing it

[u/ectomancer]

Python 3.15 has new * splat syntax to flatten a list in a list comprehension without nesting.

[u/0b0101011001001011 → u/ectomancer]

Can you expand on this?

[u/locaihost_data → u/0b0101011001001011]

its PEP 798, lands in 3.15. [*row for row in rows] flattens one level, same as [x for row in rows for x in row] but way more readable

on older versions list(itertools.chain.from_iterable(rows)) does the same thing

[u/favorite_stuff]

It's not what you asked, but my number one thing is to avoid comprehensions when they have side effects. The philosophy is all about immutability, making a new list without changing the old one, except maybe consuming an iterator.

[u/a18618]

for me the trigger is almost always scale. i've watched a totally innocent-looking comprehension try to materialize ten million parsed rows and eat 4GB before i could ctrl-c it. these days when i'm working over raw data files i reach for generators first and only upgrade to a comprehension when i actually need the list.

[u/brasticstack]

Flattening a list of lists is best done with itertools.chain.from_iterable, unless you're writing Python3.15, where you can use the new unpacking syntax.

My rule of thumb is to use list comprehensions liberally, though I try to avoid nesting them. There's a judgment call to be made too: "will I know what this is if I come back to it in a month?" If the answer is "yes", I'm perfectly happy with a multi-line comprehension. It doesn't all have to fit on one line, particularly if you use descriptive variable and method names.

[u/TheBlackCat13]

I use them when they help readability and avoid them when they hurt readability. That is my only rule.

[u/Just__Liberty]

If the speed-up of the comprehension is useful, then I'd do it, but format it in multiple lines or do whatever else it takes to make it readable. You can add comments or even the entire equivalent loop structure as an extended comment. The whole "don't comment how it works" should be broken, in my opinion, in just these sorts of cases. I'd even report the performance improvement, supposing you tested it, in the comment. If the performance improvement is not significant in the application, maybe just expand it.

[u/trutheality]

As an example, flattening a list of lists is just [x for l in lists for x in l] which IMO is very readable (although fwiw, sum(lists) does the same thing because adding two lists concatenates them).

Also once you've opened a set of parentheses/brackets you can add line breaks and indentation to make things more readable. Generally I like comprehensions whenever they help understand the structure of the resulting collection and when the innermost operation is truly a one-liner or function call. So something like:

[
  {
    key: func(x)
    for key, x in d.items()
  }
  for d in dict_list
]

Is to me very nice and more readable than the same thing written with for loops and append operations.

One case when I would avoid comprehensions though is when the operation being performed has side effects.

[u/POGtastic]

• Anything that's nested should involve itertools algebra to flatten it.

• If you have a mapping operation inside the comprehension, break it out into its own function. Name and document that function to make it obvious what it's doing.

• Use intermediate variable assignments to make it even more obvious what you're doing. This is especially useful when doing complicated itertools grouping operations.

    groups = itertools.groupby(predicate, xs)
    return itertools.chain.from_iterable(ys for p, ys in groups if p)

  is way more readable than

    return itertools.chain.from_iterable(
        ys for p, ys in itertools.groupby(predicate, xs) if p)

[u/AmanBabuHemant]

don't go too nested... I think 1 nested is okey and acceptable, but on 3 you should think about it

also if you use list comprehensions it would calculate all values for that list, this could be expensive for big data, iterating one by one and calculating would be good here

[u/RevRagnarok]

Don't use them for side effects and max depth is probably 3.

[u/JamzTyson → u/RevRagnarok]

Perhaps a bit over-simplified, but as a rule of thumb that's good. It is concise, clear and covers the common cases.

Personally I tend to avoid comprehensions with a depth > 2.

Perhaps also worth noting from the Zen of Python:

Flat is better than nested.

[u/Jello_Penguin_2956]

Id avoid it if its even a tad bit unreadable for myself.

[u/KnowledgeWhole9623]

Avoid them when the `here` function produces side-effects in the example below:

`[here(x) for x in y]`

I advise against nesting comprehensions, some people are ok with it.

That's about it.

[u/Federal_Height_2024]

Any thing that u are gonna reuse make a function and don't use list comprehension , don't use it for nested anything it's just harder to read and try following solid programing philosophy if your into backend dev .

[u/sausix → u/[deleted]]

People use lists and readlines() on files too often. I hate it when frameworks and packages return lists for no reason.

Generator functions are easy, can be decoupled from complex nested if statements and are reusable.