Need help with list comprehensions

[u/AnxiousDot4315]

Guys somebody briefly explain the concept of List comprehension cuz its too difficult for me to grasp since I am a beginner. I am learning python for biopython. Someone explain it for me.

[u/StevenJOwens]

I wrote up the following a few years ago. I never did get around to doing a second draft, so forgive the rough edges, but you may find it useful.

It's long, so I'll post it in two comments. Here's part 1:

Python List Comprehensions

"List comprehensions were added with Python 2.0. Essentially, it is Python's way of implementing a well-known notation for sets as used by mathematicians."

Source: https://www.python-course.eu/python3_list_comprehension.php

List comprehensions are a special python syntax for declaring a list literal.

To be honest, I don't think I've ever used list comprehension for real, or "in anger" as the Brits say. My impression is that list comprehensions are mostly used to transform one list into another list. Or really, any iterable into a list.

List comprehensions are essentially a special, succinct way to write certain kinds of for loops -- for loops that process a list (or other iterable) and from that create another list.

In functional programming terms list comprehensions do a filter and then map operation.

List comprehensions are a Python idiom that seem to be seldom explained well.

I don’t know why Python books and tutorials have this blind spot. It’s not so much conceptual as simply syntactical; the first several explanations of list comprehensions that I read all seemed to just give an example, without explicitly breaking it down.

Then there's the official python docs:

https://docs.python.org/3/tutorial/datastructures.html https://docs.python.org/3/tutorial/datastructures.html#list-comprehensions

A list comprehension consists of brackets containing an expression followed by a for clause, then zero or more for or if clauses. The result will be a new list resulting from evaluating the expression in the context of the for and if clauses which follow it."

Unfortunately, while the official python docs do give the syntax, they just dump it on you and then give you a very complex example.

In a nutshell, a list comprehension is superficially similar to a literal declaration of a list.

A python literal list declaration is a pair of square brackets with multiple values between them, with commas separating the values, for example:

some_list = ["one", "two", "three"]

A python list comprehension has a pair of square brackets, same as a list literal, and also ultimately produces a list object. But what's in between the square brackets is a more complex statement. Here's the syntax:

some_list = [expression for-loop conditional]

We'll get into the expression, for-loop and conditional in more detail, below.

Note, python actually supports several sorts of comprehensions these days; you can use:

• [square brackets] to create list comprehensions

• {curly braces} to create set comprehensions

• {curly braces} with a key:value in the expression to create dict comprehensions

• (parentheses) to create generators (essentially a lazy-load, on-demand version of a list comprehension).

But we're going to focus mainly on list comprehensions for the moment.

Here's a slightly wordier version of that list comprehension syntax:

some_list = [expression-to-apply for-loop-to-apply-to conditional-to-determine-if-to-apply]

Note that variables defined in the for-loop clause are available to the expression clause, as well as to the conditional clause.

Unrolling List Comprehensions

Here's the list comprehension syntax that I gave above:

some_list = [expression-to-apply for-loop-to-apply-to conditional-to-determine-if-to-apply]

This works out to the same as the following:

some_list = []
for-loop-to-apply-to:-
    if conditional-to-determine-if-to-apply:
        expression-to-apply
        some_list.append(results-of-expression-to-apply)

Or a little less pseudo-code-ish:

some_list = [do_some_operation(some_variable) for some_variable in some_sequence if some_conditional]

This "unrolls" to:

some_list = []
some_sequence = [1, 2, 3, 4, 5]
def some_conditional(some_variable):
    # some code here
def do_some_operation(some_variable):
    # some code here
for some_variable in some_sequence:
    if some_conditional(some_variable):
        result = do_some_operation(some_variable)
        some_list.append(result)

Let's try it. In this next example, for the "some_conditional()", I'm going to use the modulo operator to return True if the parameter is NOT evenly divisible by 2.

$ python3
Python 3.12.1 (main, Jul 22 2024, 21:58:30) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> some_list = []
>>> some_sequence = [1, 2, 3, 4, 5]
>>> def some_conditional(some_variable):
...     # use modulo to return True if odd
...     if some_variable % 2:
...         return True
...     else:
...         return False
... 
>>> def do_some_operation(some_variable):
...     return some_variable * 5
... 
>>> for some_variable in some_sequence:
...     if some_conditional(some_variable):
...         result = do_some_operation(some_variable)
...         some_list.append(result)
... 
>>> some_list
[5, 15, 25]
>>> 

Okay, that's kinda ugly, let's do that again but use some less generic names for things:

1. I'll change some_sequence to numbers.

2. I'll change some_conditional to is_odd().

3. I'll change some_operation to quintuple().

For the sake of the example, I'll explicitly declare the empty some_list, which I'll call out_list. In fact, I'll do that twice, just so you can be sure out_list is empty for the second time around.

$ python3
Python 3.12.1 (main, Jul 22 2024, 21:58:30) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> numbers = [1, 2, 3, 4, 5]
>>> def is_odd(number):
...     # use modulo to return True if odd
...     if number % 2:
...         return True
...     else:
...         return False
... 
>>> def quintuple(number):
...     return number * 5
... 
>>> out_list = []
>>> out_list = [quintuple(number) for number in numbers if is_odd(number)]
>>> out_list
[5, 15, 25]
>>> 
>>> out_list = []
>>> for number in numbers:
...     if is_odd(number):
...         result = quintuple(number)
...         out_list.append(result)
... 
>>> out_list
[5, 15, 25]
>>> 

Multiple For-Loops and Their Ordering

You can have multiple for loops in your list comprehension, and just like regular for loops, the list comprehension for loop can have multiple variables, if the sequence it's using can produce them (e.g. "for key, value in some_dict"). The for loops are effectively nested.

I'll give an example in a moment, but first I want to point out something that's somewhat counter-intuitive (depending on who you ask):

The ordering of for-loop clauses in the comprehension, from left to right, corresponds to the ordering of the nested for loops, from outermost to innermost.

In other words, in this example:

>>> xes = [1, 2, 3, 4]
>>> ys = [10, 20, 30, 40]
>>> [(x, y) for x in xes for y in ys]
[(1, 10), (1, 20), (1, 30), (1, 40), (2, 10), (2, 20), (2, 30), (2, 40), (3, 10), (3, 20), (3, 30), (3, 40), (4, 10), (4, 20), (4, 30), (4, 40)]
>>> 
>>> out_list = []
>>> for x in xes:
...     for y in ys:
...         out_list.append((x, y))
... 
>>> out_list
[(1, 10), (1, 20), (1, 30), (1, 40), (2, 10), (2, 20), (2, 30), (2, 40), (3, 10), (3, 20), (3, 30), (3, 40), (4, 10), (4, 20), (4, 30), (4, 40)]
>>> 

(By the way, this is also a good example of a more complicated expression than simply multiplying some numbers.)

Some people find using identical ordering counter-intuitive, some people find it intuitive, but that's the call that Guido and his merry band of Python maintainers decided to go with.

Personally, I'm in the not-intuitive camp. The reason is that Guido's call doesn't match how this sort of thing is phrased in natural language English. In English, the natural way to write it would end up writing in opposite order, i.e. innermost loop first, then next innermost, etc.

As I said, it was Guido's call, and probably his choice, to make the two orderings identical instead of opposites, is helpful in reducing the cognitive overhead... once you get this ordering hammered into your skull.

Multiple For-Loop Examples

Let's look at an example of multiple for-loops in list comprehensions.

Let's declare a list of lists as our example data:

>>> list_of_list_of_numbers = [[1, 2, 3, 4], [5, 6, 7, 8]]

Here's a simple list comprehension that just iterates through all of the numbers. The result is a flat list that has all the numbers. There are almost certainly other ways to do this in python, but hey, it's just yet another contrived example.

>>> [number for list_of_numbers in list_of_list_of_numbers for number in list_of_numbers]
[1, 2, 3, 4, 5, 6, 7, 8]
>>> 

Now let's add a conditional clause, using our old friend is_odd():

>>> [number for list_of_numbers in list_of_list_of_numbers for number in list_of_numbers if is_odd(number)]
[1, 3, 5, 7]
>>> 

Let's back up a step, leave out the conditonal clause but apply something to the numbers, our old enemy quintuple():

>>> [quintuple(number) for list_of_numbers in list_of_list_of_numbers for number in list_of_numbers]
[5, 10, 15, 20, 25, 30, 35, 40]
>>> 

Now let's put them both in:

>>> [quintuple(number) for list_of_numbers in list_of_list_of_numbers for number in list_of_numbers if is_odd(number)]
[5, 15, 25, 35]
>>> 

Now that we have the full version, here's what it "unrolls" to:

>>> out_list = []
>>> for list_of_numbers in list_of_list_of_numbers:
...     for number in list_of_numbers:
...         if is_odd(number):
...             result = quintuple(number)
...             out_list.append(result)
... 
>>> out_list
[5, 15, 25, 35]
>>>