List Comprehension vs Generator Expression

Table of Contents

List comprehension and generator expressions are commonly used in Python as they both offer a convenient way to generate and work with lists. In this short article, I want to explore the performance differences in terms of speed and memory usage between the two approaches to working with lists.

Measuring performance

At first, let's have a look at the execution speed and memory usage between the two. To measure the performance and memory footprint differences between list comprehensions and generator expressions, we will use the script below.

import timeit
import sys

TIMES = 10000


def clock(label, cmd):
    res = timeit.repeat(cmd, number=TIMES)
    print(label, *("{:.3f} seconds".format(x) for x in res))


def size(label, obj):
    print(label, "%d bytes" % sys.getsizeof(obj))


range_size = 10000

listcomp_label = "listcomp        :"
genexp_label = "genexp          :"

clock(listcomp_label, "sum([num for num in range(%d)])" % range_size)
clock(genexp_label, "sum(num for num in range(%d))" % range_size)
size(listcomp_label, [num for num in range(range_size)])
size(genexp_label, (num for num in range(range_size)))

listcomp        : 2.749 seconds 2.913 seconds 2.742 seconds
genexp          : 4.163 seconds 4.152 seconds 4.161 seconds
listcomp        : 87624 bytes
genexp          : 88 bytes

As you can see, list comprehensions are a bit faster but allocate more memory.

Memory benefits of Generator Expressions

As explained by PEP 289, generator expressions are "high performance, memory efficient generalization of list comprehensions".

The idea behind generator expressions is that the entire list does not have to be created in memory as we only need to iterate over the elements one at a time.

This results in a lower memory footprint since we don't need to store all of the objects in the memory simultaneously. As seen in our test case in the previous section, this holds true for our list of integers.

Performance benefits of List Comprehensions

List comprehensions are usually faster than generator expressions as generator expressions create another layer of overhead to store references for the iterator. However, the performance difference is often quite small.

For applications where execution speed is the top priority, list comprehension would be favoured over generator expressions if the memory constraints allow it.

When to use Generator Expressions?

The primary reason to use generator expressions is when your code needs to be memory efficient.

For example, when working with large datasets storing the entire list in memory is often not possible as you simply don't have enough memory. Another application might be single-board computers with limited memory.