Generators

In Python we can create our own iterators and generators classes implementing the required methods, but there is a very simple way of creating new custom iterators, the generator way.

A generator looks just like a function, but instead of return it uses yield. A generator function returns a generator iterator, not a value, and, like any other generator they can be iterated over until they are consumed.

Generators are not functions

Which would be the result of running the following code?

Generators might look like functions, but they are not. When we call a function we get a result back a soon as the function reaches the first return and that’s it. If we call the function again, the function execution will start from the top all over again. Generators have a very different behavior. When we call them they create a generator prepared to yield one element after another when we ask for it. Every time that we enter the generator it resumes the execution where it left.

The generator will raise a StopIteration if it reaches a return.

yield produces a value and pauses execution while return ends the generator and raises StopIteration.

Generators instead of container types

One very common use case, when you don’t need to have all members materialized in memory at the same time, is to create generators instead of list. For instance, we could create lists of squares:

We have created an empty list, we have appended the squares one by one and, finally, we have returned the whole list. If we don’t need to hold all elements in memory at once we could do the same with a generator.

By the way, the generator option uses much less memory and can be also faster. In my computer calculating the sum of the first 100.000 squares one thousand times required 3.2 seconds when using lists and 2.8 when using generators.

Generator expressions

Generators are easy to create, but generator expressions are even simpler.

Generator expressions have exactly the same syntax as the list comprehensions but instead of square brackets they use parentheses.

Generators are easy to create, but generator expressions are even simpler.

Generators are iterators

Generators are iterators, so they have the strengths, but also the limitations of the iterators.

Generators have no length.

Generators are consumed.

You have to be careful if you want to create a numpy array out of it.