< Previous Page | Home Page | Next Page >
Iterators¶
Often an important piece of data analysis is repeating a similar calculation, over and over, in an automated fashion.
For example, you may have a table of a names that you'd like to split into first and last, or perhaps of dates that you'd like to convert to some standard format.
One of Python's answers to this is the iterator syntax.
We've seen this already with the range iterator:
for i in range(10):
print(i)
Here we're going to dig a bit deeper.
It turns out that in Python 3, range is not a list, but is something called an iterator, and learning how it works is key to understanding a wide class of very useful Python functionality.
Iterating over lists¶
Iterators are perhaps most easily understood in the concrete case of iterating through a list. Consider the following:
for value in [2, 4, 6, 8, 10]:
# do some operation
print(value + 1)
The familiar "for x in y" syntax allows us to repeat some operation for each value in the list.
The fact that the syntax of the code is so close to its English description ("for [each] value in [the] list") is just one of the syntactic choices that makes Python such an intuitive language to learn and use.
But the face-value behavior is not what's really happening.
When you write something like "for val in L", the Python interpreter checks whether it has an iterator interface, which you can check yourself with the built-in iter function:
iter([2, 4, 6, 8, 10])
It is this iterator object that provides the functionality required by the for loop.
The iter object is a container that gives you access to the next object for as long as it's valid, which can be seen with the built-in function next:
I = iter([2, 4, 6, 8, 10])
print(next(I))
print(next(I))
print(next(I))
What is the purpose of this level of indirection? Well, it turns out this is incredibly useful, because it allows Python to treat things as lists that are not actually lists.
range(): A List Is Not Always a List¶
Perhaps the most common example of this indirect iteration is the range() function in Python 3 (named xrange() in Python 2), which returns not a list, but a special range() object:
range(10)
range, like a list, exposes an iterator:
iter(range(10))
So Python knows to treat it as if it's a list:
for i in range(10):
print(i, end=' ')
The benefit of the iterator indirection is that the full list is never explicitly created!
We can see this by doing a range calculation that would overwhelm our system memory if we actually instantiated it (note that in Python 2, range creates a list, so running the following will not lead to good things!):
N = 10 ** 12
for i in range(N):
if i >= 10: break
print(i)
If range were to actually create that list of one trillion values, it would occupy tens of terabytes of machine memory: a waste, given the fact that we're ignoring all but the first 10 values!
In fact, there's no reason that iterators ever have to end at all!
Python's itertools library contains a count function that acts as an infinite range:
from itertools import count
for i in count():
if i >= 10:
break
print(i)
Had we not thrown-in a loop break here, it would go on happily counting until the process is manually interrupted or killed (using, for example, ctrl-C).
Useful Iterators¶
This iterator syntax is used nearly universally in Python built-in types as well as the more data science-specific objects we'll explore in later sections. Here we'll cover some of the more useful iterators in the Python language:
enumerate¶
Often you need to iterate not only the values in an array, but also keep track of the index. You might be tempted to do things this way:
L = [2, 4, 6, 8, 10]
for i in range(len(L)):
print(i, L[i])
Although this does work, Python provides a cleaner syntax using the enumerate iterator:
for i, val in enumerate(L):
print(i, val)
This is the more "Pythonic" way to enumerate the indices and values in a list.
zip¶
Other times, you may have multiple lists that you want to iterate over simultaneously.
You could certainly iterate over the index as in the non-Pythonic example we looked at previously, but it is better to use the zip iterator, which zips together iterables:
L = [2, 4, 6, 8, 10]
R = [3, 6, 9, 12, 15]
for lval, rval in zip(L, R):
print(lval, rval)
Any number of iterables can be zipped together, and if they are different lengths, the shortest will determine the length of the zip.
map and filter¶
The map iterator takes a function and applies it to the values in an iterator:
# find the first 10 square numbers
square = lambda x: x ** 2
for val in map(square, range(10)):
print(val, end=' ')
The filter iterator looks similar, except it only passes-through values for which the filter function evaluates to True:
# find values up to 10 for which x % 2 is zero
is_even = lambda x: x % 2 == 0
for val in filter(is_even, range(10)):
print(val, end=' ')
The map and filter functions, along with the reduce function (which lives in Python's functools module) are fundamental components of the functional programming style, which, while not a dominant programming style in the Python world, has its outspoken proponents (see, for example, the pytoolz library). We'll have another look at both in the chapter on Data Science.
List Comprehensions¶
If you read enough Python code, you'll eventually come across the terse and efficient construction known as a list comprehension. This is one feature of Python we expect you will fall in love with if you've not used it before; it looks something like this:
[i for i in range(20) if i % 3 > 0]
The result of this is a list of numbers which excludes multiples of 3. While this example may seem a bit confusing at first, as familiarity with Python grows, reading and writing list comprehensions will become second nature.
Basic List Comprehensions¶
List comprehensions are simply a way to compress a list-building for-loop into a single short, readable line. For example, here is a loop that constructs a list of the first 12 square integers:
L = []
for n in range(12):
L.append(n ** 2)
L
The list comprehension equivalent of this is the following:
[n ** 2 for n in range(12)]
As with many Python statements, you can almost read-off the meaning of this statement in plain English: "construct a list consisting of the square of n for each n up to 12".
This basic syntax, then, is [expr for var in iterable], where expr is any valid expression, var is a variable name, and iterable is any iterable Python object.
Multiple Iteration¶
Sometimes you want to build a list not just from one value, but from two. To do this, simply add another for expression in the comprehension:
[(i, j) for i in range(2) for j in range(3)]
Notice that the second for expression acts as the interior index, varying the fastest in the resulting list.
This type of construction can be extended to three, four, or more iterators within the comprehension, though at some point code readibility will suffer!
Conditionals on the Iterator¶
You can further control the iteration by adding a conditional to the end of the expression. In the first example of the section, we iterated over all numbers from 1 to 20, but left-out multiples of 3. Look at this again, and notice the construction:
[val for val in range(20) if val % 3 > 0]
The expression (i % 3 > 0) evaluates to True unless val is divisible by 3.
Again, the English language meaning can be immediately read off: "Construct a list of values for each value up to 20, but only if the value is not divisible by 3".
Once you are comfortable with it, this is much easier to write – and to understand at a glance – than the equivalent loop syntax:
L = []
for val in range(20):
if val % 3:
L.append(val)
L
< Previous Page | Home Page | Next Page >