Overview

Python’s iteration system is one of its most powerful features, allowing for elegant and memory-efficient data processing. This article explores the core concepts of iterables, iterators, and generators - three closely related components that make Python’s iteration protocol both flexible and powerful.

In Python, basic data types inherit from the iterable class and can produce iterators through the iter(iterable) function. These iterators maintain independent states and return appropriate values via next(iterator) during iteration.

The relationship between iterables and iterators can be thought of as a vending machine (iterable) and the drinks it dispenses (iterator). Once an iterator is consumed, it’s discarded. However, separating iterables and iterators often requires more code than necessary for many use cases.

This is where generators come in - they combine the functionality of iterables and iterators in a clean, efficient way. Generators are particularly advantageous when handling large datasets as they create values dynamically rather than storing them all in memory.

Abstract Base Classes

Python’s collections module defines the foundation for these iteration components:

• Basic data types (lists, tuples, sets, dictionaries) are all iterators
• The collections.abc module provides Container, Collection, and Sequence as framework classes
• All basic data types inherit from the abstract Collection class
• Basic data types also inherit from iterable, but not from iterator or generator

Iterables

An iterable is any object that can be iterated over, including basic data types, files, and strings:

assert issubclass(str, iterable)
assert issubclass(io.TextIOWrapper, iterable)

Simply put, an iterable is anything that can be used in a for loop. The range() function also returns an iterable.

To qualify as an iterable, an object must:

1. Implement the __iter__ abstract method
2. Return a new iterator each time it’s called

print(iterable._abstractmethods_)
>>> frozenset({'__iter__'})

Iterators

An iterator is an object that maintains state and returns elements one by one until exhaustion. Each iterator maintains its own independent state:

assert iter(i) != iter(i)

To qualify as an iterator, an object must:

1. Implement __iter__ that returns itself
2. Implement __next__ to define the next value when called with next(iterator)

print(iterator._abstractmethods_)
>>> frozenset({'__next__'})

Creating Iterables and Iterators

Here’s a simple example of creating custom iterable and iterator classes:

from random import randint

class RandomIntIterable:
    def __init__(self, n):
        self.n = n
    
    def __iter__(self):
        return RandomIntIterator(self.n)

class RandomIntIterator:
    def __init__(self, n):
        self.count = 0
        self.n = n
        
    def __iter__(self):
        return self
        
    def __next__(self):
        if self.count < self.n:
            self.count += 1
            return randint(1, 100)
        else:
            raise StopIteration

Inheritance Relationship

In Python’s class hierarchy:

• Iterator inherits from iterable (it’s a child class)
• The reverse is not true

print(issubclass(iterator, iterable))
>>> True

Generators

Generators combine the functionality of iterators and iterables, and are commonly used in practical Python development. They’re stored in the collections.abc module like the other classes.

Creating Generators with yield

Generators are defined as functions, not classes. They use yield instead of return and significantly reduce code complexity:

from random import randint

def random_number_generator(n):
    count = 0
    while count < n:
        yield randint(1, 100)
        count += 1

Generator Comprehensions

Similar to list comprehensions, Python allows for generator comprehensions:

from random import randint
g = (randint(1, 100) for _ in range(5))

These are very concise, but may be limited for complex implementations.

Inheritance Hierarchy

The inheritance relationship between these classes can be summarized as:

iterable > iterator > Generator

Practical Examples

When a function takes a single argument expression, the parentheses in generator expressions can be omitted:

print(sum([n**2 for n in range(1, 11)]))  # List comprehension
print(sum(n**2 for n in range(1, 11)))    # Generator expression (better performance)

Conclusion

Python’s iteration system provides a powerful framework for processing data efficiently. By understanding the differences and relationships between iterables, iterators, and generators, you can write more elegant and memory-efficient Python code. Generators in particular offer a clean solution for many iteration patterns, combining the power of iterables and iterators with minimal syntax.

For most practical applications, generators provide the best balance of functionality and simplicity, especially when working with large datasets where memory efficiency is important.