Lazy tuple unpacking
In a recent discussion on python-ideas, Paul Tagliamonte suggested that tuple unpacking could be lazy, using iterators. But it was immediately pointed out that this would break lots of code:a, b, *c, d, e = range(10000000000)
If c were an iterator, you'd have to exhaust c to get to d and e.
One way to solve this would be to try to slice the iterable:
try:
a, b, c, d, e = i[0], i[1], i[2:-2], i[-2], i[-1]
except TypeError:
# current behavior
Then c ends up as a range object, which is better than either an iterator or a list.
With most types, of course, you'd get a list, because most types don't have lazy slices, but that's no worse than today. And range is by no means the only type that does; lazy slices are critical to NumPy.
So, why doesn't Python do this?
That's actually a good question. I'm sure the answer would be more obvious if it weren't 4am, and I'll be sure and edit this section to make myself look less stupid when I think of it. :)
Smarter lazy tuple unpacking through sequence views
In Swift-style map and filter views, I looked at Swift's sequence types (Swift calls them collections, but let's stick to Python terms to avoid confusion), and pointed out that they can do things that Python's can't.And if you take things a bit farther, what you get is exactly what Swift has. In Swift, some functions (although not at all consistently, unfortunately) that return iterators in Python return lazy sequences in Swift that are views on the original sequence(s) if possible, falling back to iterators if not.
For example, map acts as if written like this:
class MapView(collections.abc.Sequence):
def __init__(self, func, sequence):
self.func, self.sequence = func, sequence
def __getitem__(self, index):
if isinstance(index, slice):
# do slice stuff
else:
return self.func(self.iterable[index])
def map(func, sequence):
if isinstance(sequence, collections.abc.Sequence):
return MapView(func, sequence)
else:
return (func(item) for item in sequence)
And now, you can do this:
a, b, *c, d, e = map(lambda i: i*2, range(10000000000))
… and c is a MapView object over a range object.
And Swift takes this even further by adding bidirectional sequences—sequences that can't be indexed by integers, but can be indexed by a special kind of index that knows how to get the next or previous value.
BidirectionalSequence
Imagine these two types added to collections.abc:class BidirectionalIndex:
@abc.abstractmethod
def __succ__(self): pass
@abc.abstractmethod
def __pred__(self): pass
@abc.abstractmethod
def __begin__(self): pass
@abc.abstractmethod
def __end__(self): pass
@abc.abstractmethod
def __getitem__(self, index): pass
# existing stuff
A BidirectionalSequence is expected to return a BidirectionalIndex from __begin__ and __end__, and to accept one, or a slice of them, in __getitem__, and return a TypeError if you try to pass an int. A Sequence has the same methods but with something like an integer, as usual. (This means that int, and maybe the ABCs in numbers as well, has to implement __succ__ and __pred__.)
Now we could implement a, b, *c, d, e = i like this:
try:
a, b, c, d, e = i[0], i[1], i[2:-2], i[-2], i[-1]
except TypeError:
try:
begin, end = i.__begin__(), i.__end__()
except AttributeError:
# current behavior
else:
a = i[begin]
begin = begin.succ()
b = i[begin]
begin = begin.succ()
end = end.pred()
e = i[end]
end = end.pred()
d = i[end]
c = i[begin:end]
c = i[begin:end]
So, why would you want this? Well, just as map can return a lazy Sequence if given a Sequence, filter can return a lazy BidirectionalSequence if given a BidirectionalSequence. (Also, map can return a lazy BidirectionalSequence if given a BidirectionalSequence.) So you can do this:
a, b, *c, d, e = filter(lambda i: i*2, range(10000000000))
And this means generator expressions (or some new type of comprehension?) could similarly return the best possible type: a Sequence if given a Sequence and there's no if clauses, otherwise a BidirectionalSequence if given a BidirectionalSequence, otherwise an Iterator.
While we're at it, we could also provide a ForwardSequence, which is effectively usable in exactly the same cases as an Iterable that's not an Iterator, but provides an API consistent with BidirectionalSequence.
Reversible iterables
But that last point brings up a different possible way to get the same thing: reversible iterators.At first glance, it seems like all you need is a new Iterable subclass (that Sequence inherits) that adds a __reviter__ method. But then you also need some way to compare iterators, to see if they're pointing at the same thing. (It's worth noting that C++ iterators and Swift indexes can do that… but they're not quite the same thing as Python iterators; Swift generators, which are exactly the same thing as Python iterators, cannot.) And the code to actually use these things would be pretty complicated. For example, the unpacking would work like this:
try:
a, b, c, d, e = i[0], i[1], i[2:-2], i[-2], i[-1]
except TypeError:
try:
rit = reviter(i)
except TypeError:
# current behavior
else:
it = iter(i)
if it == rit: raise ValueError('Not enough values')
a = next(it)
if it == rit: raise ValueError('Not enough values')
e = next(rit)
if it == rit: raise ValueError('Not enough values')
b = next(it)
if it == rit: raise ValueError('Not enough values')
d = next(rit)
c = make_iterator_between(it, rit)
It should be pretty obvious how that make_iterator_between is implemented.
Double-ended iterators
But once you think about how that make_iterator_between works, you could just make it a double-ended iterator, with __next__ and __last__ methods. Since you've always got just a single object, you don't need that iterator equality comparison. And it's a lot easier to use. The unpacking would look like this:try:
a, b, c, d, e = i[0], i[1], i[2:-2], i[-2], i[-1]
except TypeError:
it = iter(i)
a = next(it)
b = next(it)
try:
e = last(it)
except TypeError:
# existing behavior
else:
d = last(it)
c = it
Summary
So, is any version of this a reasonable suggestion for Python?Maybe the first one, but the later ones, not at all.
Python hopped on the iterator train years ago, and 3.0's conversion of map and friends to iterators completed the transition. Making another transition to different kinds of lazy sequences at this point would be insane, unless the benefit was absolutely gigantic.
Reversible iterators are a much less radical change, but they're also a big pain to use, and a bit of a pain to implement.
Double-ended iterators are an even less radical change, and simpler both to use and to implement… but they're not a very coherent concept to explain. An iterator today is pretty simple to understand: It keeps track of a current location within some (possibly virtual) iterable. An iterator that can also go backward, fine. But an iterator that keeps track of a pair of locations and can only move inward? That's an odd thing.
In a new language, it might be another story. In fact, if you got lazy sequences right, iterators would be something that users rarely have to, or want to, look at. (Which makes the interface simpler, too—map doesn't have to worry about fallback behavior when called on an iterator, just don't let it be called on anything but a sequence.) Which raises the question of why Swift added iterables in the first place. (It's not because they couldn't think of how else generators and comprehensions could work, because Swift has neither… nor does it have extended unpacking, for that matter.)
View comments