The first time you try to write a network client or server directly on top of sockets, you do something like this:

for filename in filenames:
    with open(filename, 'rb') as f:
        sock.sendall(f.read())

And then, on the other side:

for i in count(0):
    msg = sock.recv(1<<32)
    if not msg: break
    with open('file{}'.format(i), 'wb') as f:
        f.write(msg)

At first, this seems to work, but it fails on larger files. Or as soon as you try to use it across the internet. Or 1% of the time. Or when the computer is busy.

In reality, it can't possibly work, except in special circumstances. A TCP socket is a stream of bytes. Every time you call send (or sendall), you put more bytes on that stream. Every time you call recv, you get some or all of the bytes on the stream.

Let's say you send 1000 bytes, then send 1000000 bytes, and the other side calls recv. It might get 1000 bytes—but it just as easily might get 1001000 bytes, or 7, or 69102. There is no way to guarantee that it gets just the first send.

Why does it seem to work in my initial tests?

If you send 10000 bytes to localhost while the other side is waiting to receive something, and there's enough idle time on your CPUs to run the other side's code, your OS will probably just copy your 10000-byte send buffer over to the other side's receive buffer and give it the data all at once. It's not guaranteed, but it will usually happen. But only if the other side is on the same machine (or at least in the same bridged LAN), and your data fits into a single buffer, and you don't finish two sends before it gets enough CPU time to do its receive, and so forth.

So, what's the solution?

The key is that the other side has to somehow know that the next 1000 bytes are the message it wants. This means that unless you have some out-of-band way of transmitting that information, or your messages are some type that inherently includes that information (e.g., JSON objects, or MIME messages), you have to create a byte stream that has not just your messages, but enough information to tell where one message ends and the next begins.

In other words, you have to design, and then implement, a protocol.

That sounds scary… but it really isn't.

A simple protocol

Assuming your messages can't be more than 4GB long, just send the length, packed into exactly 4 bytes, and then you send the data itself. So, the other side always knows how much to read: Read exactly 4 bytes, unpack it into a length, then read exactly as many bytes as that:

def send_one_message(sock, data):
    length = len(data)
    sock.sendall(struct.pack('!I', length))
    sock.sendall(data)

def recv_one_message(sock):
    lengthbuf = recvall(sock, 4)
    length, = struct.unpack('!I', lengthbuf)
    return recvall(sock, length)

That's almost a complete protocol. The only problem is that Python doesn't have a recvall counterpart to sendall, but you can write it yourself:

def recvall(sock, count):
    buf = b''
    while count:
        newbuf = sock.recv(count)
        if not newbuf: return None
        buf += newbuf
        count -= len(newbuf)
    return buf

Protocol design

Protocol implementation

(Someone pointed out to me that Ned Batchelder has a similar post called Keep data out of your variable names. As usual for him, he's got a very concise answer that covers everything that matters in only a few paragraphs, so you might want to just read that.)

One of the most common Python questions on StackOverflow is some variation of "How do I create a bunch of variables in a loop?"

The answer is: You don't.

You can…

The direct answer is simple. Depending on whether you have names or numbers, it's one of the following:

    for name in names:
        globals()[name] = 0

    for i in range(10):
        globals()['variable{}'.format(i)] = 0

If you're trying to create function locals, builtins, class attributes, or instance attributes rather than globals, it's slightly different, but no harder. (And most people who want this want to create globals.)

But you shouldn't

But the right answer is almost always that you don't want to do that.

Why not?

If you want to access the variables statically, you must have the names statically. So just create them statically:

    spam = eggs = beans = bacon = sausage = lobster = 0

But that's obvious, and you probably knew that. Presumably you want to access the variables dynamically as well. Maybe you're getting a name from the user, or computing a number with some mathematical expression.

So, every time you want to access variable name or #n, do you want to do something like this:

    value = globals()[name]
    value = globals()['variable{}'.format(n)]

Ouch! If you just used a list or dict in the first place:

    variables = {name: 0 for name in names]
    variables = [0 for _ in range(10)]

… you could just access them like this:

    value = variables[name]
    value = variables[n]

But what about…?

Even in the rare cases somewhere in between the static and dynamic cases, you _still_ usually don't want to create variables dynamically.

For example, let's say you're building a bunch of values because you're going to hit one of 20 randomly-chosen code paths. Each of those code paths accesses its variables statically, but you don't know which one you're going to take, so you need to create the variables dynamically. Using a dict means those 20 code paths are all going to be a bit clumsy. For example, instead of this:

    if cmd == 'h':
        return (variable_a**2 + variable_b**2) ** .5

You'd have:

    if cmd == 'h':
        return (variables['a']**2 + variables['b']**2) ** .5

Ick. If you're got a static name, you don't want to look it up dynamically.

The solution here is to create a namespace by, e.g., using collections.namedtuple:

    variables = namedtuple('Variables', names)._make(0 for _ in names)

Now, you can access them statically:

    if cmd == 'h':
        return (variables.a**2 + variables.b**2) ** .5

So why can I do it?

Why does Python give you mutable access to globals and locals (and let you pass substitute globals and locals to functions where it matters)?

Because "almost always" isn't the same as "always". What if you want to build a debugger, interactive interpreter, or in-game console? Or a bridge library like ctypes, win32com, or pyobjc?

More generally, Python doesn't try to prevent you from shooting yourself in the foot; it just tries to make the obvious way to do things the best way. Just like exec, getframe, marshal, etc., globals is there when you need it—but if you think you need it, your first step should be to make sure you really do, and your second step should be to make doubly sure.

Novices to Python often come up with code that tries to build and evaluate strings, like this:

for name in names:
    exec('{} = {}'.format(name, 0))

… or this:

exec('def func(x): return {} * x**2 + {} * x + {}'.format(a, b, c))
return func

99% of the time, this is a problem you shouldn't be solving in the first place. (In the first case, you should almost certainly be using a dictionary or namespace full of names, instead of a bunch of dynamically-created variables. In the second case, you probably don't need the function; you can pass the tuple (a, b, c) around, or build an Expression class with a __call__ method.)

But, even when you do need to solve such problems, you still almost never need exec to do so. Compare the above to:

for name in names:
    globals()[name] = 0

def func(x): return a * x**2 + b * x + c
return func

Functions, classes, module globals… almost everything in Python is a first-class object, and has full reflective capabilities. So, why not use them?

The counter-question is: Python also has exec and eval, so why not use them? But there's a good answer to that one.

Most people go right to performance—it's obviously slower to evaluate a string to bytecode and run it than to just run byte code. But in reality, that's rarely an issue. The real problem is that it makes the code harder to process for everyone.

First, exec makes it harder to human beings to read your code. In order to figure out what's happening, I don't just have to read your code, I have to read your code, figure out what string it's going to generate, then read that virtual code. So, if you're working on a team, or publishing open source software, or asking for help somewhere like StackOverflow, you're making it harder for other people to help you. And if there's any chance that you're going to be debugging or expanding on this code 6 months from now, you're making it harder for yourself directly.

Second, exec makes it harder for tools to read your code. The interpreter can't print good tracebacks when there's a problem to debug. An optimizer can't optimize it. An IDE or refactoring tool can't find definitions. A linter can't tell whether your code is likely to be correct. And so on.

And finally, exec makes it harder for your own code to read your code. It's a lot easier to manipulate data (including functions, etc.) than string representations of that data. For example, if I want to make a function that returns the inverse of that generated function above, compare:

s = 'def func(x): return {} * x**2 + {} * x + {}'.format(a, b, c)
wrapped = re.sub(r"return\s+(.*)", r"return 1/(\1)", s)

… to:

def func(x): return a * x**2 + b * x + c

def wrapped(x): return 1/func(x)

Older dynamic languages didn't have enough functionality to avoid exec, so it was common in Tcl, and early versions of JavaScript and PHP. If you're coming from one of those languages, you may find yourself reaching for exec pretty often. But you should always ask yourself whether there's a better way to do what you're trying to do, and the answer will almost always be yes.

So, why does Python even have exec and eval? Well, just because something is _rarely_ useful doesn't mean it's _never_ useful. For example, let's say you're building a game, and you want it to be moddable. You want to be able to load scripts and run them. You probably also want an in-game console where people can run lines of code interactively. You need to be able to execute those interactive lines of code somehow, right? You probably want to use the code module in the stdlib, rather than calling exec manually… but that module itself has to be written somehow, so, exec is still necessary.

If you look at the major changes in Python 3, other than the Unicode stuff, most of them are about replacing list-based code with iterator-based code. When you run 2to3 on your code, every map(x, seq) or d.items() gets turned into list(map(x, seq)) or list(d.items()), and if you want to write dual-version code you have to use things like six.map(x, seq) or list(six.iteritems(d)).

So, why?

Often, you don't really need a list, you just need something you can iterate over. In the case of d.items(), the Py3 version can just use a view of the dictionary and not have to copy anything. In the case of map(), you do end up with the same amount of copying, but you don't have to build the list all at once, so there's less memory allocation going on—and, if you don't consume the whole list, you don't pay for the whole thing.

But that's not the point. The point is that if you combine iterator-transforming functions, it's semantically different from combining sequence-transforming functions. Your code ends up interleaved into a pipeline, working element-by-element. Some things that would be hard to express, or even impossible, now become trivial.

When Haskell programmers want to prove that their language is cooler than yours without using the word "monad" (which, they've learned, just gives people headaches and makes them stop listening), they use examples from math, and show how easy they are to transform into Haskell. Let's pick one of their favorites: Computing the first n primes. Here's how to do it:

numbers = itertools.count(2)
while True:
    prime = next(numbers)
    yield prime
    def relatively_prime(n, prime=prime):
        return n % prime
    numbers = filter(relatively_prime, numbers)

first_100_primes = itertools.islice(primes(), 100)

That's pretty close to the way you'd describe it mathematically. The problem with coding it this way is that you're starting with an infinite list, and at each step you're filtering it into another infinite list, which takes infinite time. The fact that you're later going to throw away all but the first 100 doesn't help.

But if you're dealing with iterators instead of lists, it _does_ help. You start with an infinite iterator. At each step, you're adding a finite filter to the future values of that iterator. So, when you generate the first 100 primes, you only end up stepping through the first 540 numbers in the infinite list, and applying a finite number of filters to each one.

Think about how you'd have to express that if you didn't have iterator-transforming pipelines. Something like this:

first_100_primes = []
hopefully_big_enough = 1000
hopefully_enough_numbers = range(2, hopefully_big_enough)
while len(first_100_primes) < 100:
    prime = hopefully_enough_numbers.pop()
    first_100_primes.append(prime)
    for i, n in hopefully_enough_numbers[:]:
        if n % prime == 0:
            del hopefully_enough_numbers[i]
    while not hopefully_enough_numbers:
        hopefully_enough_numbers = range(hopefully_big_enough, hopefully_big_enough + 1000)
        hopefully_big_enough += 1000
        for prime in first_100_primes:
            for i, n in hopefully_enough_numbers[:]:
                if n % prime == 0:
                    del hopefully_enough_numbers[i]

But obviously, you wouldn't do it like that; you'd manually simulate the infinite list, and manually pipeline all of the filters:

first_100_primes = []
next_number = 2
while len(first_100_primes) < 100:
    first_100_primes.append(next_number)
    def is_prime(n, primes):
        for prime in primes:
            if n % prime == 0:
                return False
        return True
    def get_next_relatively_prime(n, primes):
        while True:
            n += 1
            if is_prime(n, primes):
                return n
    next_number = get_next_relatively_prime(next_number, first_100_primes)

The Haskell-y version is obviously a lot easier to read. Plus, it lets you laugh at the smug Haskell types by proving that your language can do everything theirs can, at which point they have to resort to talking about the rich type system and you win, because of Guido's Law, which is like Godwin's Law, but with monads in place of Hitler. (I know, anyone with dual PhDs in type theory and history knows that Hitler is more like an applicative than a monad, but just as practicality beats purity, a heavy klomp beats a PhD.)

But Haskellites aren't the only smug types out there. Coroutines are cool too, and anyone who's still programming with those old-fashioned subroutines is so last century. Python generators give you all the benefits of coroutines, but with a syntax that means you don't have to realize it when it's not important.

And if you think about your iterator-transforming pipeline, that's what you're doing. At each step, each filter suspends and hands things off to the next filter until you get to the end of the chain and come back to the loop. You create a new coroutine and add it to that chain just by writing "numbers = filter(relatively_prime, numbers)". Who yields to who doesn't need to be specified—not because of some deep magic, but because it's obvious.
Of course we all know that generators aren't _really_ coroutines. But as of PEP 342 (Python 2.5, 2005), Python generators aren't just generators. Academics continue to refer to "generators" as "what Python had before 2.5", meaning you have to say "enhanced generators" or "PEP 342 generators" to mean what every Python user just calls "generators". And "enhanced generators" _are_ coroutines. But they _still_ have that nice sequential-subroutine-style syntax as generators. You still don't have to think about them as coroutines when it isn't relevant, but you can easily do so when it is. (The lack of "yield from" meant you sometimes didn't get that advantage, but that was fixed with PEP 380 in 3.3 in 2012.)

And that leads to even cooler things. The two obvious ways to write async I/O are with threads (green/micro or otherwise), which leads to synchronization problems they bring, or with callbacks, which forces you to think of the flow of control backward in complex cases, and leads to callback-hell even in simple cases. You can improve things a little by atomic data structures like queue.queue for all inter-thread communication, or by using promises (deferreds) for chaining callbacks, but still, you're adding a lot of mental burden on top of the equivalent synchronous code. But with explicit coroutines as in PEP 3156, you don't have any of that burden. The synchronous semantics translate directly to asynchronous semantics just by putting "yield from" at all suspension points, and you're done. The resulting code is clearly readable and easily writable.