1. The first time you try to write a network client or server directly on top of sockets, you do something like this:

    for filename in filenames:
        with open(filename, 'rb') as f:
            sock.sendall(f.read())
    

    And then, on the other side:

    for i in count(0):
        msg = sock.recv(1<<32)
        if not msg: break
        with open('file{}'.format(i), 'wb') as f:
            f.write(msg)

    At first, this seems to work, but it fails on larger files. Or as soon as you try to use it across the internet. Or 1% of the time. Or when the computer is busy.

    In reality, it can't possibly work, except in special circumstances. A TCP socket is a stream of bytes. Every time you call send (or sendall), you put more bytes on that stream. Every time you call recv, you get some or all of the bytes on the stream.

    Let's say you send 1000 bytes, then send 1000000 bytes, and the other side calls recv. It might get 1000 bytes—but it just as easily might get 1001000 bytes, or 7, or 69102. There is no way to guarantee that it gets just the first send.

    Why does it seem to work in my initial tests?

    If you send 10000 bytes to localhost while the other side is waiting to receive something, and there's enough idle time on your CPUs to run the other side's code, your OS will probably just copy your 10000-byte send buffer over to the other side's receive buffer and give it the data all at once. It's not guaranteed, but it will usually happen. But only if the other side is on the same machine (or at least in the same bridged LAN), and your data fits into a single buffer, and you don't finish two sends before it gets enough CPU time to do its receive, and so forth.

    So, what's the solution?

    The key is that the other side has to somehow know that the next 1000 bytes are the message it wants. This means that unless you have some out-of-band way of transmitting that information, or your messages are some type that inherently includes that information (e.g., JSON objects, or MIME messages), you have to create a byte stream that has not just your messages, but enough information to tell where one message ends and the next begins.

    In other words, you have to design, and then implement, a protocol.

    That sounds scary… but it really isn't.

    A simple protocol

    Assuming your messages can't be more than 4GB long, just send the length, packed into exactly 4 bytes, and then you send the data itself. So, the other side always knows how much to read: Read exactly 4 bytes, unpack it into a length, then read exactly as many bytes as that:

    def send_one_message(sock, data):
        length = len(data)
        sock.sendall(struct.pack('!I', length))
        sock.sendall(data)
    
    def recv_one_message(sock):
        lengthbuf = recvall(sock, 4)
        length, = struct.unpack('!I', lengthbuf)
        return recvall(sock, length)

    That's almost a complete protocol. The only problem is that Python doesn't have a recvall counterpart to sendall, but you can write it yourself:

    def recvall(sock, count):
        buf = b''
        while count:
            newbuf = sock.recv(count)
            if not newbuf: return None
            buf += newbuf
            count -= len(newbuf)
        return buf

    Protocol design

    There are a few problems with the protocol described above:
    • Binary headers are hard to read, generate, or debug as a human.
    • Headers with no redundancy are not at all robust. One mistake, and you get out of sync and there's no way to recover. Worse, there's no way to even notice that you've made a mistake. You could read 4 arbitrary bytes from the middle of a file as a header, and think it means there's a 3GB file coming up, at which point you may run out of memory, or wait forever for data that's never coming, etc.
    • You have to pick some arbitrary limit, like 4GB. (Of course 640K ought to be enough for anyone…)
    • There's no way to pass additional information about each file, like the type, or name.
    • There's no way to extend the protocol in the future without making it completely incompatible.
    Not all of these problems will be relevant to every use case. You can solve the first 3 by using something like netstrings. Or by using delimiters instead of headers (such as a newline, assuming you're sending text messages that can't contain newlines, or you're willing to escape your data). If you need to solve all 5, consider using something like RFC2822, the "Name: Value" format used by HTTP, email, and many other internet protocols.

    Meanwhile, this is purely a data protocol. But often, you want to send commands or requests, and get back responses. For example, you might want to tell the server to "cd foo" before storing the next file. Or you may want to get a filter, and then send files matching that filter, instead of sending all of your files. Or you may want something you can build an interactive application around. This makes things more complicated, and there are many different ways to deal with the issues that arise. Look over HTTP, FTP, IMAP, SMTP, and IRC for some good examples. They're all well-known and well-documented, with both tutorials for learning and rigorous RFCs for reference. They have Python libraries to play with. They have servers and clients pre-installed or readily-available for almost every platform. And they're all relatively easy to talk with by hand, over a simple netcat connection.

    Protocol implementation

    There are also a few things that are less than ideal about the protocol handler above:
    • It's directly tied to sockets; you can't use it with, say, an SSL transport, or an HTTP tunnel, or a fake transport that feeds in prepared testing data.
    • It can only be used synchronously. Not a big deal for a urllib2-style client where it's appropriate to just block until the whole message is received, or even for a server that's only meant to handle a handful of simultaneous connections (just spawn a thread or child process for each one), but if you want to play a video as you receive it, or handle 10000 simultaneous clients, this is a problem.
    • Receiving 4 bytes is a pretty slow way to use sockets. Also, trying to receive exactly what you want makes it easy to accidentally run into the same bug you started with, and not notice it until you try a larger file or communicating across the internet. So, you generally want to receive some multiple of 4K at a time, append onto a buffer, and pull messages out of the buffer.
    Dealing with these problems can be complicated. For learning purposes, it's worth building a transport-independent protocol and a couple of transports from scratch, or an asynchronous server directly on top of select.select, etc. But for practical development, you don't want to do that.

    That's why there are networking frameworks that do all the hard stuff for you, so you only have to write the interesting parts that are relevant to your application. In Python 3.4 and later, there's a framework built in to the standard library called asyncio (and, for 3.3, a backport on PyPI). If you're using an older version, you only have asyncore and asynchat, and you don't want to use those; instead, you probably want to install and use something like Twisted or Monocle. There will be a bit of a learning curve, but it's worth it.

    If you're building on top of a higher-level protocol like HTTP or JSON-RPC, you can write even less code by using a higher-level framework. The standard library has clients for a number of major protocols, and servers for some. But it doesn't handle everything (e.g., JSON-RPC), and you still may want to reach for third-party libraries in some cases (e.g., for client-side HTTP, Requests is a lot easier to use than urllib2 for anything but the most trivial or most complex cases). And sometimes, the right answer is to go even higher-level and build a web service instead of a protocol, building on WSGI, or a server framework like Django.
    9

    View comments

  2. (Someone pointed out to me that Ned Batchelder has a similar post called Keep data out of your variable names. As usual for him, he's got a very concise answer that covers everything that matters in only a few paragraphs, so you might want to just read that.)

    One of the most common Python questions on StackOverflow is some variation of "How do I create a bunch of variables in a loop?"

    The answer is: You don't.

    You can

    The direct answer is simple. Depending on whether you have names or numbers, it's one of the following:

        for name in names:
            globals()[name] = 0
    
        for i in range(10):
            globals()['variable{}'.format(i)] = 0
    

    If you're trying to create function locals, builtins, class attributes, or instance attributes rather than globals, it's slightly different, but no harder. (And most people who want this want to create globals.)

    But you shouldn't

    But the right answer is almost always that you don't want to do that.

    Why not?

    If you want to access the variables statically, you must have the names statically. So just create them statically:

        spam = eggs = beans = bacon = sausage = lobster = 0

    But that's obvious, and you probably knew that. Presumably you want to access the variables dynamically as well. Maybe you're getting a name from the user, or computing a number with some mathematical expression.

    So, every time you want to access variable name or #n, do you want to do something like this:

        value = globals()[name]
        value = globals()['variable{}'.format(n)]
    

    Ouch! If you just used a list or dict in the first place:

        variables = {name: 0 for name in names]
        variables = [0 for _ in range(10)]
    

    … you could just access them like this:

        value = variables[name]
        value = variables[n]

    But what about…?

    Even in the rare cases somewhere in between the static and dynamic cases, you _still_ usually don't want to create variables dynamically.

    For example, let's say you're building a bunch of values because you're going to hit one of 20 randomly-chosen code paths. Each of those code paths accesses its variables statically, but you don't know which one you're going to take, so you need to create the variables dynamically. Using a dict means those 20 code paths are all going to be a bit clumsy. For example, instead of this:

        if cmd == 'h':
            return (variable_a**2 + variable_b**2) ** .5
    

    You'd have:

        if cmd == 'h':
            return (variables['a']**2 + variables['b']**2) ** .5
    

    Ick. If you're got a static name, you don't want to look it up dynamically.

    The solution here is to create a namespace by, e.g., using collections.namedtuple:

        variables = namedtuple('Variables', names)._make(0 for _ in names)

    Now, you can access them statically:

        if cmd == 'h':
            return (variables.a**2 + variables.b**2) ** .5

    So why can I do it?

    Why does Python give you mutable access to globals and locals (and let you pass substitute globals and locals to functions where it matters)?

    Because "almost always" isn't the same as "always". What if you want to build a debugger, interactive interpreter, or in-game console? Or a bridge library like ctypes, win32com, or pyobjc?

    More generally, Python doesn't try to prevent you from shooting yourself in the foot; it just tries to make the obvious way to do things the best way. Just like exec, getframe, marshal, etc., globals is there when you need it—but if you think you need it, your first step should be to make sure you really do, and your second step should be to make doubly sure.
    7

    View comments

  3. Novices to Python often come up with code that tries to build and evaluate strings, like this:

    for name in names:
        exec('{} = {}'.format(name, 0))
    

    … or this:

    exec('def func(x): return {} * x**2 + {} * x + {}'.format(a, b, c))
    return func
    

    99% of the time, this is a problem you shouldn't be solving in the first place. (In the first case, you should almost certainly be using a dictionary or namespace full of names, instead of a bunch of dynamically-created variables. In the second case, you probably don't need the function; you can pass the tuple (a, b, c) around, or build an Expression class with a __call__ method.)

    But, even when you do need to solve such problems, you still almost never need exec to do so. Compare the above to:

    for name in names:
        globals()[name] = 0
    
    def func(x): return a * x**2 + b * x + c
    return func
    

    Functions, classes, module globals… almost everything in Python is a first-class object, and has full reflective capabilities. So, why not use them?

    The counter-question is: Python also has exec and eval, so why not use them? But there's a good answer to that one.

    Most people go right to performance—it's obviously slower to evaluate a string to bytecode and run it than to just run byte code. But in reality, that's rarely an issue. The real problem is that it makes the code harder to process for everyone.

    First, exec makes it harder to human beings to read your code. In order to figure out what's happening, I don't just have to read your code, I have to read your code, figure out what string it's going to generate, then read that virtual code. So, if you're working on a team, or publishing open source software, or asking for help somewhere like StackOverflow, you're making it harder for other people to help you. And if there's any chance that you're going to be debugging or expanding on this code 6 months from now, you're making it harder for yourself directly.

    Second, exec makes it harder for tools to read your code. The interpreter can't print good tracebacks when there's a problem to debug. An optimizer can't optimize it. An IDE or refactoring tool can't find definitions. A linter can't tell whether your code is likely to be correct. And so on.

    And finally, exec makes it harder for your own code to read your code. It's a lot easier to manipulate data (including functions, etc.) than string representations of that data. For example, if I want to make a function that returns the inverse of that generated function above, compare:

    s = 'def func(x): return {} * x**2 + {} * x + {}'.format(a, b, c)
    wrapped = re.sub(r"return\s+(.*)", r"return 1/(\1)", s)
    
    
    … to:

    def func(x): return a * x**2 + b * x + c
    def wrapped(x): return 1/func(x)
    

    Older dynamic languages didn't have enough functionality to avoid exec, so it was common in Tcl, and early versions of JavaScript and PHP. If you're coming from one of those languages, you may find yourself reaching for exec pretty often. But you should always ask yourself whether there's a better way to do what you're trying to do, and the answer will almost always be yes.

    So, why does Python even have exec and eval? Well, just because something is _rarely_ useful doesn't mean it's _never_ useful. For example, let's say you're building a game, and you want it to be moddable. You want to be able to load scripts and run them. You probably also want an in-game console where people can run lines of code interactively. You need to be able to execute those interactive lines of code somehow, right? You probably want to use the code module in the stdlib, rather than calling exec manually… but that module itself has to be written somehow, so, exec is still necessary.
    0

    Add a comment

  4. If you look at the major changes in Python 3, other than the Unicode stuff, most of them are about replacing list-based code with iterator-based code. When you run 2to3 on your code, every map(x, seq) or d.items() gets turned into list(map(x, seq)) or list(d.items()), and if you want to write dual-version code you have to use things like six.map(x, seq) or list(six.iteritems(d)).

    So, why?

    Often, you don't really need a list, you just need something you can iterate over. In the case of d.items(), the Py3 version can just use a view of the dictionary and not have to copy anything. In the case of map(), you do end up with the same amount of copying, but you don't have to build the list all at once, so there's less memory allocation going on—and, if you don't consume the whole list, you don't pay for the whole thing.

    But that's not the point. The point is that if you combine iterator-transforming functions, it's semantically different from combining sequence-transforming functions. Your code ends up interleaved into a pipeline, working element-by-element. Some things that would be hard to express, or even impossible, now become trivial.

    When Haskell programmers want to prove that their language is cooler than yours without using the word "monad" (which, they've learned, just gives people headaches and makes them stop listening), they use examples from math, and show how easy they are to transform into Haskell. Let's pick one of their favorites: Computing the first n primes. Here's how to do it:

    numbers = itertools.count(2)
    while True:
        prime = next(numbers)
        yield prime
        def relatively_prime(n, prime=prime):
            return n % prime
        numbers = filter(relatively_prime, numbers)
    
    first_100_primes = itertools.islice(primes(), 100)
    
    
    That's pretty close to the way you'd describe it mathematically. The problem with coding it this way is that you're starting with an infinite list, and at each step you're filtering it into another infinite list, which takes infinite time. The fact that you're later going to throw away all but the first 100 doesn't help.

    But if you're dealing with iterators instead of lists, it _does_ help. You start with an infinite iterator. At each step, you're adding a finite filter to the future values of that iterator. So, when you generate the first 100 primes, you only end up stepping through the first 540 numbers in the infinite list, and applying a finite number of filters to each one.

    Think about how you'd have to express that if you didn't have iterator-transforming pipelines. Something like this:

    first_100_primes = []
    hopefully_big_enough = 1000
    hopefully_enough_numbers = range(2, hopefully_big_enough)
    while len(first_100_primes) < 100:
        prime = hopefully_enough_numbers.pop()
        first_100_primes.append(prime)
        for i, n in hopefully_enough_numbers[:]:
            if n % prime == 0:
                del hopefully_enough_numbers[i]
        while not hopefully_enough_numbers:
            hopefully_enough_numbers = range(hopefully_big_enough, hopefully_big_enough + 1000)
            hopefully_big_enough += 1000
            for prime in first_100_primes:
                for i, n in hopefully_enough_numbers[:]:
                    if n % prime == 0:
                        del hopefully_enough_numbers[i]
    

    But obviously, you wouldn't do it like that; you'd manually simulate the infinite list, and manually pipeline all of the filters:

    first_100_primes = []
    next_number = 2
    while len(first_100_primes) < 100:
        first_100_primes.append(next_number)
        def is_prime(n, primes):
            for prime in primes:
                if n % prime == 0:
                    return False
            return True
        def get_next_relatively_prime(n, primes):
            while True:
                n += 1
                if is_prime(n, primes):
                    return n
        next_number = get_next_relatively_prime(next_number, first_100_primes)
    

    The Haskell-y version is obviously a lot easier to read. Plus, it lets you laugh at the smug Haskell types by proving that your language can do everything theirs can, at which point they have to resort to talking about the rich type system and you win, because of Guido's Law, which is like Godwin's Law, but with monads in place of Hitler. (I know, anyone with dual PhDs in type theory and history knows that Hitler is more like an applicative than a monad, but just as practicality beats purity, a heavy klomp beats a PhD.)

    But Haskellites aren't the only smug types out there. Coroutines are cool too, and anyone who's still programming with those old-fashioned subroutines is so last century. Python generators give you all the benefits of coroutines, but with a syntax that means you don't have to realize it when it's not important.

    And if you think about your iterator-transforming pipeline, that's what you're doing. At each step, each filter suspends and hands things off to the next filter until you get to the end of the chain and come back to the loop. You create a new coroutine and add it to that chain just by writing "numbers = filter(relatively_prime, numbers)". Who yields to who doesn't need to be specified—not because of some deep magic, but because it's obvious.

    Of course we all know that generators aren't _really_ coroutines. But as of PEP 342 (Python 2.5, 2005), Python generators aren't just generators. Academics continue to refer to "generators" as "what Python had before 2.5", meaning you have to say "enhanced generators" or "PEP 342 generators" to mean what every Python user just calls "generators". And "enhanced generators" _are_ coroutines. But they _still_ have that nice sequential-subroutine-style syntax as generators. You still don't have to think about them as coroutines when it isn't relevant, but you can easily do so when it is. (The lack of "yield from" meant you sometimes didn't get that advantage, but that was fixed with PEP 380 in 3.3 in 2012.)

    And that leads to even cooler things. The two obvious ways to write async I/O are with threads (green/micro or otherwise), which leads to synchronization problems they bring, or with callbacks, which forces you to think of the flow of control backward in complex cases, and leads to callback-hell even in simple cases. You can improve things a little by atomic data structures like queue.queue for all inter-thread communication, or by using promises (deferreds) for chaining callbacks, but still, you're adding a lot of mental burden on top of the equivalent synchronous code. But with explicit coroutines as in PEP 3156, you don't have any of that burden. The synchronous semantics translate directly to asynchronous semantics just by putting "yield from" at all suspension points, and you're done. The resulting code is clearly readable and easily writable.
    2

    View comments

  5. Let's say you've got the list [-10, 3, -2, 14, 5], and you want the list filtered to just the non-negative values. You could do this by mutating the list:

    for i, e in enumerate(lst[:]):
        if e < 0:
            del lst[i]
    

    Or you could do it by making a new list:

    newlst = []
    for e in lst:
        if e >= 0:
            newlst.append(e)
    return newlst
    

    But you can always transform a "l=[] … l.append … return l" into a list comprehension:

    return [e for e in lst if e >= 0]
    

    Or, in many cases, a call to map, filter, or something out of itertools:

    return filter(lambda e: e >= 0, lst)
    

    But there's no equivalent way to simplify the mutating version. You have to write an explicit for loop. You also have to make a copy of the list, because you can't change the shape of a collection while iterating over it (this wouldn't be the case for a map-equivalent that just changed some of the values without inserting or deleting any). And you have to use enumerate and explicitly refer to lst[i] to change anything, rather than just referring to e.

    There are a few shortcuts for mutating lists, such as remove, but nothing flexible or general.

    And you can, of course, abuse the non-mutating tools in some cases, but there are also many cases where this doesn't work, and it's clear that this is considered abuse even when it does work. (Consider the fact that map and filter now return iterators instead of lists, which means if you want to abuse them for their side-effects, nothing actually happens, unless you borrow the consume function from the itertools docs recipes and pass the iterator into a deque(maxlen=0) or something equivalent.)

    It's not that these operations would be hard to define, explain, or implement. Look at C++, which has matched pairs for everything. Instead of map and filter, they have transform and remove_if_copy—and, right alongside those, they have for_each and remove_if, which mutate in place. In fact, just about every algorithm in <algorithm> has an in-place variant.

    Of course you could point out that C++ is designed for people who want programming to be difficult. After all, they call all of the non-mutating algorithms "modifying", and some but not all of the mutating-in-place algorithms "non-modifying". And sometimes the matched pairs are "foo" vs. "foo_copy", other times "inplace_foo" vs. "foo", and other times completely unrelated names like "for_each" vs. "transform".

    But the point is, Python could easily have a map_inplace or for_each, extend replace with replace_if, or allow one-liner list modifiers using comprehension-style syntax. And it doesn't. Why?

    Well, first, there's historical reasons. Python borrowed map, filter, and friends from mostly-immutable mostly-functional languages, and list comprehensions from a purely-immutable pure-functional language. The kind of people who wanted to use them didn't want to mutate anything. Meanwhile, C++, borrowed the same ideas from the same sources but redesigned them to fit into a more C-ish imperative-mutating style of programming, but pretty much nobody used them, so nobody was going to borrow them into other languages.

    But after years of use, list comprehensions, generator expressions, map, etc. have become common, idiomatic Python that even novices understand—even though those novices mostly think of mutating algorithms first as a matter of course. So, why don't they all demand mutating equivalents? Why didn't Python 3 include them?

    There's a conservative explanation. Look at the two functions at the top of the page. The first one does take 3 lines to do something that could be done in 1, but they're all very simple lines that say exactly what they do, and there's really nothing that gets in the way of reading the actual logic. The second one, however, is half boilerplate. Only half of it says what it does, and the other half is unimportant details about how it does it. Reducing line count for its own sake is not worth changing syntax for; reducing boilerplate to make code more readable is.

    There's also a more radical explanation. If you look at the major changes in Python 3, other than the Unicode stuff, most of them are about replacing list-based code with iterator-based code. And there's a good reason for this: When you write everything in terms of iterator transformations, you can pipeline functions value-by-value instead of collection-by-collection. And that turns out to be pretty cool, for a number of reasons, enough to deserve its own separate post. And that's enough to add tools to help the non-mutating style.
    2

    View comments

Blog Archive
About Me
About Me
Loading
Dynamic Views theme. Powered by Blogger. Report Abuse.