1. On the Python-ideas list, in yet another thread on a way to embed statements in expressions, I raised the issue that the statement-expression distinction, and not having a way to escape it, is important to why Python is so readable. But I couldn't explain exactly why.

    The question ultimately comes down to: why have statements in the first place? After all, there's no reason you can't make (almost) everything an expression, even in an imperative language (Ruby and CoffeeScript do so), and use significant indentation within expressions (again, CoffeeScript does so).

    Guido's answer

    Hm... Practically every language I knew before I designed Python had this distinction built right into the grammar and other assumptions: Algol-60, Fortran, Pascal, C, ABC. Even Basic. I was aware of the alternative design choice: Algol-68 had statements-as-expression, and Lisp of course -- but I wasn't a big Lisp fan, and in Algol-68 it was largely a curiosity for people who wanted to write extra-terse code (also, IIRC the prevailing custom was to stick to a more conservative coding style which was derived from Algol-60). 
    So it's hard to say to what extent this was a conscious choice and to what extent it was just tradition. But there's nothing necessarily wrong with tradition (up to a point). I think it still makes sense that statements are laid out vertically while expressions are laid out horizontally. Come to think of it, mathematics uses a similar convention -- a formula is laid out (primarily) horizontally, while a sequence of formulas (like a proof or a set of axioms) is laid out vertically. 
    I think several Zen items apply: readability counts, and flat is better than nested. There is a lot to be said for the readability that is the result of the constraints of the blackboard or the page. (And this reminds me of how infuriating it is to me when this is violated -- e.g. 2up text in a PDF that's too tall to fit on the screen vertically, or when a dumb editor breaks lines but doesn't preserve indentation.)
    Guido's analogy with mathematical proofs is compelling. It's a large part of the design philosophy of Python that code should read like English when possible (e.g., the use of the colon is based on the way colons are used in English sentences), or like mathematics when English doesn't make sense (e.g., operator precedence).

    And ultimately, "readability counts" is the answer here. But hopefully there's a way to explain how statements help readability, that gets to the actual fact behind Guido's analogy, and behind the intuition behind the tradition.

    Ron Adam's followup

    Expressions evaluate in unique name spaces, while statements generally do
    not. Consider "a + b"; it is evaluated in a private method after the values
    a and b are passed to it. 
    Statements are used to mutate the current name space, while expressions
    generally do not. 
    Statements can alter control flow, while expressions generally do not. 
    Having a clear distinction between expressions and statements makes reading
    and understanding code much easier.

    I'm not as sure about Ron's first three points. For example, in Python, "a[0] = 2" is evaluated by calling an a.__setitem__ method with the values a, 0, and 2 passed to it, exactly as "a + b" is evaluated by calling an a.__add__ method with the values a and b passed to it. And C++ takes this farther, where even normal assignment is a method call, and Ruby takes it even farther, where a for loop is evaluated by passing the object being looped over and the proc to call on each element to a method. So, that only requires a handful of things to be statements, like break, continue, and return (the very things that are statements in Ruby and CoffeeScript).

    But his last point, that's the whole crux of the matter: I think having a clear distinction does make reading and understanding code easier, and it's a large part of why idiomatic Python code is more readable than Ruby or CoffeeScript code. (Guido raised another point earlier about CoffeeScript: its grammar is basically not defined at all, except in terms of translation rules to JavaScript, which means it's impossible to hold the syntax in your head. And I agree—but not many other languages suffer from that problem, and yet they're less readable than Python.) The question is why it makes reading and understanding code easier.

    My thoughts

    Actually, I don't think having the distinction does necessarily make reading code easier.  It enables a language to make reading code easier, but depending on how it makes other choices, it may not get that benefit. After all, JavaScript is less readable than CoffeeScript, and I think Java and C++ would both gain from being able to do more in expressions, but Python wouldn't.

    What's the difference? Partly the fact that compound statements are indentation-driven rather than brace-driven, but standard style guidelines for other languages (which the vast majority of code follows) already recommend that. And partly the fact that terse forms (like comprehensions) are only for purely declarative code, but those other languages don't have any corresponding forms in the first place.

    The big difference is that mutating functions (idiomatically) don't return the mutated object. This means that fluent style code is impossible to write in Python, which comes at a cost. More generally, Python is not a great language for writing some kinds of complex expressions that other languages can handle easily. But it also provides a benefit. When combined with the two preceding features, and the statement-expression divide (and a well-designed language and set of idioms) you get two things that are nearly unique to Python:
    • Flow control is immediately visible by scanning the code, because it's represented by indentation levels and very little else is.
    • State mutation is almost immediately visible by scanning the code, because each line of code usually represents exactly one mutation, and usually to the thing on the left.
    That's a lot of important information to be able to pick up without thoroughly reading the code and thinking through it. It often lets you quickly find the part that you do need to read thoroughly and think about, just by skimming the rest.

    This implies that there might be different ways to make languages readable than the way Guido came up with. And to some extent, I think Ruby and CoffeeScript are both examples of steps in one possible direction. But, even though they've broken with tradition more than Python has, they haven't gotten as far yet. The real question is whether you can find a way to make functional-style code as nice as it is in Ruby (or even ML or Haskell) while making imperative/OO-style code as nice as it is in Python. I think that would require some big clever new idea nobody's come up with yet, not just tweaking the basic designs of Ruby and Python, but maybe I'm wrong.

    Further thoughts

    Earlier in the thread, Franklin Lee pointed to a great blog post by Guido, Language Design is Not Just Solving Puzzles. In that post, Guido's main point is that attempting to "solve the puzzle" of how to embed statements into expressions in Python is wrong-headed. And he's got an interesting argument.

    I had suggested that the inability to escape the division was important for reasons beyond just keeping the parser simpler, but Guido suggests that keeping the parser simpler is already enough reason. Anything that's complicated for the compiler to parse is also likely to be too complicated for a human to parse instinctively; if you have to stop and think about the syntax, it gets in the way of you reading the code.

    C and its derivatives provide a perfect example of what he means. The declaration syntax is complicated enough that there are interview questions and online puzzles asking you to parse your way through this and explain what the type is:*
        const char **(*)(const char *(*)(const char *), const char **)
    
    Or, in C++, to explain why this function doesn't construct a list named lines:**
        list<string> lines(istream_iterator<string>(cin), istream_iterator<string>());
    
    Anyway, Guido suggests that parsing Python is essentially modal: there's an indent-sensitive statement-reading mode, and a non-indent-sensitive expression-reading mode. You can embed expressions in statements, but not the other way around, so you basically just need a mode flag rather than a stack of modes. And it's not that the stack would be too complicated to code into a parser, it's that it would be too complicated to fit into a reader's intuition. Anything extra you have to keep in mind while reading is something that gets in the way of your reading. (He puts it better than me, and not just because he devotes a whole post to it rather than just a couple of paragraphs, so go read it.)

    Elsewhere in the thread, Guido also raised the point that being able to hold the syntax in your head is important, pointing out that part of the problem with understanding CoffeeScript is the fact that you can't possibly understand the syntax because it isn't even defined anywhere except as a set of translation rules to JavaScript.

    I think he's right about both these points. And if you combine them with the fact that statements can be (and, in Python, are) used to make code more skimmable, that explains why being able to embed a statement in an expression is probably a losing proposition.


    * It's a pointer to a string-specific version of the map function. C represents strings as const char *, so const char ** is an array of strings. The parentheses around the next asterisk are necessary to make it part of the whole declaration rather than part of the return type, so we're declaring a pointer to a function that returns an array of strings. The first parameter type is similarly a pointer to a function returning a string, and taking a string, and the second parameter type is an array of strings. It may be easier to read in SML syntax (or, really, almost any other syntax…): (string -> string) * (string list) -> string list.

    ** In C, some constructions can be parsed as either an expression or a declaration, both of which are valid statements; C resolves this by treating any such ambiguous constructions as declarations. That doesn't come up too often in C, but in C++, it does all the time. Here, we're trying to construct a list<string> by calling the list constructor with a pair of begin and end iterators. For the begin, we're passing istream_iterator<string>(cin), which iterates lines off standard input. For the end, we're passing istream_iterator<string>(), which is the default end iterator of the same type. But istream_iterator<string>() can also be read as either a type—a function that takes nothing and returns an iterator—or an expression—a call of the constructor. So now we have to see whether the rest of the statement can be read as a declaration to know which one it is. istream_iterator<string>(cin) can be read as a type followed by an identifier (in unnecessary, but legal, parentheses). Which means the whole thing can be read as a function declaration: lines takes an iterator named cin, and a function that returns an iterator with no parameter name, and returns a list.
    3

    View comments

  2. In IEEE Floats and Python, I chose to use a tiny "binary6" type because it's easy to show a table of values that helps clarify things.

    Someone suggested that you could just as easily write a table of binary64 values, ellipsizing large chunks of it, and it would also be useful for clarifying things. And I think he's right.

    The Table

        sign  exponent significand    binary scientific              decimal
           0 000...000 0000...0000    0.0...0000e-1022 = 0           0.0
           0 000...000 0000...0001    0.0...0001e-1022 = 1.0e-1074   5e-324
           0 000...000 0000...0010    0.0...0010e-1022 = 1.0e-1073   1e-323
           0 000...000 0000...0011    0.0...0011e-1022 = 1.1e-1073   1.5e-323
           0 000...000 0000...0100    0.0...0100e-1022 = 1.0e-1072   2e-323
           0 000...000 0000...0101    0.0...0101e-1022 = 1.01e-1072  2.5e-323
           0 000...000 0000...0110    0.0...0110e-1022 = 1.10e-1072  3e-323
           0 000...000 0000...0111    0.0...0111e-1022 = 1.11e-1072  3.5e-323
           0 000...000 0000...1100    0.0...1100e-1022 = 1.0e-1071   4e-323
           0 000...000 0000...1101    0.0...1101e-1022 = 1.01e-1071  4.4e-323
           0 000...000 0000...1110    0.0...1110e-1022 = 1.10e-1072  5e-323
           0 000...000 0000...1111    0.0...1111e-1022 = 1.11e-1072  5.4e-323
           ...
           0 000...000 1111...1110    0.1...1110e-1022 = 1.11e-1023  2.2250738585072004e-308
           0 000...000 1111...1111    0.1...1111e-1022 = 1.11e-1023  2.225073858507201e-308
           0 000...001 0000...0000    1.0...0000e-1022 = 1.0e-1022   2.2250738585072014e-308
           0 000...001 0000...0001    1.0...0001e-1022               2.225073858507202e-308
           ...
           0 000...001 1111...1111    1.1...1111e-1022               4.4501477170144023e-308
           0 000...010 0000...0000    1.0...0000e-1021               4.450147717014403e-308
           ...
           0 011...110 1111...1111    1.1...111e-1                   0.9999999999999999
           0 011...111 0000...0000    1.0...000e0                    1.0
           0 011...111 0000...0001    1.0...001e0                    1.0000000000000002
           ...
           0 111...101 1111...1111    1.1...111e+1022                8.988465674311579e+307
           0 111...110 0000...0000    1.0...000e+1023                8.98846567431158e+307
           0 111...110 0000...0001    1.0...001e+1023                8.988465674311582e+307
           ...
           0 111...110 1111...1110    1.1...110e+1023                1.7976931348623155e+308
           0 111...110 1111...1111    1.1...111e+1023                1.7976931348623157e+308
           0 111...111 0000...0000    inf                            inf
           0 111...111 0000...0001    snan1                          snan1
           0 111...111 0000...0010    snan10                         snan2
           ...
           0 111...111 0111...1111    snan1...1                      snan2251799813685247
           0 111...111 1000...0000    qnan0                          qnan0
           0 111...111 1000...0001    qnan1                          qnan1
           0 111...111 1000...0010    qnan10                         qnan2
           0 111...111 1111...1111    qnan1...1                      qnan2251799813685247
           1 000...000 0000...0000    -0.0...0000e-1022 = -0         -0.0
           0 000...000 0000...0001    -0.0...0001e-1022 = -1.0e-1074 -5e-324
           ...
           1 111...111 0000...0000    -inf                           -inf
           1 111...111 0000...0001    -snan1                         -snan1
           ...
           1 111...111 1000...0000    -qnan0                         -qnan0
           1 111...111 1111...1111    -qnan1...1                     -qnan2251799813685247
    

    Notes

    For finite numbers, I've displayed the decimal representation the same way Python 3.5 does (the shorted decimal fraction that rounds to the same binary fraction). For NaN values, Python displays them all as just "nan". Some C libraries will display something like "QNaN(1)" or "qnan1" or the like, so I've chosen to do that for clarity.

    As mentioned in the previous post, this is for IEEE 754-2008 binary64, even though your platform (assuming you're not reading this in some far-future era where C11 and Python 3.5 are distant memories but blogspot posts are still relevant) probably uses the older IEEE 754-1985 double type, which is a slightly looser version of the same thing. So if you're on some older platform, a few things may be different (e.g., Irix on MIPS has snan and qnan reversed).

    Generating the table yourself

    Using the floatextras module, you can generate the values yourself. If it's not quite as easy as maybe it could be, suggestions are welcome.

    One trick that helps a bit is that the make_tuple function can take any sequence of digits, or an int, which will be interpreted as an unsigned 52-bit integer:
        >>> from_tuple((0, -1, 0))
        1.9999999999999998
    
    Unfortunately, you still have to turn that into bits to print the significand column of the table. I guess you could recover them from the float with as_tuple, but that seems silly. (I used the bitstring library again.) Maybe a function for generating table rows given either a float or a tuple and a +/- range would be useful?
    1

    View comments

  3. Everybody knows that "floating point numbers cause problems." Many people have misconceptions about what those problems are, and how to deal with them. There are some great references for people who want to understand things, the (justly) most famous being What Every Computer Scientist Should Know About Floating-Point Arithmetic by David Goldberg.

    But many Python programmers like to learn by experimenting. That's why you've chosen a language with an interactive REPL, powerful introspection, and, when it comes down to it, a readable open-source implementation, right?

    So, let's look at how floating point numbers are represented in Python; understanding what all the bits mean may make it easier to understand how rounding errors and so forth work.

    On nearly any platform you care about, floating point numbers—both binary (float) and decimal (Decimal)—are going to be stored in IEEE 754-2008 formats. So, let's look at how those work.

    IEEE binary floats

    Scientific notation

    IEEE floats are based on scientific notation, but in binary.

    Scientific notation looks like:
    -1.042e3
    The "-" sign is obvious. The "1.042" is almost obvious, but note that in "normalized form" there always must be exactly one nonzero digit left of the "." (that's what makes it scientific notation), and you're allowed to have trailing zeros (if, e.g., you had 11 to 4 significant digits, you'd write 1.100e1, not 1.1e1). The "e3" means that you multiple by 10**3.

    So:
    - (1 + 0/10 + 4/100 + 2/1000) * 10**3 == -104.2
    For binary scientific notation, everything works the same way, except with 2 in place of 10. So:
    -1.01e2
    - (1 + 0/2 + 1/4) * 2**2 = -101 (binary) or -5 (decimal) 

    Representation in bits

    Obviously, any float based on scientific notation is going to have three things to store: the sign, the mantissa (the decimal 1.042 or binary 1.01), and the exponent (the 3 in 10**3 or 2 in 2**2).

    The standard doesn't really specify how the bits have to be stored inside the computer, but they have to be represented for interchange as if they were stored as:
    • 1 sign bit
    • w exponent bits
    • p-1 significand bits
    Let's pick a specific, very small, size to make our lives easy: just 3 exponent bits and 2 significand bits. Of course in real life you'd never deal with 6-bit floats, but having only 64 possible values means we can look at an exhaustive table of them (which I've included below), which can be very helpful.

    We'll call this 6-bit format "binary6", even though technically it isn't a legal interchange format. (For one thing, interchange formats must have either 16 bits, or a multiple of 32 other than 96. For another, a k-bit interchange format for any k but 16 and 32 must have w = round(4 * math.log2(k)) − 13, which in our case would give -3 bits.)

    For the binary64 format, everything is exactly the same as our binary6 format, except that w is 11 instead of 3, and p-1 is 52 instead of 2.

    Sign

    The sign is easy. Unlike integers, there's no 2s complement or anything fancy going on here. You compute the number from the exponent and significand, and then if the sign bit is 1, you negate it. So, -5.0 looks exactly like 5.0 except for the first bit. This does have some strange consequences—it means -0.0 and +0.0 are different numbers, and it means the bits for the smallest negative number and the smallest positive number are very far apart—but it also makes a lot of things easier. So, in our example of -1.01e2, the sign bit is 1.

    Mantissa and significand

    Next, we've got the mantissa, the 1.01 part. But notice that we don't have 3 mantissa bits, we have 2 significand bits. How does this work? Well, in scientific notation, there's always exactly one non-zero digit to the left of the decimal point. In binary, the only non-zero digit is 1, so the format saves a bit by not storing that 1.

    So, for 2 bits, the possible values are 1.00 (stored as 00), 1.01 (as 01), 1.10 (as 10), and 1.11 (as 11).

    Exponent

    Finally, we've got the exponent of 2. To handle the possibility of negative exponents, instead of storing the exponent itself, we store the biased exponent—that is, we add 2**(w-1)-1. Also, the smallest and largest values are reserved for special purposes, which we'll get to in a moment. For 3 bits, the possible values are special (stored as 000), -2 (as 001), -1 (as 010), 0 (as 011), 1 (as 100), 2 (as 101), 3 (as 110), and special (as 111).

    So, our number, -1.01e2, is stored as 1 (sign) 101 (exponent) 01 (significand), or 110101.

    Infinity, NaN, and subnormals

    The smallest exponent value is used to store 0. After all, if there were no special rule, 0 000 00 would mean +1.00 * 2**-3, which is obviously not 0. But just saying exponent 0 means the value is 0 is a waste of bits—that would mean we have 8 different zero values (0 000 00, 0 000 01, 0 000 10, 0 000 11, 1 000 00, 1 000 01, 1 000 10, and 1 000 11). Having positive and negative zero can sometimes be useful, but having 4 different positive zeros is pointless.

    So, instead, a 000 exponent actually means the same thing as a 001 exponent, except that a 0 instead of a 1 is prepended to the significand to get the mantissa. A number like 0.01e-2 isn't properly normalized scientific notation—that would be 1.00e-4. So, these numbers are called "subnormal" or "denormal".

    Meanwhile, the largest exponent value is used to store infinity. Again, though, that would give us 8 infinities when we only need 2. So, if the significand is nonzero, instead of infinity, it's NaN. The first bit of the significand is the quiet bit, and the remaining bits (in this case, just 1) are the payload, which is reserved for anything a user-level library wants.

    Table

    So, in our 6-bit format, here's what all the possible numbers mean:

        sign exponent significand    binary scientific  decimal
           0      000          00    0.00e-2 = 0        0.0
           0      000          01    0.01e-2 = 1e-4     0.0625
           0      000          10    0.10e-2 = 1e-3     0.125
           0      000          11    0.11e-2 = 1.1e-3   0.1875
           0      001          00    1.00e-2            0.25
           0      001          01    1.01e-2            0.3125
           0      001          10    1.10e-2            0.375
           0      001          11    1.11e-2            0.4375
           0      010          00    1.00e-1            0.5
           0      010          01    1.01e-1            0.625
           0      010          10    1.10e-1            0.75
           0      010          11    1.11e-1            0.875
           0      011          00    1.00e0             1.
           0      011          01    1.01e0             1.25
           0      011          10    1.10e0             1.5
           0      011          11    1.11e0             1.75
           0      100          00    1.00e1             2.
           0      100          01    1.01e1             2.5
           0      100          10    1.10e1             3.
           0      100          11    1.11e1             3.5
           0      101          00    1.00e2             4.
           0      101          01    1.01e2             5.
           0      101          10    1.10e2             6.
           0      101          11    1.11e2             7
           0      110          00    1.00e3             8.
           0      110          01    1.01e3             10.
           0      110          10    1.10e3             12.
           0      110          11    1.11e3             14.
           0      111          00    inf                inf
           0      111          01    snan1              snan1
           0      111          10    qnan0              qnan0
           0      111          11    qnan1              qnan1
           1      000          00    -0.00e-2 = -0      -0.0
           1      000          01    -0.01e-2 = -1e-4   -0.0625
           1      000          10    -0.10e-2 = -1e-3   -0.125
           1      000          11    -0.11e-2 = -1.1e-3 -0.1875
           1      001          00    -1.00e-2           -0.25
           1      001          01    -1.01e-2           -0.3125
           1      001          10    -1.10e-2           -0.375
           1      001          11    -1.11e-2           -0.4375
           1      010          00    -1.00e-1           -0.5
         ...
           1      110          00    1.00e3             -8.
           1      110          01    1.01e3             -10.
           1      110          10    1.10e3             -12.
           1      110          11    1.11e3             -14.
           1      111          00    -inf               -inf
           1      111          01    -snan1             -snan1
           1      111          10    -qnan0             -qnan0
           1      111          11    -qnan1             -qnan1
    

    Rounding

    Given the table above, rounding errors should be pretty easy to understand.

    If you want to store 1.1 as a binary6 float, the closest value is 1.0 (0 100 00), which is off by 0.1; the next best would be 1.25 (0 100 01), which is off by even more 0.15.

    Now, let's say you want to test 0.6 / 4 == 0.15. The closest binary6 values to start with are 0.625 and 4. Divide those, and the closest result is 0.1875. But the closest value to 0.15 is 0.125, and 0.1875 != 0.125.

    You can work out the same thing with more familiar examples with binary64, like 0.3 / 3 == 0.1.

    Reading floats as ints

    Notice that, as long as you stick to finite nonnegative numbers, you can treat the exact same bits as if they were integers, and two adjacent floats will always be adjacent integers.

    For example, Python doesn't provide a function like C99/POSIX-2001 nextafter, but it's easy to implement yourself. For a quick&dirty case, if you know you're dealing with finite positive numbers, just add or subtract 1 as an integer, and you've got the next or previous float. Dealing with negative numbers and nonfinite numbers gets a bit trickier, but it should be obvious how to do it (once you choose exactly what you want to do for, e.g., nextafter(-0) and nextbefore(inf)).

    Similarly, to get the "float difference" in ulps (Units of Least Precision, or Units in the Last Place), just subtract the two floats as integers (again, with the obvious extensions for negative and nonfinite numbers). Note that this is often very different from the delta, the difference in real numbers. For example, 1.25 is 2 ulp away from both 0.875 and 1.75, but the deltas are not the same (0.375 and 0.5, or relative 30% and 40%).

    You can even begin to understand tricks like the fast inverse sqrt hack.

    Manipulating float bits in Python

    OK, now we know how a float is stored. But how can we play with this in Python?

    While it's possible (at least in CPython) to get at the values under the covers via ctypes, it's generally simpler (and a lot safer) to just use the struct module to pack a float in big-endian C double format. Then we've got the bits as a sequence of bytes—or, if you prefer, you can turn it into a 64-bit unsigned integer, or use a third-party library like bitarray or bitstring to make your life a whole lot easier.

    So, let's look at how -123.456 is stored as a float:
        >>> f = -123.456
        >>> b = bitstring.pack('>d', f)
        >>> sbit, wbits, pbits = b[:1], b[1:12], b[12:]
        >>> sbit.bin
        '1'
    
    That's easy: sign bit 1 means negative.
        >>> wbits.bin
        '10000000101'
        >>> wbits.uint - (1<<10) + 1
        6
    
    So, the exponent is (decimal) 6.
        >>> pbits.bin
        '1110110111010010111100011010100111111011111001110111'
        >>> 1 + pbits.uint / (1<<52)
        1.929
    
    So, the mantissa is (decimal) 1.929.

    So, the number is -1.929 * 2**6. Which checks out:
        >>> -1.929 * 2**6
        -123.456
    
    Or, if you prefer:
        >>> pbits.hex
        'edd2f1a9fbe77'
    
    So, in Python's float hex format, the number is -1.edd2f1a9fbe77p+6, which we can check:
        >>> f.hex()
        '-0x1.edd2f1a9fbe77p+6'
    
    You can also use this to create and parse NaN payloads, which Python doesn't do natively:
        >>> quiet, payload = 1, 3
        >>> bits = (0b11111111111 << 52) | (quiet << 51) | payload
        >>> qnan3 = struct.unpack('>d', struct.pack('>Q', bits))[0]
    
    Of course Python is going to print qnan3 as just plain "nan", but if your platform has some way of printing out non-default NaN values, you may be able to see it like this:
        >>> ctypes.printf(b'%f', ctypes.c_double(qnan3))
        QNaN3
    
    And if you want to do the nextafter or float_difference hacks described in the last section:
        >>> f = 1.1
        >>> d = struct.unpack('>Q', struct.pack('>d', f))[0]
        >>> struct.unpack('>d', struct.pack('>Q', d-1))[0]
        1.0999999999999999
        >>> struct.unpack('>d', struct.pack('>Q', d+1))[0]
        1.1000000000000003
    

    The floatextras package

    Now that you know how to do this all manually, the floatextras package on PyPI wraps it all up for you:
        >>> f = 1.1
        >>> floatextras.next_plus(f)
        1.1000000000000003
        >>> floatextras.float_difference(1.0999999999999999, 1.1000000000000003)
        -2
        >>> floatextras.as_tuple(-123.456)
        FloatTuple(sign=1, digits=(1, 1, 1, 0, 1, 1, 0, 1, 1, 1, 0, 1, 0, 0, 1, 0, 1, 1, 1, 1, 0, 0, 0, 1, 1, 0, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1, 1), exponent=6)
        >>> floatextras.as_tuple(qnan3)
        FloatTuple(sign=0, digits=(1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1), exponent=1024)
    

    Caveat for some platforms

    What I described above is all true for the current IEEE 754-2008 standard, but most platforms are actually based on the older IEEE 754-1987. There are a number of differences and expansions in areas I didn't get to above, and some of the terminology is different (e.g., "double" instead of "binary64").

    Also, there are also a few things that the older standard left open to the implementation, which I've assumed will always be interpreted the same way. If you're on almost any platform on x86, x86_64, or ARM hardware, everything will be as above; on different hardware, there may be minor differences (e.g., Irix for MIPS used the NaN quiet/signaling bit backward, so 0 means quiet; another platform which I've mercifully forgotten used the last bit of the significand rather than the first; POWER stored floats big-endian even in little-endian mode so you had to flip the bits around to do the "treat them as ints" trick).

    Decimal floats

    There's no reason floats have to be stored in binary. In fact, IEEE 754-2008 specifies how you can use the same basic ideas with decimal fractions instead of binary. And Python includes an implementation of this (well, of its predecessor, IEEE 854-1987, but again, it's close enough) in the decimal module.

    Most of the differences between binary and decimal formats are obvious, except for one. Decimals can't use the "free mantissa bit" digit, because the digit left of the decimal point can be anything from 1 through 9, so you have to store it. This also means decimal floats can have non-canonical representations (after all, 0.1e2 means the same thing as 1.0e1). Which also means you can store subnormal values, and 0, with different levels of precision, and some operations do so.

    The decimal module docs, and the PEP linked from there, cover all of the details very nicely.

    The obvious advantage of decimal floats is that there's no rounding error converting to and from decimal strings. For example, the closest binary64 float to 1.1 is 1.10000000000000009e+0; the closest decimal float (as long as you have at least 2 digits of precision) is 1.1e+0. Magic, right? Well, no; plenty of things that are exact with binary floats are inexact with decimal floats. Still, decimals are often a good solution to many floating-point problems, as long as you understand how they fall short of magic. Which, hopefully, you now do.

    Also, because the decimal module is implemented from scratch, rather than relying on a native C type, it provides a lot more flexibility. Instead of being stuck with a decimal64 format, you can define and use any format. You can specify rounding modes and signaling modes. All of the optional APIs from the standard are available, instead of just the ones C/POSIX/your platform chose to implement, plus some extras to do things like picking a float apart into sign, exponent, and mantissa, or building NaN values with custom payloads.
    0

    Add a comment

  4. Many tutorials on object-oriented programming conflate inheritance and subtyping. In fact, it's often considered part of the OO dogma that they should be conflated.

    This is wrong for Python in a wide variety of ways.

    Substitutability

    Let's start with the Liskov Substitution Principle: A is a subtype of B iff instances of type A can substitute for instances of type B without affecting correctness or semantics. That's what it means to be a subtype.*

    * Three things to note here, without getting into details (yet). First, mutability makes this more complicated than most people think at first—see the section on the ellipse-circle problem in the linked article. Second, languages like Python that have more complicated signatures—optional parameters, keyword-only parameters, vararg parameters, etc.—are harder to define rigorously, but all of those cases are easy to understand intuitively (e.g., a method that takes two integers can be overridden by a method that takes two integers and an optional string, because any call to the supertype's method will just passing two integers, which is valid for the subtype's method). Third, Python makes extensive use of what are implicitly union types and parametric types, but has no obvious way to represent them explicitly—a strstr-like function that returns None if not found is of type (str, str)->(int | NoneType); a read4k function could be described as of type (TextIOBase->str) | (IOBase->bytes) or of type (IOBase<T> -> String<T>). The MyPy-based static typing syntax for Python 3.5 or 3.6 will make some of this representable, but it's still not going to be completely parametric in the sense of ML or Haskell.

    Nominal vs. structural substitutability

    In many modern OO languages, only subclasses can be subtypes (and all subclasses should be subtypes, but this is usually only partly enforced). If I write a function that takes a File object, so I can read from it a line at a time (or 4K at a time), you can only call my function with something that inherits from File.*

    In Python, this not only isn't necessary, it isn't idiomatic. If I write a function that wants to read a line at a time, I document that I want a file-like object that iterates a line at a time, and I don't verify that what I've gotten is-a file in any way, I just try to iterate it and expect to get strings. You can pass me an io.StringIO (which is basically a TextIOWrapper that isn't wrapping a file), or just a list of strings, or some type from a library that knew nothing about my library and vice-versa, and everything still works.

    That's the difference between nominal and structural** (or behavioral***) subtyping, and that difference is half of what duck typing is all about. I'll get to the other half in a bit.

    * A few languages (like C++) make things more complicated by allowing coercive typing: any type A that can be implicitly coerced to type B is a subtype of A, at least in some contexts… But let's ignore that.

    ** Structural subtyping can also include layout. In Python, since you generally store attributes in a dict rather than at specific offsets, layout is irrelevant beyond the names and types of the attributes. (Except, of course, that "generally" doesn't mean "always"; you can use __slots__, or inherit from ctypes.Structure, or write an extension module, or use nothing but descriptors in the class dictionary, or…) Also, in Python, classes generally don't even directly describe the attributes of their instances; they're created dynamically in __init__ (or even later), so checking that something is a subtype isn't even a sensible question except dynamically.

    *** Behavioral subtyping has two different meanings: it can mean subtyping relationships that include semantics beyond those normally discussed in type theory, but it can also mean ad-hoc structural subtyping of exactly the kind Python uses.

    What about isinstance?

    But hold on. If I subtype without subclassing in Python, the result isn't really substitutable, because there's nothing stopping a function from calling isinstance and detecting that I'm not really what I'm claiming to be, right?

    Well, that's true. But it's also true that even if I do subclass, there's nothing stopping a function from calling type (or looking at my __class__ attribute, or using the inspect module) and detecting that I'm not really what I'm claiming to be.

    This is why idioms and conventions are important. In Java or C++, switching on isinstance-like checks (e.g., C++ dynamic_cast) is perfectly acceptable, because it gives you the exact same results that virtual function dispatch would give you. Switching on the exact type is strongly discouraged, because it effectively makes subtyping impossible.

    In Python, isinstance is considered better than type, but still to be avoided whenever possible, precisely because it doesn't give you the same results that dynamic method lookup would give you, and it effectively makes subtyping impossible.

    ABCs: practicality beats purity

    So, structural subtyping good, nominal subtyping bad… But there really are some cases where it's useful to be able to check whether A is a subtype of B (this is just a specific case of "there are some cases where LBYL is more useful than EAFP"), and how else are you going to do that but isinstance?

    But in most of those cases, you should use an abstract base class, not a normal base class. Don't check isinstance(arg, list), check isinstance(arg, collections.abc.MutableSequence). Note that this is exactly the same as using interfaces vs. base classes in Java or C#—the difference being that in Java, you always have to define an interface, while in Python you always can define an ABC, but usually don't have to.

    But notice that ABC's don't require inheritance for subclassing. In fact, there are three ways to make a class a subclass of an ABC:
    • In the subclass's implementation, by inheriting from the ABC.
    • In the ABC's implementation, by writing a subclass hook that returns True for the subclass. (Note that this is usually done to implement structural subtyping, but it can just as easily be used to supertype a specific existing class that you can't modify.)
    • Outside of both implementations, in an application that just wants to make the two play together nicely, by registering the class with the ABC.
    And meanwhile, Python doesn't even strictly require substitutability for ABCs. A TextIOBase is-a IOBase, even though IOBase.readline returns a bytes-like object and TextIOBase.readline returns a str-like object. Why? Well, as I'll explain later in the section on mixins, IOBase is more useful as a mixin class than as a subtyping class, and it's useful for IOBase to provide a default readline implementation for any subclass that provides readinto but doesn't provide readline. Speaking from a purist point of view, this means that TextIOBase isn't a subtype of IOBase at all. But from a practical point of view, it just means that readline isn't part of the IOBase type, even though it's part of the IOBase ABC.

    What it all means

    In (purist) theory, Python doesn't allow subtyping at all (because you can always call type), but in practice, Python uses subtyping all over the place—and as long as you write reasonably Pythonic code, it works wonderfully.

    That's because duck typing, not subclassing, is the normal way to represent subtyping; inheritance is only one of three equally valid ways to implement subclassing; isinstance is used sparingly and only when you know you have ABCs to deal with; type is used only for debugging and very rare edge cases.

    To anyone coming to Python from Lisp, Smalltalk, or JavaScript, this is all pretty obvious, but people coming from Java or C# (or, worse, languages that either mix nominal and duck typing haphazardly like Ruby, or mix them in a complicated way that many people never quite figure out like C++) end up writing non-Pythonic code because they think they need subclassing all over the place, when they actually don't.

    Is this really subtyping at all?

    The reason subtyping is important in OO in the first place is that it's how OO does polymorphism—that is, functions that can work with different types. You can write a function that parses an RFC822 header out of any file-like object, and it doesn't matter whether that object is a file on disk, a transport wrapper around a socket, or a wrapper around a string that you've built up in memory.

    Above, I've described Python (and other duck-typed languages) as using implicit, dynamic, structural subtyping for its polymorphism. But there's a different way to look at it: You can just argue that Python's polymorphism isn't based on subtypes at all. Instead, its polymorphic functions are generic functions. The fact that many of those functions are written as OO-style methods and called using OO-style dot syntax doesn't necessarily mean there's any subtyping going on.

    Well, I could point out that any language whose type system is powerful enough to represent both parametric types and union types has generic functions that can fully simulate OO methods (pause here for every Haskell programmer to say "duh"), although the converse is not true (pause again), and that duck typing automatically gives you a sufficiently powerful type system except for the compile-time part (kick off an async task to deal with the Haskell programmers who are now saying "the except is the whole point"*).

    But subtyping is the most useful way to express much of what people frequently do with Python. For example, extend my RFC822 parser to an HTML handler: It reads an RFC822 header by reading line by line until it gets a blank line, then it does a binary read on the rest of the data. Sure, you _could_ describe that as taking type {__iter__: ()->Iterable<String>, detach: ()->{read: ()->Iterable<Byte>}}, but, at least for purposes of human documentation (as opposed to, say, compiler type checking or optimization), it's much more useful to say it takes any TextIOBase. Of course you could further document that it only uses part of the TextIOBase interface (which adds helpful information because that allows me to give you a duck-typed file-like object without having to implement the whole ABC), but you'd still want to start with the TextIOBase if you want humans to understand your code.

    And anyway, if you don't want to think of Python as having subtyping at all, then you're obviously not going to implement subtyping with inheritance in Python, so I have nothing to convince you of.

    * Briefly, they're forgetting the fact that both Haskell and Python—and many other dynamic languages, but no other remotely mainstream static language—allow you to write functions and data structures that are not expressible at all in ML, Java, etc. The fact that Haskell can catch errors at compile time, rather than runtime, and optimize at compile time rather than just by tracing-JIT (which works OK for time optimization, not so much for space…), etc. is definitely an extra feature on top of that, which Python doesn't have. But it doesn't add any expressiveness that Python doesn't have. (Now, a language that had a sufficiently powerful type system plus overloading might be able to express things that Python can't, but so far, C++ and Swift are the closest any mainstream languages have come to that, and they don't come very close.) Anyway, I can write further on this if anyone is interested and/or shaking their heads in anger and disbelief.

    Not necessarily beautiful, but mutated

    Getting back to the LSP, the novel idea that makes it so important is that it applies to mutable objects.

    After all, for immutable objects, subtyping is simple: A is a subtype of B if, for every attribute in B, A has an attribute with the same name whose type is a subtype. For methods, being a subtype means all arguments* are supertypes, and the return value is a subtype. There are a few complexities for more complicated signatures like those Python allows, but, while they're hard to get down rigorously, they're intuitively obvious.**

    But mutation makes everything trickier. First, for A to be a subtype of B, every directly mutable attribute must be of the same type, not a subtype, because you have to be able to assign the same values. And, while methods are obviously not usually mutable, they can be mutating, and a mutating method has to be able to mutate the object in the same way.***

    The (in)famous example here is Circle and Ellipse. One of the standard examples for OO design is that a Circle is-a Ellipse whose width and height are always the same. Any non-mutating methods on an Ellipse will work perfectly well on a Circle. But if width is mutable, or there's a method that sets the width independently from the height, a Circle no longer is-a Ellipse. (If this isn't clear, consider an immutable stretched(aspect) method that returns a new Ellipse stretched to the given aspect ratio, vs. a mutable stretch(aspect) method that stretches self to the given aspect ratio. Obviously, Circle can implement the first one just fine, but can't implement the second.****)

    For a more common real-life example, consider a sorted sequence—something that acts like a tuple, but always has all of its elements in sorted order. A sorted tuple can be do anything a tuple can do, and can be used anywhere a tuple can be used, but it can also do other things, like find all values within a certain range.***** But now consider a mutable sorted sequence—something that acts like a list, but always has all of its elements in sorted order. This is not a subtype of a mutable sequence (although it is still a subtype of sequence, and sorted sequence). The most basic mutable sequence operation is seq[idx] = value, with the postcondition that seq[idx] == value; a mutable sorted sequence can't provide that. Or consider calling seq.sort(key=different_key). In fact, almost any mutating sequence method makes no sense for a sorted sequence.

    * Except for self, of course. Or the equivalent. Which can get pretty complicated when you're dealing with multiple dispatch—including the ad hoc multiple dispatch that Python's operator protocols provide—but let's not get into that here.

    ** For example, think about optional arguments: (int, int, int=0)->int is obviously a subtype of (int, int)->int.

    *** To make that rigorous, you generally put it in terms of how subtyping handles either preconditions and postconditions, or invariants (or both), plus a history constraint.

    **** Unless you incorporate type-changing into your language's OO model. In that case, if you define Circle such that it's no longer an invariant that every mutation leave it a Circle, only that every mutation leave it an Ellipse, then a Circle is-a Ellipse again. I believe people actually have written serious code like this in Lisp, and Python allows it as well (normal classes allow you to mutate the __class__ attribute on an instance), but I don't know of anyone taking advantage of it.

    ***** Well, a tuple can do that, it just takes linear time instead of logarithmic. And, if tuple slicing returned sub-views instead of copies, a sorted tuple could also return a view for this value-slicing operation, while a regular tuple would have to return a copy. That raises the whole question of to whether, how, and/or to what extent performance guarantees are part of a type, which I definitely don't want to get into here.

    So what is inheritance for in Python?

    Subtyping

    Again, ABCs aren't completely useless for subtyping, and there are some cases where isinstance tests make sense.

    And, to add to what I said before, they help document the requirements of a group of related functions, which would be handy even if they didn't help to check those requirements. Consider the C++ standard library, and its predecessor the STL: it relies on clearly-defined "concepts" that specify a number of abstract types, to allow pieces to be connected up without being coupled together, but the language has no support for putting the concepts into the actual implementation. (It came close to getting such support in C++11, and probably will have something eventually…) The fact that Python's type system is powerful enough to represent its concepts can't possibly hurt; it means you can inherit your class from collections.abc.Sequence, or register with it, instead of just adding a comment saying "# this class acts like a sequence". It's obvious, easy to spot, and even introspectable (which can be a pretty important thing for a language that's built around experimenting with components in a REPL). And within the ABCs themselves, inheritance is similarly useful. A Sequence is literally defined as a Sized, Iterable Container with some additional methods, not just in the documentation, but in the code.

    Hooks

    Often, the simplest way to provide some library functionality is to write the main code and provide a series of hooks for application developers to override whichever parts they want. And one obvious way to allow people to override code is method overriding.

    For example, look at socketserver. A TCPServer object handles all the basic functionality of listening on a server socket, accepting client sockets, and processing their requests. There are a number of different levels at which the application can hook that processing, but generally you do it by subclassing BaseRequestHandler and overriding one or more of its methods. Or you can use an HTTPServer, which hooks TCPServer for you, and then subclass BaseHTTPRequestHandler and override some of its methods instead.

    Of course the same thing could be done with a more functional hook interface—instead of subclassing BaseRequestHandler and overriding its handle method, you'd create a handle function that takes some kind of request parameter, then call set_handle_hook(my_handle_function). When the hook functions are all stateless or independent, this is identical, but when they have state that may be shared between them, the obvious place to put that state is in the self parameter—basically, the same reason you sometimes want to use objects instead of functions with an explicit cookie argument (or closures) in the first place.

    Mixins (implementation sharing)

    When you're learning OO theory, one of the first things you learn is that inheritance is for sharing interfaces, not implementation. But in many modern languages, that isn't true: you often want to build a class by composing behavior from multiple mixin or trait classes. Some languages have special syntax to do this, but in a language with proper multiple inheritance like Python, you can do this just by inheriting from mixins the same way you inherit from supertypes. For example, if you want to make a SocketServer concurrent, you can just inherit from ThreadingMixIn or ForkingMixIn as well as SocketServer (or write your own concurrency mixin that works differently).

    And there's nothing stopping a class from being an interface and a mixin at the same time. In fact, this is common and idiomatic in Python. For example, most of the collection ABCs are written this way—on the one hand, they're a way to check types nominally, but on the other hand, they make it easier to implement structural subtypes. For example, you implement __getitem__, and the Sequence mixing automatically implements __iter__ for you. This isn't always the simplest way to go in Python (the total_ordering decorator shows another approach), but it's often a good design.

    OO purists will argue that this is not proper OO—but again, practicality beats purity. Compare the various file-like objects in the stdlib, such as the one returned by socket.makefile, in 3.4 vs. 2.5. The modern version is almost always simpler and shorter, and it replaces the ad-hoc and inconsistent notion of "file-like object" with a set of well-specified (and nominally-checkable) interfaces.
    1

    View comments


  5. It's very common in a program to want to do two things at once: repaginate a document while still responding to user input, or handle requests from two (or 10000) web browsers at the same time. In fact, pretty much any GUI application, network server, game, or simulator needs to do this.

    It's possible to write your program to explicitly switch off between different tasks, and there are many higher-level approaches to this, which I've covered in previous posts. But an alternative is to have multiple "threads of control", each doing its own thing independently.

    There are three ways to do this: processes, threads, or greenlets. How do you decide between them?

    • Processes are good for running tasks that need to use CPU in parallel and don't need to share state, like doing some complex mathematical calculation to hundreds of inputs.
    • Threads are good for running a small number of I/O-bound tasks, like a program to download hundreds of web pages.
    • Greenlets are good for running a huge number of simple I/O-bound tasks, like a web server. Update: See greenlets vs. explicit coroutines on deciding whether to use automatic greenlets or explicit ones.
    If your program doesn't fit one of those three, you have to understand the tradeoffs.

    Multiprocessing

    Traditionally, the way to have separate threads of control was to have entirely independent programs. And often, this is still the best answer. Especially in Python, where you have helpers like multiprocessing.Process, multiprocessing.Pool, and concurrent.futures.ProcessPoolExecutor to wrap up most of the scaffolding for you.

    Separate processes have one major advantage: They're completely independent of each other. They can't interfere with each others' global objects by accident. This can make it easier to design your program. It also means that if one program crashes, the others are unaffected.

    Separate processes also have a major disadvantage: They're completely independent of each other. They can't share high-level objects. Processes can pass objects around—which is often a better solution. The standard library solutions do this by pickling the objects; this means that any object that can't be pickled (like a socket), or that would be too expensive to pickle and copy around (like a list of a billion numbers) won't work. Processes can also share buffers full of low-level data (like an array of a billion 32-bit C integers). In some cases, you can pass explicit requests and responses instead (e.g., if the background process is only going to need to get or set a few of those billion numbers, you can send get and set messages; the stdlib has Manager classes that do this automatically for simple lists and dicts). But sometimes, there's just no easy way to make this work.

    As a more minor disadvantage, on many platforms (especially Windows), starting a new process is a pretty heavy thing to do. We're not talking minutes here, just milliseconds, but still, if you're kicking off jobs that may only take 5ms to finish, and you add 30ms of overhead to each one, that's not exactly an optimization. Usually, using a Pool or Executor is the easy way around this problem, but it's not always appropriate.

    Finally, while modern OS's are pretty good at running, say, a couple dozen active processes and a couple hundred dormant ones, if you push things up to hundreds of active processes or thousands of dormant ones, you may end up spending more time in context-switching and scheduling overhead than doing actual work. If you know that your program is going to be using most of the machine's CPU, you generally want to try to use exactly as many processes as there are cores. (Again, using a Pool or Executor makes this easy, especially since they default to creating one process per core.)

    Threading

    Almost all modern operating systems have threads. These are like separate processes as far as the operating system's scheduler is concerned, but are still part of the same process in terms of the memory heap, open file table, etc. are concerned.

    The advantage of threads over processes is that everything is shared. If you modify an object in one thread, another thread can see it.

    The disadvantage of threads is that everything is shared. If you modify an object in two different threads, you've got a race condition. Even if you only modify it in one thread, it's not deterministic whether another thread sees the old value or the new one—which is especially bad for operations that aren't "atomic", where another thread could see some invalid intermediate value.

    One way to solve this problem is to use locks and other synchronization objects. (You can also use low-level "interlocked" primitives, like "atomic compare and swap", to build your own synchronization objects or lock-free objects, but this is very tricky and easy to get wrong.)

    The other way to solve this problem is to pretend you're using separate processes and pass around copies even though you don't have to.

    Python adds another disadvantage to threads: Under the covers, the Python interpreter itself has a bunch of globals that it needs. The CPython implementation (the one you're using if you don't know otherwise) does this by protecting its global state with a Global Interpreter Lock (GIL). So, a single process running Python can only execute one instruction at a time. So, if you have 16 processes, your 16 core machine can execute 16 instructions at once, one per process. But if you have 16 threads, you'll only execute one instruction, while the other 15 cores sit around idle. Custom extensions can work around this by releasing the GIL when they're busy doing non-Python work (NumPy, for example, will often do this), but it's still a problem that you have to profile. Some other implementations (Jython, IronPython, and some non-default-as-of-early-2015 optional builds of PyPy) get by without a GIL, so it may be worth looking at those implementations. But for many Python applications, multithreading means single-core.

    So, why ever use threads? Two reasons.

    First, some designs are just much easier to think of in terms of shared-everything threading. (However, keep in mind that many designs look easier this way, until you try to get the synchronization right…)

    Second, if your code is mostly I/O-bound (meaning you spend more time waiting on the network, the filesystem, the user, etc. than doing actual work—you can tell this because your CPU usage is nowhere near 100%), threads will usually be simpler and more efficient.

    Greenlets

    Greenlets—aka cooperative threads, user-level threads, green threads, or fibers—are similar to threads, but the application has to schedule them manually. Unlike a process or a thread, your greenlet function just keeps running until it decides to yield control to someone else.

    Why would you want to use greenlets? Because in some cases, your application can schedule things much more efficiently than the general-purpose scheduler built into your OS kernel. In particular, if you're writing a server that's listening on thousands of sockets, and your greenlets spend most of their time waiting on a socket read, your greenlet can tell the scheduler "Wake me up when I've got something to read" and then yield to the scheduler, and then do the read when it's woken up. In some cases this can be an order of magnitude more scalable than letting the OS interrupt and awaken threads arbitrarily.

    That can get a bit clunky to write, but third-party libraries like gevent and eventlet make it simple: you just call the recv method on a socket, and it automatically turns that into a "wake me up later, yield now, and recv once we're woken up". Then it looks exactly the same as the code you'd write using threads.

    Another advantage of greenlets is that you know that your code will never be arbitrarily preempted. Every operation that doesn't yield control is guaranteed to be atomic. This makes certain kinds of race conditions impossible. You still need to think through your synchronization, but often the result is simpler and more efficient.

    The big disadvantage is that if you accidentally write some CPU-bound code in a greenlet, it will block the entire program, preventing any other greenlets from running at all instead, whereas with threads it will just slow down the other threads a bit. (Of course sometimes this is a good thing—it makes it easier to reproduce and recognize the problem…)

    Update: Greenlets can also be used with explicit waiting. While the older ways of doing this were a bit clumsy, with newer frameworks, the question really is as simple as whether you mark each wait with await (as in asyncio) vs. whether they happen automatically when you call magic functions like socket.recv (as in gevent). What happens under the covers is the same either way. The tl;dr is that there's usually more advantage than disadvantage in marking your yields explicitly, but read greenlets vs. explicit coroutines if you want (a few) more details.
    6

    View comments

  6. If you've migrated to Python from C++ or one of its descendants (Java, C#, D, etc.), or to a lesser extent from other OO languages (Objective C, Ruby, etc.), the first time you asked for help on StackOverflow or CodeReview or python-list or anywhere else, the first response you got was probably: "Get rid of those getters and setters."

    If you asked why, you probably got no more of an answer than "You don't need them in Python."

    That's exactly the response you should get—but that doesn't mean it's a bad question, just that SO is the wrong place to answer it (especially in comments).

    To answer the question, we have to look at why getters and setters are idiomatic in those other languages, and see why the same idioms aren't appropriate in Python.

    Encapsulation

    The first reason you're usually given for using getters and setters is "for encapsulation: consumers of your class shouldn't even know that it has an attribute x, much less be able to change it willy-nilly." 

    As an argument for getters and setters, this is nonsense. (And the people who write the C++ standard agree, but they can't stop people from writing misleading textbooks or tutorials.) Consumers of your class do know that you have an attribute x, and can change it willy-nilly, because you have a method named set_x. That's the conventional and idiomatic name for a setter for the x attribute, and it would be highly confusing to have a method with that name that did anything else.

    The point of encapsulation is that the class's interface should be based on what the class does, not on what its state is. If what it does is just represent a point with x and y values, then the x and y attributes themselves are meaningful, and a getter and setter don't make them any more meaningful. If what it does is fetch a URL, parse the resulting JSON, fetch all referenced documents, and index them, then the HTTP connection pool is not meaningful, and a getter and setter don't make it any more meaningful. Adding a getter and setter almost never improves encapsulation in any meaningful way.

    Computed properties

    "Almost always" isn't "always". Say what your class does is represent a point more abstractly, with x, y, r, and theta values. Maybe x and y are attributes, maybe they're computed on the fly from r and theta. Or maybe they're sometimes stored as attributes, sometimes computed, depending on how you constructed or most recently set the instance. 

    In that case, you obviously do need getters and setters—not to hide the x and y attributes, but to hide the fact that there may not even be any x and y attributes. 

    That being said, is this really part of the interface, or just an implementation artifact? Often it's the latter. In that case, in Python (as in some C++ derived languages, like C#, but not in C++ itself), you can, and often should, still present them as attributes even if they really aren't, by using @property:

        class Point:
            @property
            def x(self):
                if self._x is None:
                    self._x, self._y = Point._polar_to_rect(self._r, self._theta)
                return self.x
            @x.setter
            def x(set, value):
                if self._x is None:
                    _, self._y = Point._polar_to_rect(self._r, self._theta)
                self._x = x
                self._r, self._theta = None, None
            # etc.

    Lifecycle management

    Sometimes, at least in C++ and traditional ObjC, even the "dumb" version of encapsulation idea isn't actually wrong, just oversimplified. Preventing people from setting your attribute by giving them a method to set your attribute is silly—but in C++, direct access to an attribute allows you to do more than just set the attribute, it allows you to store a reference to it. And, since C++ isn't garbage-collected, this means that there's nothing stopping some code from keeping that reference around after your instance goes out of scope, at which point they've now got a reference to garbage. In fact, this can be a major source of bugs in C++ code.

    If you instead provide a getter or setter, you can ensure that consumers can only get a copy of the attribute. Or, if you need to provide a reference, you can hold the object by smart pointer and return a new smart pointer to the object. (Of course this means that people who—usually as a misguided attempt at optimization—write getters that return a const reference, or that return objects that aren't copy-safe, like raw pointers, are defeating the entire purpose of having getters and setters in the first place…)

    This rationale, which makes perfect sense in C++, is irrelevant to Python. Python is garbage collected. And Python doesn't provide any way to take references to attributes in the first place; the only thing you can do is get a new (properly-GC-tracked) reference to the value of the attribute. If your instance goes away, but someone else is still referencing one of the values you had in an attribute, that value is still alive and perfectly usable.

    Interface stability

    Maybe today, you're storing x and y as attributes, but what if you want to change your implementation to compute them on the fly?

    In some languages, like C++, if you're worried about that, the only option you have is to create useless getters and setters today, so that if you change the implementation tomorrow, your interface doesn't have to change.

    In Python, just expose the attribute; if you change the implementation tomorrow, change the attribute to a @property, and your interface doesn't have to change.

    Interface inheritance

    In most languages, a subclass can change the semantics by overriding a getter and setter, but they can't do the same to a normal attribute.

    Even in languages that have properties, in most cases, defining a property with the same name as a base-class attribute will not affect the base class's code—or, in many languages, even consumers of the base class's interface; all it'll do is shadow the base class's attribute with a different attribute for subclasses and direct consumers of the derived class's interface.

    Also, in most languages without two-stage initialization, the derived class's code doesn't get a chance to run until the base class's initializer has finished, so it's not just difficult, but impossible, to affect how it stores its attributes.

    In Python, attribute access is fully dynamic, and two-stage initialization means that __init__ methods can work downward toward the root instead of upward toward the leaf, so the answer is, again, just @property.

    Design by contract

    Some tutorials and textbooks explain the need for setters by arguing that you can add pre- and post-condition tests to the setter, to preserve class invariants. Which is all well and good, but every case I've ever seen, they go on to show a simple void set_x(int x) {x_ = x; } in the first example, and every subsequent one.

    Needless to say, if you really are going to implement DBC today, this is just a case of "computed properties", and if you're just thinking you might want to implement it later, it's a case of "interface stability".

    Read-only attributes

    Sometimes, you want to expose an attribute, but make it read-only. In many languages, like C++, there's no way to do that but with a getter (and no corresponding setter). In Python, again, the answer is @property.

    C++, Java, etc. also give you a way to create class-wide constants. Python doesn't really have constants, so some people try to simulate this by adding a getter. Again, though, the answer is @property. Or, consider whether you really need to enforce the fact that it's a constant. Why would someone try to change it? What would happen if they did?

    Access control

    C++ and most of its descendants provide three levels of access control—public is the interface to everyone, protected is additional interface for subclasses, and private is only usable by the class's own methods and its friends. Sometimes you might want to make an attribute read-only for your subclasses but writable for your own methods (or read-only for your consumers but writable for your subclasses). The only way to do this is to make the attribute private (or protected) and add a protected (or public) getter.

    First, almost any OO design where this makes sense is probably not idiomatic for Python. (In fact, it's probably not even idiomatic for modern C++, but a lot of people are still programming 90s-style C++, and at any rate, it's much more idiomatic in Java. However, nobody is writing 90s-style OO in Python.) Often you can flatten out and simplify your hierarchy by passing around closures or methods, or by duck typing (that is, implicitly subtyping by just implementing the right methods, instead of subclassing); if you really do need subclassing, often you want to refactor your classes into small ABCs and/or mixins.

    It's also worth noting that access control doesn't provide any actual security against malicious subclasses or consumers. (Except in Java; see below.) In ObjC, they can just ask the runtime to inspect your ivars and change them through pointers. In C++, they can't do that—but everyone using your class has to be able to see all of your members, even if they're private, as part of the header, and if they really want to force your class to do something it wouldn't normally do by changing its private members, there's nothing stopping them from reinterpret_cast<>ing their way to any part of that structure. 

    At any rate, in Python, even if you wanted to control access this way, you can't do it.

    Some people teach that _x is Python's equivalent of protected, and __x its equivalent of private, but that's very misleading.

    The single underscore has only a conventional meaning: don't count on this being part of the useful and/or stable interface. Many introspection tools (e.g., tab completion in the interactive interface) will skip over underscore-prefixed names by default, but nothing stops a consumer from writing spam._eggs to access the value.

    The double underscore mangles the name—inside your own methods, the attribute is named __x, but from anywhere else, it's named _MyClass__x. But this is not there to add any more protection—after all, _MyClass__x will still show up in dir(my_instance), and someone can still write my_instance._MyClass__x = 42. What it's there for is to prevent subclasses from accidentally shadowing your attributes or methods. (This is primarily important when the base classes and subclasses are implemented independently—you wouldn't want to add a new _spam attribute to your library and accidentally break any app that subclasses your library and adds a _spam attribute.)

    Security

    Java was designed to allow components that don't trust each other to run securely. That means that in some cases, access control does actually provide security. (Of course if you're just going to hide your x behind a public set_x method, that isn't adding anything…) That's great for Java, but it's irrelevant to Python, or any language without a secure class loader, etc.

    Generic attributes

    As far as I know, this one is really only relevant to C++ , because most other languages' generics were designed around offering most of the useful features of C++ templates without all of the mess, while C++'s templates were designed before anyone knew what they wanted to do with generics.

    There are many cases where it's hard to specify a type for an attribute in a class template. C++ has much more limited type inference for classes and objects (before C++11, there was none at all) than for functions and types. Also, a class or class template can have methods that are themselves templates, but there are no "object templates", so you have to simulate them with either traits classes or function templates—that is, templated getters and setters. And there are also cases where it's hard to specify a constant or initializer value for an attribute, so again you have to simulate them with either traits classes or function templates.

    Needless to say, none of these applies at all in duck-typed Python.

    Libraries and tools

    In some languages, there are libraries for reflection or serialization, observer-notification frameworks, code-generating wizards for UIs or network protocols, tools like IDEs and refactoring assistants, etc., that expect you to use getters and setters. This is particularly true in Java, and to a lesser extent C# and ObjC. Obviously you don't want to fight against these tools.

    The tools and libraries for Python are, of course, designed around the idea that your attributes are attributes, not hidden behind getters and setters. But if you're, say, using PyObjC to write Python code that's bound to a NIB both at runtime and in InterfaceBuilder, you have to do things the way InterfaceBuilder wants you to.

    Breakpoints

    In some debuggers, it's impossible to place a watchpoint on a variable, or at least much harder or much less efficient than placing a breakpoint on a setter. If you expect this to be a problem, and you need to be able to debug code in the field without editing it, you might want to hide some attributes behind @property for easier debugging. But this doesn't come up very often.

    Other languages

    So, what if you're a Python programmer and you have to write some code in Java or D? Just as you shouldn't make your Python code look like Java, you shouldn't make your Java code look like Python. This means you'll probably want a lot more getters and setters than you're used to.

    In Java, C#, D, etc. many of the same motivations (both good and bad) for getters and setters apply the same as in C++. In some cases, some of the motivations don't apply (e.g., C# has properties, just like Python; Java generics don't work like C++ templates). So there are a few cases where there are additional reasons for getters and setters.

    But, more importantly, Java (and, to a lesser extent, C#, ObjC, etc.) has a much stronger culture than C++ of idiomatically requiring getters and setters even when there's no objectively good reason. There are wizards that generate them for you, linters that warn if you don't use them, IDEs that expect them to exist, coworkers or customers that complain… And the fact that it's idiomatic is in itself a good reason to follow the idiom, even if there's no objective basis for it.
    8

    View comments

Blog Archive
About Me
About Me
Loading
Dynamic Views theme. Powered by Blogger. Report Abuse.