In a recent thread on python-ideas, Stephan Sahm suggested, in effect, changing the method resolution order (MRO) from C3-linearization to a simple depth-first search a la old-school Python or C++. I don't think he realized that's what he was proposing at first, but the key idea is that he wants Mixin to override A in B as well as overriding object in A in this code:

class Mixin: pass
class A(Mixin): pass
class B(Mixin, A): pass

In other words, the MRO should be B-Mixin-A-Mixin-object. (Why not B-Mixin-object-A-Mixin-object? I think he just didn't think that through, but let's not worry about that.) After all, why would he put Mixin before A if he didn't want it to override A in B? And why would he attach Mixin to A if he didn't want it to override object in A?

Well, that doesn't actually work. The whole point of linearization is that each class appears only once in the MRO, and many feature of Python--including super, which he wanted to make extensive use of--depend on that. For example, with his MRO, inside Mixin.spam, super().spam() is going to call A.spam, and its super().spam() is going to call Mixin.spam(), and you've obviously got a RecursionError on your hands.

I think ultimately, his problem is that what he wants isn't really a mixin class (in typical Python terms--in general OO programming terms, it's one of the most overloaded words you can find...). For example, a wrapper class factory could do exactly what he wants, like this:

def Wrapper(cls): return cls
class A(Wrapper(object)): pass
class B(Wrapper(A)): pass

And there are other ways to get where he wants.

Anyway, changing the default MRO in Python this way is a non-starter. But if he wants to make that change manually, how hard is it? And could he build a function that lets his classes could cooperate using that function instead of super?

Customizing MRO


The first step is to build the custom MRO. This is pretty easy. He wants a depth-first search of all bases, so:

[cls] + list(itertools.chain.from_iterable(base.__mro__ for base in cls.__bases__))

Or, if leaving the extra object out was intentional, that's almost as easy:

[cls] + list(itertools.chain.from_iterable(base.__mro__[:-1] for base in cls.__bases__)) + [object]

But now, how do we get that into the class's __mro__ attribute?

It's a read-only property; you can't just set it. And, even if you could, you need type.__new__ to actually return something for you to modify--but if you give it a non-linearizable inheritance tree, it'll raise an exception. And finally, even if you could get it set the way you want, every time __bases__ is changed, __mro__ is automatically rebuilt.

So, we need to hook the way type builds __mro__.

I'm not sure if this is anywhere in the reference documentation or not, but the answer is pretty easy: the way type builds __mro__ is by calling its mro method. This is treated as a special method (that part definitely isn't documented anywhere), meaning it's looked up on the class (that is, the metaclass of the class being built) rather than the instance (the class being built), doesn't go through __getattribute__, etc., so we have to build a metaclass.

But once you know that, it's all trivial:

class MyMRO(type): 
    def mro(cls): 
        return ([cls] + 
                list(itertools.chain.from_iterable(base.__mro__[1:] for base in cls.__bases__)) +
                [object])

class Mixin(metaclass=MyMRO): pass
class A(Mixin): pass
class B(Mixin, A): pass

And now, B.__mro__ is B-Mixin-A-Mixin-object, exactly as desired.

For normal method calls, this does what the OP wanted: Mixin gets to override A.

But, as mentioned earlier, it obviously won't enable the kind of super he wants, and there's no way it could. So, we'll have to build our own replacement.

Bizarro Super


If you want to learn how super works, I think the documentation in Invoking Descriptors is complete, but maybe a little terse to serve as a tutorial. I know there's a great tutorial out there, but I don't have a link, so... google it.

Anyway, how super works isn't important; what's important is the define what we want here. Once we actually know exactly what we want, anything is possible as long as you believe, that's what science is all about.

Since we're defining something very different from super but still sort of similar, the obvious name is bizarro.

Now, we want a call to bizarro().spam() inside B.spam to call Mixin.spam, a call inside Mixin.spam to call A.spam, a call inside A.spam to call Mixin.spam, and a call inside Mixin.spam to call object.spam.

The first problem is that calling object.spam is just going to raise an AttributeError. Multiple inheritance uses of super are all about cooperative class hierarchies, and part of that cooperation is usually that the root of your tree knows not to call super. But here, Mixin is the root of our tree, but it also appears in other places on the tree, so that isn't going to work.

Well, since we're designing our own super replacement, there's no reason it can't also cooperate with the classes, instead of trying to be fully general. Just make it return a do-nothing function if the next class is object, or if the next class doesn't have the method, or if the next class has a different metaclass, etc. Pick whatever rule makes sense for your specific use case. Since I don't have a specific use case, and don't know what the OP's was (he wanted to create a "Liftable" mixin that helps convert instances of a base class into instances of a subclass, but he didn't explain how he wanted all of the edge cases to work, and didn't explain enough about why he wanted such a thing for me to try to guess on his behalf), I'll go with the "doesn't have the method".

While we're at it, we can skip over any non-cooperating classes that end up in the MRO (which would obviously be important if we didn't block object from appearing multiple times--but even with the MRO rule above, you'll have the same problem if your root doesn't directly inherit object).

The next problem--the one that's at the root of everything we're trying to work around here--is that we want two different things to happen "inside Mixin.spam", depending on whether it's the first time we're inside or the second. How are we going to do that?

Well, obviously, we need to keep track of whether it's the first time or the second time. One obvious way is to keep track of the index, so it's not A.spam if we're in Mixin.spam or object.spam if we're in Mixin.spam, it's B.__mro__[2] if we're in B.__mro__[1], and B.__mro__[4] if we're in B.__mro__[3]. (After first coding this up, I realized that an iterator might be a lot nicer than an index, so if you actually need to implement this yourself, try it that way. But I don't want to change everything right now.)

How can we keep track of anything? Make the classes cooperate. Part of the protocol for calling bizarro is that you take a bizarro_index parameter and pass it into the bizarro call. Let's make it as an optional parameter with a default value of 0, so your users don't have to worry about it, and make it keyword-only, so it doesn't interfere with *args or anything. So:

class Mixin(metaclass=MyMRO):
    def doit(self, *, bizarro_index=0):
        print('Mixin')
        bizarro(Mixin, self, bizarro_index).doit()

class A(Mixin):
    def doit(self, *, bizarro_index=0):
        print('A')
        bizarro(A, self, bizarro_index).doit()

class B(Mixin, A):
    def doit(self, *, bizarro_index=0):
        print('B')
        bizarro(B, self, bizarro_index).doit()

And now, we just have to write bizarro.

The key to writing something like super is that it returns a proxy object whose __getattribute__ looks in the next class on the MRO. If you found that nice tutorial on how super works, you can start with the code from there. We then have to make some changes:

  1. The way we pick the next class has to be based on the index.
  2. Our proxy has to wrap the function up to pass the index along.
  3. Whatever logic we wanted for dealing with non-cooperating classes has to go in there somewhere.

Nothing particularly hard. So:

def bizarro(cls, inst, idx): 
    class proxy: 
        def __getattribute__(self, name): 
            for superidx, supercls in enumerate(type(inst).__mro__[idx+1:], idx+1): 
                try:
                    method = getattr(supercls, name).__get__(inst) 
                except AttributeError: 
                    continue 
                if not callable(method):
                    return method # so bizarro works for data attributes
                @functools.wraps(method) 
                def wrapper(*args, **kw):
                    return method(*args, bizarro_index=superidx, **kw)
                return wrapper 
            return lambda *args, **kw: None 
    return proxy() 

And now, we're done.

Bizarro am very beautiful


In Python 3, super(Mixin, self) was turned into super(). This uses a bit of magic, and we can use the same magic here.

Every method has a cell named __class__ that tells you which class it's defined in. And every method takes its self as the first parameter. So, if we just peek into the caller's frame, we can get those easily. And while we're peeking into the frames, since we know the index has to be the bizarro_index parameter to any function that's going to participate in bizarro super-ing, we can grab that too:

def bizarro():
    f = sys._getframe(1)
    cls = f.f_locals['__class__']
    inst = f.f_locals[f.f_code.co_varnames[0]]
    idx = f.f_locals['bizarro_index']
    # everything else is the same as above

This is cheating a bit; if you read PEP 3135, the super function doesn't actually peek into the parent frame; instead, the parser recognizes calls to super() and changes them to pass the two values. I'm not sure that's actually less hacky--but it is certainly more portable, because other Python implementations aren't required to provide CPython-style frames and code objects. Also leaving the magic up to the parser means that, e.g., PyPy can still apply its no-frames-unless-needed optimization, trading a tiny bit of compile-time work for a small savings in every call.

If you want to do the same here, you can write an import hook that AST-transforms bizarro calls in the same way. But I'm going to stick with the frame hack.

Either way, now you can write this:

class Mixin(metaclass=MyMRO):
    def doit(self, *, bizarro_index=0):
        print('Mixin')
        bizarro().doit()

class A(Mixin):
    def doit(self, *, bizarro_index=0):
        print('A')
        bizarro().doit()

class B(Mixin, A):
    def doit(self, *, bizarro_index=0):
        print('B')
        bizarro().doit()

Meanwhile, notice that we don't actually use cls anywhere anyway, so... half a hack is only 90% as bad, right?

But still, that bizarro_index=0 bit. All that typing. All that reading. There's gotta be a better way.

Well, now you can!

We've already got a metaclass. We're already peeking under the covers. We're already wrapping functions. So, let's have our metaclass peek under the covers of all of our functions and automatically wrap anything that uses bizarro to take that bizarro_index parameter. The only problem is that the value will now be in the calling frame's parent frame (that is, the wrapper), but that's easy to fix too: just look in f_back.f_locals instead of f_locals.

import functools
import itertools
import sys

class BizarroMeta(type):
    def mro(cls):
        return ([cls] +
                list(itertools.chain.from_iterable(base.__mro__[:-1] for base in cls.__bases__)) +
                [object])
    def __new__(mcls, name, bases, attrs):
        def _fix(attr):
            if callable(attr) and 'bizarro' in attr.__code__.co_names:
                @functools.wraps(attr)
                def wrapper(*args, bizarro_index=0, **kw):
                    return attr(*args, **kw)
                return wrapper
            return attr
        attrs = {k: _fix(v) for k, v in attrs.items()}
        return super().__new__(mcls, name, bases, attrs)

def bizarro():
    f = sys._getframe(1)
    inst = f.f_locals[f.f_code.co_varnames[0]]
    idx = f.f_back.f_locals['bizarro_index']
    class proxy: 
        def __getattribute__(self, name): 
            for superidx, supercls in enumerate(type(inst).__mro__[idx+1:], idx+1):
                try:
                    method = getattr(supercls, name).__get__(inst)
                except AttributeError: 
                    continue 
                if not callable(method):
                    return method # so bizarro works for data attributes
                @functools.wraps(method) 
                def wrapper(*args, **kw): 
                    return method(*args, bizarro_index=superidx, **kw)
                return wrapper 
            return lambda *args, **kw: None 
    return proxy() 

class Mixin(metaclass=BizarroMeta):
    def doit(self):
        print('Mixin')
        bizarro().doit()

class A(Mixin):
    def doit(self):
        print('A')
        bizarro().doit()

class B(Mixin, A):
    def doit(self):
        print('B')
        bizarro().doit()

B().doit()

Run this, and it'll print B, then Mixin, then A, then Mixin.

Unless I made a minor typo somewhere, in which case it'll probably crash in some way you can't possibly debug. So you'll probably want to add a bit of error handling in various places. For example, it's perfectly legal for something to be callable but not have a __code__ member--a class, a C extension function, an instance of a class with a custom __call__ method... Whether you want to warn that you can't tell whether Spam.eggs uses bizarro or not because you can't find the code, assume it doesn't and skip it, assume it does and raise a readable exception, or something else, I don't know, but you probably don't want to raise an exception saying that type objects don't have __code__ attributes, or whatever comes out of this mess by default.

Anyway, the implementation is pretty it's pretty small, and not that complicated once you understand all the things we're dealing with, and the API for using it is about as nice as you could want.

I still don't know why you'd ever want to do this, but if you do, go for it.
1

View comments

It's been more than a decade since Typical Programmer Greg Jorgensen taught the word about Abject-Oriented Programming.

Much of what he said still applies, but other things have changed. Languages in the Abject-Oriented space have been borrowing ideas from another paradigm entirely—and then everyone realized that languages like Python, Ruby, and JavaScript had been doing it for years and just hadn't noticed (because these languages do not require you to declare what you're doing, or even to know what you're doing). Meanwhile, new hybrid languages borrow freely from both paradigms.

This other paradigm—which is actually older, but was largely constrained to university basements until recent years—is called Functional Addiction.

A Functional Addict is someone who regularly gets higher-order—sometimes they may even exhibit dependent types—but still manages to retain a job.

Retaining a job is of course the goal of all programming. This is why some of these new hybrid languages, like Rust, check all borrowing, from both paradigms, so extensively that you can make regular progress for months without ever successfully compiling your code, and your managers will appreciate that progress. After all, once it does compile, it will definitely work.

Closures

It's long been known that Closures are dual to Encapsulation.

As Abject-Oriented Programming explained, Encapsulation involves making all of your variables public, and ideally global, to let the rest of the code decide what should and shouldn't be private.

Closures, by contrast, are a way of referring to variables from outer scopes. And there is no scope more outer than global.

Immutability

One of the reasons Functional Addiction has become popular in recent years is that to truly take advantage of multi-core systems, you need immutable data, sometimes also called persistent data.

Instead of mutating a function to fix a bug, you should always make a new copy of that function. For example:

function getCustName(custID)
{
    custRec = readFromDB("customer", custID);
    fullname = custRec[1] + ' ' + custRec[2];
    return fullname;
}

When you discover that you actually wanted fields 2 and 3 rather than 1 and 2, it might be tempting to mutate the state of this function. But doing so is dangerous. The right answer is to make a copy, and then try to remember to use the copy instead of the original:

function getCustName(custID)
{
    custRec = readFromDB("customer", custID);
    fullname = custRec[1] + ' ' + custRec[2];
    return fullname;
}

function getCustName2(custID)
{
    custRec = readFromDB("customer", custID);
    fullname = custRec[2] + ' ' + custRec[3];
    return fullname;
}

This means anyone still using the original function can continue to reference the old code, but as soon as it's no longer needed, it will be automatically garbage collected. (Automatic garbage collection isn't free, but it can be outsourced cheaply.)

Higher-Order Functions

In traditional Abject-Oriented Programming, you are required to give each function a name. But over time, the name of the function may drift away from what it actually does, making it as misleading as comments. Experience has shown that people will only keep once copy of their information up to date, and the CHANGES.TXT file is the right place for that.

Higher-Order Functions can solve this problem:

function []Functions = [
    lambda(custID) {
        custRec = readFromDB("customer", custID);
        fullname = custRec[1] + ' ' + custRec[2];
        return fullname;
    },
    lambda(custID) {
        custRec = readFromDB("customer", custID);
        fullname = custRec[2] + ' ' + custRec[3];
        return fullname;
    },
]

Now you can refer to this functions by order, so there's no need for names.

Parametric Polymorphism

Traditional languages offer Abject-Oriented Polymorphism and Ad-Hoc Polymorphism (also known as Overloading), but better languages also offer Parametric Polymorphism.

The key to Parametric Polymorphism is that the type of the output can be determined from the type of the inputs via Algebra. For example:

function getCustData(custId, x)
{
    if (x == int(x)) {
        custRec = readFromDB("customer", custId);
        fullname = custRec[1] + ' ' + custRec[2];
        return int(fullname);
    } else if (x.real == 0) {
        custRec = readFromDB("customer", custId);
        fullname = custRec[1] + ' ' + custRec[2];
        return double(fullname);
    } else {
        custRec = readFromDB("customer", custId);
        fullname = custRec[1] + ' ' + custRec[2];
        return complex(fullname);
    }
}

Notice that we've called the variable x. This is how you know you're using Algebraic Data Types. The names y, z, and sometimes w are also Algebraic.

Type Inference

Languages that enable Functional Addiction often feature Type Inference. This means that the compiler can infer your typing without you having to be explicit:


function getCustName(custID)
{
    // WARNING: Make sure the DB is locked here or
    custRec = readFromDB("customer", custID);
    fullname = custRec[1] + ' ' + custRec[2];
    return fullname;
}

We didn't specify what will happen if the DB is not locked. And that's fine, because the compiler will figure it out and insert code that corrupts the data, without us needing to tell it to!

By contrast, most Abject-Oriented languages are either nominally typed—meaning that you give names to all of your types instead of meanings—or dynamically typed—meaning that your variables are all unique individuals that can accomplish anything if they try.

Memoization

Memoization means caching the results of a function call:

function getCustName(custID)
{
    if (custID == 3) { return "John Smith"; }
    custRec = readFromDB("customer", custID);
    fullname = custRec[1] + ' ' + custRec[2];
    return fullname;
}

Non-Strictness

Non-Strictness is often confused with Laziness, but in fact Laziness is just one kind of Non-Strictness. Here's an example that compares two different forms of Non-Strictness:

/****************************************
*
* TO DO:
*
* get tax rate for the customer state
* eventually from some table
*
****************************************/
// function lazyTaxRate(custId) {}

function callByNameTextRate(custId)
{
    /****************************************
    *
    * TO DO:
    *
    * get tax rate for the customer state
    * eventually from some table
    *
    ****************************************/
}

Both are Non-Strict, but the second one forces the compiler to actually compile the function just so we can Call it By Name. This causes code bloat. The Lazy version will be smaller and faster. Plus, Lazy programming allows us to create infinite recursion without making the program hang:

/****************************************
*
* TO DO:
*
* get tax rate for the customer state
* eventually from some table
*
****************************************/
// function lazyTaxRateRecursive(custId) { lazyTaxRateRecursive(custId); }

Laziness is often combined with Memoization:

function getCustName(custID)
{
    // if (custID == 3) { return "John Smith"; }
    custRec = readFromDB("customer", custID);
    fullname = custRec[1] + ' ' + custRec[2];
    return fullname;
}

Outside the world of Functional Addicts, this same technique is often called Test-Driven Development. If enough tests can be embedded in the code to achieve 100% coverage, or at least a decent amount, your code is guaranteed to be safe. But because the tests are not compiled and executed in the normal run, or indeed ever, they don't affect performance or correctness.

Conclusion

Many people claim that the days of Abject-Oriented Programming are over. But this is pure hype. Functional Addiction and Abject Orientation are not actually at odds with each other, but instead complement each other.
5

View comments

Blog Archive
About Me
About Me
Loading
Dynamic Views theme. Powered by Blogger. Report Abuse.