One common "advanced question" on places like StackOverflow and python-list is "how do I dynamically create a function/method/class/whatever"? The standard answer is: first, some caveats about why you probably don't want to do that, and then an explanation of the various ways to do it when you really do need to.

But really, creating functions, methods, classes, etc. in Python is always already dynamic.
Some cases of "I need a dynamic function" are just "Yeah? And you've already got one". More often, you do need something a little more complicated, but still something Python already gives you. Occasionally, even that isn't enough. But, once you understand how functions, methods, classes, etc. work in Python, it's usually pretty easy to understand how to do what you want. And when you really need to go over the edge, almost anything you can think of, even if it's almost always a bad idea, is probably doable (either because "almost always" isn't "always", or just because almost nobody would ever think to try to do it, so it wasn't worth preventing).

Functions

A normal def statement compiles to code that creates a new function object at runtime.

For example, consider this code:
def spam(x):
    return x+1
Let's say you type that into the REPL (the interactive interpreter). The REPL reads lines until it gets a complete statement, parses and compiles that statement, and then interprets the resulting bytecode. And what does that definition get compiled to? Basically the same thing as this:
spam = types.FunctionType(
    compile('return x+1\n', '__main__', mode='function'),
    globals(),
    'spam',
    (),
    ())
spam.__qualname__ = 'spam'
You can't quite write this, because the public interface for the compile function doesn't expose all of the necessary features--but outside of that compile, the rest is all real Python. (For simple lambda functions, you can use, e.g., compile('1 + 2', '__main__', mode='eval') and the whole thing is real Python, but that doesn't work for def functions. When you really need to create code objects, there are ways to do it, but you very rarely need to, so let's not worry about that.)

If you put the same thing in a module instead of typing it at the REPL, the only difference is that the body is compiled ahead of time and stored in a marshaled code object inside the .pyc file so it never needs to be compiled again. The def statement is still compiled and then interpreted as top-level module code that constructs a function on the fly out of that code constant, every time you import the module..

For a slightly more complicated example, consider this:
def add_one(x: int=0) -> int:
    """Add 1"""
    return x+1
This is equivalent to:
spam = types.FunctionType(
    compile('return x+1\n', '__main__', mode='function'),
    globals(),
    'add_one',
    (0,), # This is where default values go
    ())
spam.__qualname__ = 'add_one'
spam.__doc__ = """Add 1"""
adder.__annotations__ = {'return': 'int', 'x': 'int'}
Notice that the default values are passed into that FunctionType constructor. That's why defaults are bound in at the time the def statement is executed, which is how you can do tricks like using a dict crammed into a default value as a persistent cache.

Closures

The real point of functions always being created on the fly is that this means any function can be a closure--it can capture values from the environment that the function was defined in. The standard Lisp-style example looks like this:
def make_adder(n):
    def adder(x):
 return x+n
    return adder
That's equivalent to:
adder = types.FunctionType(
    compile('return x+n', '__main__', mode='exec'),
    globals(),
    'adder',
    (),
    (CellType(locals(), 'n'),)) # tuple of closure cells
adder.__qualname__ = 'make_adder.<locals>.adder'
So every time you call make_adder, you get back a new adder function, created on the fly, referencing the particular n local variable from that particular call to make_adder.

(Unfortunately, I cheated a bit. Unlike function objects, and even code objects, you can't actually manually create closure cells like this. But you rarely want to. And if you ever do need it, you can just do a trivial lambda that captures n and then do a minor frame hack to get at the cell.)

Even if you never want to do anything this Lispy, closures get used all over the place. For example, if you've ever written a Tk GUI, you may have done something like this in your Frame subclass:
def __init__(self):
    Frame.__init__(self)
    self.hello_button = tkinter.Button(
        self, text='Hello',
        command=lambda: self.on_button_click(self.hello_button))
That lambda is creating a new function that captures the local self variable so it can access self.hello_button whenever the button is clicked.

(A lambda compiles in almost the same way as a def, except that it's a value in the middle of an expression rather than a statement, and it doesn't have a name, docstring, etc.).

Another common way to write the same button is with functools.partial:
    self.hello_button = tkinter.Button(
        self, text='Hello',
        command=partial(self.on_button_click, self.hello_button)
If Python didn't come with partial, we could easily write it ourselves:
def partial(func, *args, **kw):
    def wrapped(*more_args, **more_kw):
        return func(*args, *more_args, **kw, **more_kw)
    return wrapped
This is also how decorators work:
def simple_memo(func):
    cache = {}
    def wrapped(*args):
        args = tuple(args)
        if args not in cache:
     cache[args] = func(*args)
 return cache[args]
    return wrapped

@simple_memo
def fib(n):
    if n < 2: return n
    return fib(n-1) + fib(n-2)
I've written a dumb exponenially-recursive Fibonacci function, and that @simple_memo magically turns it into a linear-time function that takes a fraction of a second instead of hours. How does this work? Simple: after the usual fib = types.FunctionType blah blah stuff, it does fib = simple_memo(fib). That's it. Because function are already always created on the fly, decorators don't need anything complicated.

By the way, if you can follow everything above, you pretty much know all there is to know about dynamic higher-order programming, except for how the theory behind it maps to advanced math. (And that part is simple if you already know the math, but meaningless if you don't.) That's one of those things that sounds scary when functional programmers talk about it, but if you go from using higher-order functions to building them to understanding how the work before the theory, instead of going from theory to implementation to building to using, it's not actually hard.

Fake functions

Sometimes, you can describe what code should run when you call spam, but it's not obvious how to construct a function object that actually runs that code. Or it's easy to write the closure, but hard to think about it when you later come back and read it.

In those cases, you can create a class with a __call__ method, and it acts like a function. For example:
class Adder:
    def __init__(self, n):
        self._n = n
    def __call__(self, x):
        return x+n
An Adder(5) object behaves almost identical to a make_adder(5) closure. It's just a matter of which one you find more readable. Even for experienced Python programmers, the answer is different in different cases, which is why you'll find both techniques all over the stdlib and popular third-party modules.

In fact, functools.partial isn't actually a closure, but a class, like this:
class partial:
    def __init__(self, func, *args, **kw):
        self.func, self.args, self.kw = func, args, kw
    def __call__(self, *more_args, **more_kw):
        return self.func(*self.args, *more_args, **self.kw, **more_kw)
(Actually, the real partial has a lot of bells and whistles. But it's still not that complicated. The docs link to the source code, if you want to see it for yourself.)

Methods

OK, so you can create functions on the fly; what about methods?

Again, they're already always created on the fly, and once you understand how, you can probably do whatever it was you needed.

Let's look at an example:
class Spam:
    def __init__(self, x, y):
        self.x, self.y = x, y
    def eggs(self):
        return self.x + self.y
spam = Spam()
print(spam.eggs())
The definition of eggs is compiled and interpreted exactly the same as the definition of any other function. And the result is just stored as a member of the Spam class (see the section on classes to see how that works), so when you write Spam.eggs you just get that function.

This means that if you want to add a new method to a class, there's no special trick, you just do it:
def cheese(self):
    return self.x * self.y
Spam.cheese = cheese
print(spam.cheese())
That's all it takes to add a method to a class dynamically.

But meanwhile, on the instance, spam.eggs is not just a function, it's a bound method. Try print(spam.eggs) from the interactive REPL. A bound method knows which instance it belongs to, so when you call it, that instance can get passed as the self argument.

The details of how Python turns the function Spam.eggs into the bound method spam.eggs are a bit complicated (and I've already written a whole post about them), but we don't need to know that here.

Obviously, bound methods get created dynamically. Every time you do spam.eggs or Spam().cheese or string.ascii_letters.find, you're getting a new bound method.

And if you want to create one manually, you can just call types.MethodType(func, obj).

So, if you want to add a new method beans to just the spam instance, without adding it to the Spam class? Just construct the same bound method that Python would have constructed for you whenever you looked up spam.beans, and store it there:
def beans(self):
    return self.x / self.y
spam.beans = types.MethodType(beans, spam)
And now you know enough to implement Javascript-style object literals, or even prototype inheritance. Not that you should do either, but if you ever run into something that you really do want to do, that requires creating methods on the fly, either on classes or on instances, you can do it. Because creating methods on the fly is what Python always does.

Classes

What if we want to create a class dynamically?

Well, I shouldn't have to tell you at this point. You're always creating classes dynamically.

Class definitions work a bit differently from function definitions. Let's start with a simple example again:
class Spam:
    z = 0
    def __init__(self, x, y):
        self.x, self.y = x, y
    def eggs(self):
        return self.x + self.y + self.z
First, Python interprets the class body the same as any other code, but it runs inside an empty environment. So, those def calls create new functions named __init__ and eggs in that empty environment, instead of at the global level. Then, it dynamically creates a class object out of that environment, where every function or other value that got created becomes a method or class attribute on the class. The code goes something like this:
_Spam_locals = {}
exec('def __init__(self, x, y):\n    self.x, ... blah blah ...\n',
     globals=globals(), locals=_Spam_locals)
Spam = type('Spam', (object,), _Spam_locals)
This is why you can't access the Spam class object inside the class definition--because there is nothing named Spam until after Python calls type and stores the result in Spam. (But of course you can access Spam inside the methods; by the time those methods get called, it'll exist.)

So, what if you want to create some methods dynamically inside the class? No sweat. By the time it gets to calling type, nobody can tell whether eggs gets into the locals dict by you calling def eggs(...): or eggs = fancy_higher_order_function(...), so they both do the same thing.

In fact, one idiom you'll see quite often in the stdlib is this:
    def __add__(self, other):
        blah blah
    __radd__ = __add__
This just makes __radd__ another name for the same method as __add__.

And yes, you can call type manually if you need to, passing it any dict you want as an environment:
def __init__(self, x, y):
    self.x, self.y = x, y
Spam = type('Spam', (object,),
    {'z': 0, '__init__': __init__,
     'eggs': lambda self: self.x + self.y + self.z}
There are a few more details to classes. A slightly more complicated example covers most of them:
@decorate_my_class
class Spam(metaclass=MetaSpam, Base1, Base2):
    """Look, I've got a doc string"""
    def __init__(self, x, y):
        self.x, self.y = x, y
This is equivalent to:
_Spam_locals = {}
exec('def __init__(self, x, y):\n    self.x, ... blah blah ...\n',
     globals=globals(), locals=_Spam_locals)
Spam = MetaSpam('Spam', (Base1, Base2), _Spam_locals)
Spam.__doc__ = """Look, I've got a doc string"""
Spam = decorate_my_class(Spam)
As you can see, if there's a metaclass, it gets called in place of type, and if there are base classes, they get passed in place of object, and docstrings and decorators work the same way as in functions.

There are a few more complexities with qualnames, __slots__, closures (if you define a class inside a function), and the magic to make super() work, but this is almost everything.

Remember from the last section how easy it is to add methods to a class? Often that's simpler than trying to programmatically generate methods from inside the class definition, or customize the class creation. (See functools.total_ordering for a nice example.) But when you really do need a dynamically-created class for some reason, it's easy.

Generating code

Occasionally, no matter how hard you try, you just can't come up with any way to define or modify a function or class dynamically with your details crammed into the right place the right way, at least not readably. In that case, you can always fall back to generating, compiling, and executing source code.

The simplest way to do this is to just build a string and call exec on it. You can find a few examples of this in the stdlib, like collections.namedtuple. (Notice the trick it uses of calling exec in a custom empty namespace, then copying the value out of it. This is a bit cleaner that just executing in your own locals and/or globals.)

You've probably hard "exec is dangerous". And of course it is. But "dangerous" really just means two things: "powerful" and "hard to control or reason about". When you need the first one badly enough that you can accept the second, danger is justified. If you don't understand things like how to make sure you're not letting data some user sent to your web service end up inside your exec don't use it. But if you're still reading at this point, I think you can learn how to reason through the issues.

Sometimes you don't want to exec right now, you want to compile something that you can pass around and exec later. That works fine too.

Sometimes, you even want to generate a whole module and save it as a .py file. Of course people do that kind of thing in C all the time, as part of the build process--but in Python, it doesn't matter whether you're generating a .py file during a build or install to be used later, or generating one at normal runtime to be used right now; they're the same case as far as Python is concerned.

Sometimes you want to build an AST (abstract source tree) or a stream of tokens instead of source code, and then compile or execute that. (And this is a great time to yet again plug macropy, one of the coolest projects ever.)

Sometimes you even want to do the equivalent of inline assembly (but assembling Python bytecode, not native machine code, of course) with a library like byteplay or cpyasm (or, if you're a real masochist, just assembling it in your head and using struct.pack on the resulting array of 16-bit ints...). Again, unlike C, you can do this at runtime, then wrap that code object up in a function object, and call it right now.

You can even do stuff like marshaling code objects to and from a database to build functions out of later.

Conclusion

Because almost all of this is accessible from within Python itself, and all of it is inherently designed to be executed on the fly, almost anything you can think of is probably doable.

So, if you're thinking "I need to dynamically create X", you need to think through exactly what you need, but whether that turns out to be "just a normal function" or something deeply magical, you'll be able to do it, or at least explain the magic you're looking for in specific enough terms that someone can actually show you how to do it.
1

View comments

Blog Archive
About Me
About Me
Loading
Dynamic Views theme. Powered by Blogger. Report Abuse.