But really, creating functions, methods, classes, etc. in Python is always already dynamic.
Some cases of "I need a dynamic function" are just "Yeah? And you've already got one". More often, you do need something a little more complicated, but still something Python already gives you. Occasionally, even that isn't enough. But, once you understand how functions, methods, classes, etc. work in Python, it's usually pretty easy to understand how to do what you want. And when you really need to go over the edge, almost anything you can think of, even if it's almost always a bad idea, is probably doable (either because "almost always" isn't "always", or just because almost nobody would ever think to try to do it, so it wasn't worth preventing).
Functions
A normaldef
statement compiles to code that creates a
new function object at runtime.For example, consider this code:
def spam(x): return x+1Let's say you type that into the REPL (the interactive interpreter). The REPL reads lines until it gets a complete statement, parses and compiles that statement, and then interprets the resulting bytecode. And what does that definition get compiled to? Basically the same thing as this:
spam = types.FunctionType( compile('return x+1\n', '__main__', mode='function'), globals(), 'spam', (), ()) spam.__qualname__ = 'spam'You can't quite write this, because the public interface for the
compile
function doesn't expose all of the necessary
features--but outside of that compile
, the rest is all
real Python. (For simple lambda
functions, you
can use, e.g., compile('1 + 2', '__main__', mode='eval')
and the whole thing is real Python, but that doesn't work for
def
functions. When you really need to create code
objects, there are ways to do it, but you very rarely need to, so
let's not worry about that.)If you put the same thing in a module instead of typing it at the REPL, the only difference is that the body is
compile
d
ahead of time and stored in a marshaled code object inside the
.pyc
file so it never needs to be compiled again. The
def
statement is still compiled and then interpreted as
top-level module code that constructs a function on the fly out of
that code constant, every time you import the module..For a slightly more complicated example, consider this:
def add_one(x: int=0) -> int: """Add 1""" return x+1This is equivalent to:
spam = types.FunctionType( compile('return x+1\n', '__main__', mode='function'), globals(), 'add_one', (0,), # This is where default values go ()) spam.__qualname__ = 'add_one' spam.__doc__ = """Add 1""" adder.__annotations__ = {'return': 'int', 'x': 'int'}Notice that the default values are passed into that
FunctionType
constructor. That's why defaults are bound
in at the time the def
statement is executed, which is
how you can do tricks like using a dict crammed into a default value
as a persistent cache.Closures
The real point of functions always being created on the fly is that this means any function can be a closure--it can capture values from the environment that the function was defined in. The standard Lisp-style example looks like this:def make_adder(n): def adder(x): return x+n return adderThat's equivalent to:
adder = types.FunctionType( compile('return x+n', '__main__', mode='exec'), globals(), 'adder', (), (CellType(locals(), 'n'),)) # tuple of closure cells adder.__qualname__ = 'make_adder.<locals>.adder'So every time you call
make_adder
, you get back a new
adder
function, created on the fly, referencing the particular
n
local variable from that particular call to
make_adder
.(Unfortunately, I cheated a bit. Unlike function objects, and even code objects, you can't actually manually create closure cells like this. But you rarely want to. And if you ever do need it, you can just do a trivial
lambda
that captures n
and then
do a minor frame hack to get at the cell.)Even if you never want to do anything this Lispy, closures get used all over the place. For example, if you've ever written a Tk GUI, you may have done something like this in your
Frame
subclass:def __init__(self): Frame.__init__(self) self.hello_button = tkinter.Button( self, text='Hello', command=lambda: self.on_button_click(self.hello_button))That
lambda
is creating a new function that captures the
local self
variable so it can access
self.hello_button
whenever the button is clicked.(A
lambda
compiles in almost the same way as a
def
, except that it's a value in the middle of an
expression rather than a statement, and it doesn't have a name,
docstring, etc.).Another common way to write the same button is with
functools.partial
:self.hello_button = tkinter.Button( self, text='Hello', command=partial(self.on_button_click, self.hello_button)If Python didn't come with
partial
, we could easily write
it ourselves:def partial(func, *args, **kw): def wrapped(*more_args, **more_kw): return func(*args, *more_args, **kw, **more_kw) return wrappedThis is also how decorators work:
def simple_memo(func): cache = {} def wrapped(*args): args = tuple(args) if args not in cache: cache[args] = func(*args) return cache[args] return wrapped @simple_memo def fib(n): if n < 2: return n return fib(n-1) + fib(n-2)I've written a dumb exponenially-recursive Fibonacci function, and that
@simple_memo
magically turns it into a linear-time
function that takes a fraction of a second instead of hours. How does
this work? Simple: after the usual fib = types.FunctionType blah
blah
stuff, it does fib = simple_memo(fib)
. That's
it. Because function are already always created on the fly, decorators
don't need anything complicated.By the way, if you can follow everything above, you pretty much know all there is to know about dynamic higher-order programming, except for how the theory behind it maps to advanced math. (And that part is simple if you already know the math, but meaningless if you don't.) That's one of those things that sounds scary when functional programmers talk about it, but if you go from using higher-order functions to building them to understanding how the work before the theory, instead of going from theory to implementation to building to using, it's not actually hard.
Fake functions
Sometimes, you can describe what code should run when you callspam
, but it's not obvious how to construct a function
object that actually runs that code. Or it's easy to write the
closure, but hard to think about it when you later come back and read
it.In those cases, you can create a class with a
__call__
method, and it acts like a function. For example:class Adder: def __init__(self, n): self._n = n def __call__(self, x): return x+nAn
Adder(5)
object behaves almost identical to a
make_adder(5)
closure. It's just a matter of which one
you find more readable. Even for experienced Python programmers, the
answer is different in different cases, which is why you'll find both
techniques all over the stdlib and popular third-party modules.In fact,
functools.partial
isn't actually a closure, but
a class, like this:class partial: def __init__(self, func, *args, **kw): self.func, self.args, self.kw = func, args, kw def __call__(self, *more_args, **more_kw): return self.func(*self.args, *more_args, **self.kw, **more_kw)(Actually, the real
partial
has a lot of bells and
whistles. But it's still not that complicated. The docs link to the
source code, if you want to see it for yourself.)Methods
OK, so you can create functions on the fly; what about methods?Again, they're already always created on the fly, and once you understand how, you can probably do whatever it was you needed.
Let's look at an example:
class Spam: def __init__(self, x, y): self.x, self.y = x, y def eggs(self): return self.x + self.y spam = Spam() print(spam.eggs())The definition of
eggs
is compiled and interpreted
exactly the same as the definition of any other function. And the
result is just stored as a member of the Spam
class (see
the section on classes to see how that works), so when you write
Spam.eggs
you just get that function.This means that if you want to add a new method to a class, there's no special trick, you just do it:
def cheese(self): return self.x * self.y Spam.cheese = cheese print(spam.cheese())That's all it takes to add a method to a class dynamically.
But meanwhile, on the instance,
spam.eggs
is not
just a function, it's a bound method. Try
print(spam.eggs)
from the interactive REPL. A bound
method knows which instance it belongs to, so when you
call it, that instance can get passed as the self
argument.The details of how Python turns the function
Spam.eggs
into the bound method spam.eggs
are a bit complicated
(and I've already written a whole post about them), but we don't need
to know that here.Obviously, bound methods get created dynamically. Every time you do
spam.eggs
or Spam().cheese
or
string.ascii_letters.find
, you're getting a new bound
method.And if you want to create one manually, you can just call
types.MethodType(func, obj)
.So, if you want to add a new method
beans
to just the
spam
instance, without adding it to the Spam
class? Just construct the same bound method that Python would have
constructed for you whenever you looked up spam.beans
,
and store it there:def beans(self): return self.x / self.y spam.beans = types.MethodType(beans, spam)And now you know enough to implement Javascript-style object literals, or even prototype inheritance. Not that you should do either, but if you ever run into something that you really do want to do, that requires creating methods on the fly, either on classes or on instances, you can do it. Because creating methods on the fly is what Python always does.
Classes
What if we want to create a class dynamically?Well, I shouldn't have to tell you at this point. You're always creating classes dynamically.
Class definitions work a bit differently from function definitions. Let's start with a simple example again:
class Spam: z = 0 def __init__(self, x, y): self.x, self.y = x, y def eggs(self): return self.x + self.y + self.zFirst, Python interprets the class body the same as any other code, but it runs inside an empty environment. So, those
def
calls
create new functions named __init__
and eggs
in that empty environment, instead of at the global level. Then, it
dynamically creates a class object out of that environment, where
every function or other value that got created becomes a method or
class attribute on the class. The code goes something like this:_Spam_locals = {} exec('def __init__(self, x, y):\n self.x, ... blah blah ...\n', globals=globals(), locals=_Spam_locals) Spam = type('Spam', (object,), _Spam_locals)This is why you can't access the
Spam
class object inside
the class definition--because there is nothing named Spam
until
after Python calls type
and stores the result in
Spam
. (But of course you can access Spam
inside the methods; by the time those methods get called, it'll
exist.)So, what if you want to create some methods dynamically inside the class? No sweat. By the time it gets to calling
type
,
nobody can tell whether eggs
gets into the locals dict by
you calling def eggs(...):
or
eggs = fancy_higher_order_function(...)
, so they both do
the same thing.In fact, one idiom you'll see quite often in the stdlib is this:
def __add__(self, other): blah blah __radd__ = __add__This just makes
__radd__
another name for the same
method as __add__
.And yes, you can call
type
manually if you need to,
passing it any dict you want as an environment:def __init__(self, x, y): self.x, self.y = x, y Spam = type('Spam', (object,), {'z': 0, '__init__': __init__, 'eggs': lambda self: self.x + self.y + self.z}There are a few more details to classes. A slightly more complicated example covers most of them:
@decorate_my_class class Spam(metaclass=MetaSpam, Base1, Base2): """Look, I've got a doc string""" def __init__(self, x, y): self.x, self.y = x, yThis is equivalent to:
_Spam_locals = {} exec('def __init__(self, x, y):\n self.x, ... blah blah ...\n', globals=globals(), locals=_Spam_locals) Spam = MetaSpam('Spam', (Base1, Base2), _Spam_locals) Spam.__doc__ = """Look, I've got a doc string""" Spam = decorate_my_class(Spam)As you can see, if there's a metaclass, it gets called in place of
type
, and if there are base classes, they get passed in
place of object
, and docstrings and decorators work the
same way as in functions.There are a few more complexities with qualnames,
__slots__
,
closures (if you define a class inside a function), and the magic to
make super()
work, but this is almost everything.Remember from the last section how easy it is to add methods to a class? Often that's simpler than trying to programmatically generate methods from inside the class definition, or customize the class creation. (See
functools.total_ordering
for a nice
example.) But when you really do need a dynamically-created class for
some reason, it's easy.Generating code
Occasionally, no matter how hard you try, you just can't come up with any way to define or modify a function or class dynamically with your details crammed into the right place the right way, at least not readably. In that case, you can always fall back to generating, compiling, and executing source code.The simplest way to do this is to just build a string and call
exec
on it. You can find a few examples of this in the
stdlib, like collections.namedtuple
. (Notice the trick it
uses of calling exec
in a custom empty namespace, then
copying the value out of it. This is a bit cleaner that just executing
in your own locals and/or globals.)You've probably hard "
exec
is dangerous". And of course
it is. But "dangerous" really just means two things: "powerful" and
"hard to control or reason about". When you need the first one badly
enough that you can accept the second, danger is justified. If you
don't understand things like how to make sure you're not letting data
some user sent to your web service end up inside your
exec
don't use it. But if you're still reading at this
point, I think you can learn how to reason through the issues.Sometimes you don't want to
exec
right now, you want to
compile
something that you can pass around and
exec
later. That works fine too.Sometimes, you even want to generate a whole module and save it as a
.py
file. Of course people do that kind of thing in C all
the time, as part of the build process--but in Python, it doesn't
matter whether you're generating a .py
file during a
build or install to be used later, or generating one at normal runtime
to be used right now; they're the same case as far as Python is
concerned.Sometimes you want to build an AST (abstract source tree) or a stream of tokens instead of source code, and then compile or execute that. (And this is a great time to yet again plug
macropy
, one of the coolest projects ever.)Sometimes you even want to do the equivalent of inline assembly (but assembling Python bytecode, not native machine code, of course) with a library like
byteplay
or cpyasm
(or, if
you're a real masochist, just assembling it in your head and using
struct.pack
on the resulting array of 16-bit
ints...). Again, unlike C, you can do this at runtime, then wrap that
code object up in a function object, and call it right now.You can even do stuff like marshaling code objects to and from a database to build functions out of later.
Conclusion
Because almost all of this is accessible from within Python itself, and all of it is inherently designed to be executed on the fly, almost anything you can think of is probably doable.So, if you're thinking "I need to dynamically create X", you need to think through exactly what you need, but whether that turns out to be "just a normal function" or something deeply magical, you'll be able to do it, or at least explain the magic you're looking for in specific enough terms that someone can actually show you how to do it.
View comments