What is this?

PyCLOS is a small(ish) library that adds a version of Common Lisp's method dispatch system to Python. PyCLOS makes your object-oriented code cleaner, more reusable, and easier to modify or extend. Finally, inheritance done right!

CLOS methods allow you to cleanly handle situations where a class and all its descendants need to inherit behaviour from the base class in a consistent manner. For example, you might have a method that needs to perform cleanup operations after it executes. In normal Python, you'd have to trust that every subclass either has the correct code copy-pasted in, or calls super() in a consistent manner. However, with CLOS methods, you can break up a method's definition into separate parts:

class Database:
    @clos.method
    def add_people(self, names):
        for name in names:
            self.file.write(name + '\n')

    # Need to make sure this lock is held by all subclasses?
    # Just write that code once in a different method part!
    @add_people.around
    def add_people(self, names):
        with self.lock.acquire():
            yield call_next(names)

class SQLDatabase(Database):
    @add_people.impl
    def add_people(self, names):
        for name in names:
            self.sql.exec('INSERT INTO people (name) VALUES (?)', name)
    # No messy code duplication!

Each method part is overriden separately by subclasses, making it easy to change only the functionality you actually need to. Plus, you can customize the way their part definitions are dispatched and combined—for example, you can have a method whose result is appended to the results from its base classes:

class Container:
    @clos.method(type=ListMethod)
    def get_contents(self):
        return self.__internal_contents

class ExtraStuff(Container):
    @get_contents.impl
    def get_contents(self):
        return ['extra1', 'extra2', 'extra3']
    # No ugly super() call!

Or anything the dark recesses of your mind can conjure!

Downloads

PyCLOS is licensed under the GNU GPLv3. You are allowed to use, modify, and distribute it, as long as you make the source code publicly available.

v1.0 (Latest)

• Source code tarball

How this is possible: The SHOCKING TRUTH Guido van Rossum doesn't want you to know!

At the most basic level, PyCLOS works by converting a normal Python method in a class to an object with a custom implementation of __call__. By storing all the method's definitions in this object, alongside the classes they came from, it can implement different behaviour than what Python has by default.

But to understand PyCLOS in more detail, we must first understand how Python itself handles method dispatch and inheritance. Methods are just functions stored in a class's attribute dictionary, so they are looked up just like any other attribute. If Python cannot find an attribute on the instance itself, it will search the attribute dictionary of the instance's class and everything it inherits from. To know which classes to search, and what order to search them in, Python calculates a sorted list of the class's ancestors, called its MRO. This allows Python to maintain certain invariants when a class inherits from multiple bases; however, a subclass's MRO might not match its base class's MRO. So we can't just naively look up attributes on the base class and expect that to work correctly.

This brings us to super(). It's not like the other girls. Calling super() constructs a proxy object around the instance, which uses the correct MRO, but skips all entries up to and including the class the super() call came from. But there's a problem with this: it's impossible for a function to determine that on its own without inspecting the stack trace—so Python cheats. It lies to you, and does its best to prevent you from ever discovering the truth. The zero-argument form of super() isn't real. The Python bytecode compiler automatically passes in self and the lexical class (which it has to create an implicit closure for! Why the hell can't I do that?!)—but in a weird, different way that only the built-in implementation of super() can access. Attempting to replace it with your own will result in those parameters seemingly being missing. Even writing raw Python bytecode by hand to access them didn't work when I tried it. There is simply no way to hijack or replicate the zero-argument form of super()... or is there?

We can't call our own code and implicitly pass it the lexical class at the call site... but who said we need to call anything? Why not simply pass control back up to the dispatch function, who knows which class the function it just called was defined in? Luckily, Python has a feature that lets us transfer control out of a function, then resume it at a later point: the yield keyword. It was meant for writing iterators, but Guido can't tell me what to do. You can even "return" a value to the yielding function—it's perfect!

There's one other little wrinkle: how does PyCLOS know what class a method definition belongs to? That information isn't passed to it, and at the time the decorator is called, the class doesn't even exist! We could work around this with metaclass shenanigans, and PyCLOS does provide a metaclass, but all it does is provide some small quality-of-life enhancements—CLOS methods work completely fine without it!

The secret lies in a lesser-known "dunder" method: __set_name__. I can only presume this one (along with __get__ and __set__) was invented as a result of the Python maintainers all getting drunk together and reading the INTERCAL Reference Manual...

Anyways, __set_name__ is part of the class creation process—after everything in the class body has been executed and the class has been brought into existence, Python scans the class dictionary for any objects with a __set_name__ method, and calls it with a reference to the freshly-baked class (still piping hot!) and the name of the attribute it's assigned to. So PyCLOS method objects simply hold on to definitions they don't yet know the class for, then register them all to whichever lucky class is the next to call __set_name__.