The call order of python3 metaclass











up vote
3
down vote

favorite
1












I am in confusion when trying to understand the order that metaclass creates a class instance. According to this diagram, enter image description here



I type the following codes to verify it.



class Meta(type):
def __call__(self):
print("Meta __call__")
super(Meta, self).__call__()

def __new__(mcs, name, bases, attrs, **kwargs):
print("Meta __new__")
return super().__new__(mcs, name, bases, kwargs)

def __prepare__(msc, name, **kwargs):
print("Meta __prepare__")
return {}

class SubMeta(Meta):
def __call__(self):
print("SubMeta __call__!")
super().__call__()

def __new__(mcs, name, bases, attrs, **kwargs):
print("SubMeta __new__")
return super().__new__(mcs, name, bases, kwargs)

def __prepare__(msc, name, **kwargs):
print("SubMeta __prepare__")
return Meta.__prepare__(name, kwargs)

class B(metaclass = SubMeta):
pass

b = B()


However, the result seems not like this follow.



SubMeta __prepare__
Meta __prepare__
SubMeta __new__
Meta __new__
SubMeta __call__!
Meta __call__


Any help will be appreciated.










share|improve this question
























  • Where did you get that diagram? Google image search didn't turn up anything like it.
    – user2357112
    Nov 12 at 3:56










  • For those arriving at this question trying to understand meta-classes in Python, have a look at stackoverflow.com/questions/17801344/… instead.
    – flindeberg
    Nov 12 at 8:13















up vote
3
down vote

favorite
1












I am in confusion when trying to understand the order that metaclass creates a class instance. According to this diagram, enter image description here



I type the following codes to verify it.



class Meta(type):
def __call__(self):
print("Meta __call__")
super(Meta, self).__call__()

def __new__(mcs, name, bases, attrs, **kwargs):
print("Meta __new__")
return super().__new__(mcs, name, bases, kwargs)

def __prepare__(msc, name, **kwargs):
print("Meta __prepare__")
return {}

class SubMeta(Meta):
def __call__(self):
print("SubMeta __call__!")
super().__call__()

def __new__(mcs, name, bases, attrs, **kwargs):
print("SubMeta __new__")
return super().__new__(mcs, name, bases, kwargs)

def __prepare__(msc, name, **kwargs):
print("SubMeta __prepare__")
return Meta.__prepare__(name, kwargs)

class B(metaclass = SubMeta):
pass

b = B()


However, the result seems not like this follow.



SubMeta __prepare__
Meta __prepare__
SubMeta __new__
Meta __new__
SubMeta __call__!
Meta __call__


Any help will be appreciated.










share|improve this question
























  • Where did you get that diagram? Google image search didn't turn up anything like it.
    – user2357112
    Nov 12 at 3:56










  • For those arriving at this question trying to understand meta-classes in Python, have a look at stackoverflow.com/questions/17801344/… instead.
    – flindeberg
    Nov 12 at 8:13













up vote
3
down vote

favorite
1









up vote
3
down vote

favorite
1






1





I am in confusion when trying to understand the order that metaclass creates a class instance. According to this diagram, enter image description here



I type the following codes to verify it.



class Meta(type):
def __call__(self):
print("Meta __call__")
super(Meta, self).__call__()

def __new__(mcs, name, bases, attrs, **kwargs):
print("Meta __new__")
return super().__new__(mcs, name, bases, kwargs)

def __prepare__(msc, name, **kwargs):
print("Meta __prepare__")
return {}

class SubMeta(Meta):
def __call__(self):
print("SubMeta __call__!")
super().__call__()

def __new__(mcs, name, bases, attrs, **kwargs):
print("SubMeta __new__")
return super().__new__(mcs, name, bases, kwargs)

def __prepare__(msc, name, **kwargs):
print("SubMeta __prepare__")
return Meta.__prepare__(name, kwargs)

class B(metaclass = SubMeta):
pass

b = B()


However, the result seems not like this follow.



SubMeta __prepare__
Meta __prepare__
SubMeta __new__
Meta __new__
SubMeta __call__!
Meta __call__


Any help will be appreciated.










share|improve this question















I am in confusion when trying to understand the order that metaclass creates a class instance. According to this diagram, enter image description here



I type the following codes to verify it.



class Meta(type):
def __call__(self):
print("Meta __call__")
super(Meta, self).__call__()

def __new__(mcs, name, bases, attrs, **kwargs):
print("Meta __new__")
return super().__new__(mcs, name, bases, kwargs)

def __prepare__(msc, name, **kwargs):
print("Meta __prepare__")
return {}

class SubMeta(Meta):
def __call__(self):
print("SubMeta __call__!")
super().__call__()

def __new__(mcs, name, bases, attrs, **kwargs):
print("SubMeta __new__")
return super().__new__(mcs, name, bases, kwargs)

def __prepare__(msc, name, **kwargs):
print("SubMeta __prepare__")
return Meta.__prepare__(name, kwargs)

class B(metaclass = SubMeta):
pass

b = B()


However, the result seems not like this follow.



SubMeta __prepare__
Meta __prepare__
SubMeta __new__
Meta __new__
SubMeta __call__!
Meta __call__


Any help will be appreciated.







python metaclass






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Nov 11 at 5:17

























asked Nov 11 at 5:10









return long

286




286












  • Where did you get that diagram? Google image search didn't turn up anything like it.
    – user2357112
    Nov 12 at 3:56










  • For those arriving at this question trying to understand meta-classes in Python, have a look at stackoverflow.com/questions/17801344/… instead.
    – flindeberg
    Nov 12 at 8:13


















  • Where did you get that diagram? Google image search didn't turn up anything like it.
    – user2357112
    Nov 12 at 3:56










  • For those arriving at this question trying to understand meta-classes in Python, have a look at stackoverflow.com/questions/17801344/… instead.
    – flindeberg
    Nov 12 at 8:13
















Where did you get that diagram? Google image search didn't turn up anything like it.
– user2357112
Nov 12 at 3:56




Where did you get that diagram? Google image search didn't turn up anything like it.
– user2357112
Nov 12 at 3:56












For those arriving at this question trying to understand meta-classes in Python, have a look at stackoverflow.com/questions/17801344/… instead.
– flindeberg
Nov 12 at 8:13




For those arriving at this question trying to understand meta-classes in Python, have a look at stackoverflow.com/questions/17801344/… instead.
– flindeberg
Nov 12 at 8:13












2 Answers
2






active

oldest

votes

















up vote
2
down vote



accepted










Despite @torek's lenghty answer, with a lot of other details on class creation, what you brought together to this question is mostly correct.



The only thing that is wrong in your code, which propably puzzled you is that te class you call Meta have to be itself the metaclass from SubMeta and not its parent.



Simply change Submeta declaration to:



class SubMeta(type, metaclass=Meta):
...


(No need for it to inherit from "Meta" as well - it can derive only from type. It is otherwise though to think of a customization to type.__call__ that would be usefull at the same time for creating instances of your classes (that is when SubMeta.__call__ is called), and your classes themselves (Meta.__call__ called))



Here is another, shorter example I just typed at the terminal. Sorry for the naming inconsistencies, and for being less complete - but it shows the main point:



class M(type):
def __call__(mmcls, *args, **kwargs):
print("M's call", args, kwargs)
return super().__call__(*args, **kwargs)

class MM(type, metaclass=M):
def __prepare__(cls, *args, **kw):
print("MM Prepare")
return {}
def __new__(mcls, *args, **kw):
print("MM __new__")
return super().__new__(mcls, *args, **kw)

class klass(metaclass=MM):
pass


Upon processing the klass body, Python output was:



MM Prepare
M's call ('klass', (), {'__module__': '__main__', '__qualname__': 'klass'}) {}
MM __new__


Moreover



As you can see from this, with a meta-meta class it is possible to customize the call order and parameters to the metaclass __init__ and __new__, but there are still steps that can't be customized from pure-Python code, and would require native calls to API's (and possibly raw object structure manipulation) - that are:




  • One can't control the call to __prepare__

  • One can't control the call to __init_subclass__ on the created classes

  • One can control when descriptors' __set_name__ are called


The last two items take place after meta-meta's __call__ return, and before resuming the flow to the module where the class module is.






share|improve this answer






























    up vote
    3
    down vote













    The trick, identified



    Update 2: Based on behavior, the fact that M0.__call__ is called below must be a side effect of this line in builtin__build_class in the CPython source (Python/bltinmodule.c).



    In order to define a class that has a metaclass, we call the metaclass's __prepare__, __new__, and __init__ as usual. This creates a class—in the example below, Meta—that is callable, but its internal PyFunction_GET_CODE slot points not to its own __call__ but rather to its metaclass's __call__. Hence if we call Meta() (the metaclass object), we invoke M0.__call__:



    print("call Meta")
    print("Meta returns:", Meta('name', (), {}))
    print("finished calling Meta")


    produces:



    call Meta
    M0 __call__: mmcls=<class '__main__.Meta'>, args=('name', (), {}), kwargs={}
    Meta __new__: mcs=<class '__main__.Meta'>, name='name', bases=(), attrs={}, kwargs={}
    Meta __init__: mcs=<class '__main__.name'>, name='name', bases=(), attrs={}, kwargs={}
    Meta returns: <class '__main__.name'>
    finished calling Meta


    In other words, we see that Meta acts like type, but it (rather magically and not very well documented) invokes M0.__call__. This is no doubt due to looking up __call__ in the class's type, rather than in an instance of the class (and indeed there is no instance except for the one we're creating). This is in fact the general case: it falls out of the fact that we call __call__ on the type of Meta, and the type of Meta is M0:



    print("type(Meta) =", type(Meta))


    prints:



    type(Meta) = <class '__main__.M0'>


    which explains where this comes from. (I still think this should be emphasized in the documentation, which also should describe the constraints on metaclass typing—these are enforced in _calculate_winner in Lib/types.py and, as C code, in _PyType_CalculateMetaclass in Objects/typeobject.c.)



    Updated original answer



    I don't know where your diagram came from, but it's wrong. UPDATE: You can in fact have a metaclass for your metaclass; see jsbueno's answer, and I've updated the example below. New sentences / text are in bold, except for the final section describing my puzzlement at the apparent lack of documentation.



    Your existing metaclass code has at least one error. Most significantly, its __prepare__ needs to be a class-method. See also Using the __call__ method of a metaclass instead of __new__? and PEP 3115. And, to use a meta-meta-class, your metaclass needs to have a metaclass of its own, not a base class.



    Chris's answer contains correct definitions. But there are some unfortunate asymmetries between metaclass method arguments and class method arguments, which I'll illustrate below.



    One other thing that may help: note that the metaclass __prepare__ method is called before creating any instances of class B: it is called when class B itself is being defined. To show this, here is a corrected metaclass-and-class. I have also added a few more illustrators. I've added a meta-metaclass as well, based on jsbueno's answer. I cannot find formal Python documentation on this, but I've updated the output below.



    class M0(type):
    def __call__(mmcls, *args, **kwargs):
    print("M0 __call__: mmcls={!r}, "
    "args={!r}, kwargs={!r}".format(mmcls, args, kwargs))
    return super().__call__(*args, **kwargs)

    class Meta(type, metaclass=M0):
    def __call__(cls, *args, **kwargs):
    print("Meta __call__: cls={!r}, "
    "args={!r}, kwargs={!r}".format(cls, args, kwargs))
    return super().__call__(*args, **kwargs)

    def __new__(mcs, name, bases, attrs, **kwargs):
    print("Meta __new__: mcs={!r}, name={!r}, bases={!r}, "
    "attrs={!r}, kwargs={!r}".format(mcs, name, bases, attrs, kwargs))
    return super().__new__(mcs, name, bases, attrs)

    def __init__(mcs, name, bases, attrs, **kwargs):
    print("Meta __init__: mcs={!r}, name={!r}, bases={!r}, "
    "attrs={!r}, kwargs={!r}".format(mcs, name, bases, attrs, kwargs))
    super().__init__(name, bases, attrs, **kwargs)

    @classmethod
    def __prepare__(cls, name, bases, **kwargs):
    print("Meta __prepare__: name={!r}, "
    "bases={!r}, kwargs={!r}".format(name, bases, kwargs))
    return {}

    print("about to create class A")
    class A(metaclass=Meta): pass
    print("finished creating class A")

    print("about to create class B")

    class B(A, metaclass=Meta, foo=3):
    @staticmethod
    def __new__(cls, *args, **kwargs):
    print("B __new__: cls={!r}, "
    "args={!r}, kwargs={!r}".format(cls, args, kwargs))
    return super().__new__(cls)

    def __init__(self, *args, **kwargs):
    print("B __init__: args={!r}, kwargs={!r}, ".format(args, kwargs))

    print("finished creating class B")

    print("about to create instance b = B()")
    b = B('hello', bar=7)
    print("finished creating instance b")


    Now, let's observe what happens when I run this, and take each piece apart:



    $ python3.6 meta.py
    about to create class A
    Meta __prepare__: name='A', bases=(), kwargs={}
    M0 __call__: mmcls=<class '__main__.Meta'>, args=('A', (), {'__module__': '__main__', '__qualname__': 'A'}), kwargs={}
    Meta __new__: mcs=<class '__main__.Meta'>, name='A', bases=(), attrs={'__module__': '__main__', '__qualname__': 'A'}, kwargs={}
    Meta __init__: mcs=<class '__main__.A'>, name='A', bases=(), attrs={'__module__': '__main__', '__qualname__': 'A'}, kwargs={}
    finished creating class A


    To create class A itself, Python first calls the metaclass's __prepare__, delivering it the name of the class (A), the list of base classes (an empty tuple—it's called a list but is actually a tuple), and any keyword arguments (none). As PEP 3115 notes, the metaclass needs to return a dictionary or dict-like object; this one does by just returning an empty dictionary, so we are good here.



    (I don't print cls itself here, but if you do, you will see it is just <class '__main__.Meta'>.)



    Next, having gotten a dictionary from __prepare__, Python first calls the meta-meta __call__, i.e., M0.__call__, passing the entire set of arguments as the args tuple. It then populates the __prepare__-supplied dictionary with all the attributes for the class, passing this as the attrs to the metaclass __new__ and __init__. If you print the id of the dictionary returned from __prepare__ and passed to __new__ and __init__ you will see they all match.



    Since class A has no methods or data members, we see only the magic __module__ and __qualname__ attributes here. We also see no keyword arguments, so now let's move on to creating class B:



    about to create class B
    Meta __prepare__: name='B', bases=(<class '__main__.A'>,), kwargs={'foo': 3}
    M0 __call__: mmcls=<class '__main__.Meta'>, args=('B', (<class '__main__.A'>,), {'__module__': '__main__', '__qualname__': 'B', '__new__': <staticmethod object at 0x800ad0a58>, '__init__': <function B.__init__ at 0x800ad2840>, '__classcell__': <cell at 0x800a749d8: empty>}), kwargs={'foo': 3}
    Meta __new__: mcs=<class '__main__.Meta'>, name='B', bases=(<class '__main__.A'>,), attrs={'__module__': '__main__', '__qualname__': 'B', '__new__': <staticmethod object at 0x800ad0940>, '__init__': <function B.__init__ at 0x800ad27b8>, '__classcell__': <cell at 0x800a745b8: empty>}, kwargs={'foo': 3}
    Meta __init__: mcs=<class '__main__.B'>, name='B', bases=(<class '__main__.A'>,), attrs={'__module__': '__main__', '__qualname__': 'B', '__new__': <staticmethod object at 0x800ad0940>, '__init__': <function B.__init__ at 0x800ad27b8>, '__classcell__': <cell at 0x800a745b8: Meta object at 0x802047018>}, kwargs={'foo': 3}
    finished creating class B


    This one is rather more interesting. Now we have one base class, namely __main__.A. Class B also defines several methods (__new__ and __init__) and we see them in the attrs dictionaries passed to the metaclass __new__ and __init__ methods (which, remember, are just the now-populated dictionary returned by the metaclass's __prepare__). As before, the passing-on happens through the meta-meta-class M0.__call__. We also see one keyword argument throughout, {'foo': 3}. In the attribute dictionary, we can also observe the magic __classcell__ entry: see Provide __classcell__ example for Python 3.6 metaclass for a short description as to what this is about, but to be, er, super-short, it's for making super() work.



    The keyword argument is passed to all three metaclass methods, plus that of the meta-meta-class. (I'm not quite sure why. Note that modifying the dictionary in any metaclass method does not affect it in any other, as it's a copy each time of the original keyword arguments. However, we can modify it in the meta-meta-class: add kwargs.pop('foo', None) to M0.__call__ to observe this.)



    Now that we have our classes A and B, we can get on to the process of creating an actual instance of class B. Now we see the metaclass's __call__ invoked (not the meta-meta-class's):



    about to create instance b = B()
    Meta __call__: cls=<class '__main__.B'>, args=('hello',), kwargs={'bar': 7}


    It's possible to change the args or kwargs passed on, but I don't; the sample code above winds up calling type.__call__(cls, *args, **kwargs) (through the magic of super().__call__). This in turn calls B.__new__ and B.__init__:



    B __new__: cls=<class '__main__.B'>, args=('hello',), kwargs={'bar': 7}
    B __init__: args=('hello',), kwargs={'bar': 7},
    finished creating instance b


    which finishes the realization of the new instance of class B, which we then bind to the name b.



    Note that B.__new__ says:



    return super().__new__(cls)


    so we invoke object.__new__ to create the instance—this is more or less a requirement of all versions of Python; you can only "cheat" when you return a singleton instance (ideally, one that's non-modifiable). It's type.__call__ that calls B.__init__ on this object, passing the arguments and keyword-arguments we passed it. If we replace Meta's __call__ with:



        def __call__(cls, *args, **kwargs):
    print("Meta __call__: cls={!r}, "
    "args={!r}, kwargs={!r}".format(cls, args, kwargs))
    return object.__new__(cls)


    we will see that B.__new__ and B.__init__ are never called:



    about to create instance b = B()
    Meta __call__: cls=<class '__main__.B'>, args=('hello',), kwargs={'bar': 7}
    finished creating instance b


    This would, in effect, create a useless/uninitialized instance b. It's therefore critical that the metaclass __call__ method call the underlying class's __init__, usually by invoking type.__call__ via super().__call__. If the underlying class has a __new__, the metaclass should call that first, again usually by invoking type.__call__.



    Side note: what the documentation says



    To quote section 3.3.3.6:




    Once the class namespace has been populated by executing the class body, the class object is created by calling metaclass(name, bases, namespace, **kwds) (the additional keywords passed here are the same as those passed to __prepare__).




    This explains the call to Meta.__call__ when creating b as an instance of class B, but not the fact that Python first calls M0.__call__ before calling Meta.__new__ and Meta.__init__ when creating classes A and B themselves.



    The next paragraph mentions the __classcell__ entry; the one after that goes on to describe the use of __set_name__ and __init_subclass__ hooks. Nothing here tells us how or why Python calls M0.__call__ at this point.



    Earlier, in sections 3.3.3.3 through 3.3.3.5, the documentation describes the process of determining the metaclass, preparing the class namespace, and executing the class body. This is where the meta-metaclass action should be described, but isn't.



    Several additional sections describe a few additional constraints. One important one is 3.3.10, which talks about how special methods are found via the object type, bypassing both regular member attribute lookups and even (sometimes) a metaclass getattribute, saying:




    Bypassing the __getattribute__() machinery in this fashion provides significant scope for speed optimisations within the interpreter, at the cost of some flexibility in the handling of special methods (the special method must be set on the class object itself in order to be consistently invoked by the interpreter).




    Update 2: This is really the secret of the trick: the special __call__ method is found via the type's type. If the metaclass has a metaclass, the meta-meta-class provides the __call__ slot; otherwise the type of the metaclass is type, so that the __call__ slot is type.__call__.






    share|improve this answer























    • Despite your lenghty answer, and rather usefull information on the process of class creation, you did not touch the subject of the "meta-meta-class" call method, the OP is in doubt about. In fact, there is only a simple mistake in the OP's code - and the diagram posted in the question is actually correct.
      – jsbueno
      Nov 12 at 3:53










    • While I'm not sure whether I agree with jsbueno about what the mistake in the questioner's code is, I agree that you seem to have missed the meta-metaclass aspect, and that the diagram in the question is correct.
      – user2357112
      Nov 12 at 3:58










    • @jsbueno: interesting. I haven't seen this described elsewhere. Indeed, docs.python.org/3/reference/datamodel.html makes no mention of it while describing how regular metaclasses work.
      – torek
      Nov 12 at 7:06










    • @user2357112: indeed, testing shows it works—but there seems to be no hint of this in any documentation I can find.
      – torek
      Nov 12 at 8:01










    • Thanks a million for your detailed answer, I learned a lot. The diagram comes from the blog — understanding-python-metaclasses. Just like mentioned by jsbueno, Meta-metaclass should class SubMeta(type, metaclass=Meta): rather than class SubMeta(Meta):.
      – return long
      Nov 12 at 8:17











    Your Answer






    StackExchange.ifUsing("editor", function () {
    StackExchange.using("externalEditor", function () {
    StackExchange.using("snippets", function () {
    StackExchange.snippets.init();
    });
    });
    }, "code-snippets");

    StackExchange.ready(function() {
    var channelOptions = {
    tags: "".split(" "),
    id: "1"
    };
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function() {
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled) {
    StackExchange.using("snippets", function() {
    createEditor();
    });
    }
    else {
    createEditor();
    }
    });

    function createEditor() {
    StackExchange.prepareEditor({
    heartbeatType: 'answer',
    convertImagesToLinks: true,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: 10,
    bindNavPrevention: true,
    postfix: "",
    imageUploader: {
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    },
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    });


    }
    });














     

    draft saved


    draft discarded


















    StackExchange.ready(
    function () {
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53246017%2fthe-call-order-of-python3-metaclass%23new-answer', 'question_page');
    }
    );

    Post as a guest















    Required, but never shown

























    2 Answers
    2






    active

    oldest

    votes








    2 Answers
    2






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes








    up vote
    2
    down vote



    accepted










    Despite @torek's lenghty answer, with a lot of other details on class creation, what you brought together to this question is mostly correct.



    The only thing that is wrong in your code, which propably puzzled you is that te class you call Meta have to be itself the metaclass from SubMeta and not its parent.



    Simply change Submeta declaration to:



    class SubMeta(type, metaclass=Meta):
    ...


    (No need for it to inherit from "Meta" as well - it can derive only from type. It is otherwise though to think of a customization to type.__call__ that would be usefull at the same time for creating instances of your classes (that is when SubMeta.__call__ is called), and your classes themselves (Meta.__call__ called))



    Here is another, shorter example I just typed at the terminal. Sorry for the naming inconsistencies, and for being less complete - but it shows the main point:



    class M(type):
    def __call__(mmcls, *args, **kwargs):
    print("M's call", args, kwargs)
    return super().__call__(*args, **kwargs)

    class MM(type, metaclass=M):
    def __prepare__(cls, *args, **kw):
    print("MM Prepare")
    return {}
    def __new__(mcls, *args, **kw):
    print("MM __new__")
    return super().__new__(mcls, *args, **kw)

    class klass(metaclass=MM):
    pass


    Upon processing the klass body, Python output was:



    MM Prepare
    M's call ('klass', (), {'__module__': '__main__', '__qualname__': 'klass'}) {}
    MM __new__


    Moreover



    As you can see from this, with a meta-meta class it is possible to customize the call order and parameters to the metaclass __init__ and __new__, but there are still steps that can't be customized from pure-Python code, and would require native calls to API's (and possibly raw object structure manipulation) - that are:




    • One can't control the call to __prepare__

    • One can't control the call to __init_subclass__ on the created classes

    • One can control when descriptors' __set_name__ are called


    The last two items take place after meta-meta's __call__ return, and before resuming the flow to the module where the class module is.






    share|improve this answer



























      up vote
      2
      down vote



      accepted










      Despite @torek's lenghty answer, with a lot of other details on class creation, what you brought together to this question is mostly correct.



      The only thing that is wrong in your code, which propably puzzled you is that te class you call Meta have to be itself the metaclass from SubMeta and not its parent.



      Simply change Submeta declaration to:



      class SubMeta(type, metaclass=Meta):
      ...


      (No need for it to inherit from "Meta" as well - it can derive only from type. It is otherwise though to think of a customization to type.__call__ that would be usefull at the same time for creating instances of your classes (that is when SubMeta.__call__ is called), and your classes themselves (Meta.__call__ called))



      Here is another, shorter example I just typed at the terminal. Sorry for the naming inconsistencies, and for being less complete - but it shows the main point:



      class M(type):
      def __call__(mmcls, *args, **kwargs):
      print("M's call", args, kwargs)
      return super().__call__(*args, **kwargs)

      class MM(type, metaclass=M):
      def __prepare__(cls, *args, **kw):
      print("MM Prepare")
      return {}
      def __new__(mcls, *args, **kw):
      print("MM __new__")
      return super().__new__(mcls, *args, **kw)

      class klass(metaclass=MM):
      pass


      Upon processing the klass body, Python output was:



      MM Prepare
      M's call ('klass', (), {'__module__': '__main__', '__qualname__': 'klass'}) {}
      MM __new__


      Moreover



      As you can see from this, with a meta-meta class it is possible to customize the call order and parameters to the metaclass __init__ and __new__, but there are still steps that can't be customized from pure-Python code, and would require native calls to API's (and possibly raw object structure manipulation) - that are:




      • One can't control the call to __prepare__

      • One can't control the call to __init_subclass__ on the created classes

      • One can control when descriptors' __set_name__ are called


      The last two items take place after meta-meta's __call__ return, and before resuming the flow to the module where the class module is.






      share|improve this answer

























        up vote
        2
        down vote



        accepted







        up vote
        2
        down vote



        accepted






        Despite @torek's lenghty answer, with a lot of other details on class creation, what you brought together to this question is mostly correct.



        The only thing that is wrong in your code, which propably puzzled you is that te class you call Meta have to be itself the metaclass from SubMeta and not its parent.



        Simply change Submeta declaration to:



        class SubMeta(type, metaclass=Meta):
        ...


        (No need for it to inherit from "Meta" as well - it can derive only from type. It is otherwise though to think of a customization to type.__call__ that would be usefull at the same time for creating instances of your classes (that is when SubMeta.__call__ is called), and your classes themselves (Meta.__call__ called))



        Here is another, shorter example I just typed at the terminal. Sorry for the naming inconsistencies, and for being less complete - but it shows the main point:



        class M(type):
        def __call__(mmcls, *args, **kwargs):
        print("M's call", args, kwargs)
        return super().__call__(*args, **kwargs)

        class MM(type, metaclass=M):
        def __prepare__(cls, *args, **kw):
        print("MM Prepare")
        return {}
        def __new__(mcls, *args, **kw):
        print("MM __new__")
        return super().__new__(mcls, *args, **kw)

        class klass(metaclass=MM):
        pass


        Upon processing the klass body, Python output was:



        MM Prepare
        M's call ('klass', (), {'__module__': '__main__', '__qualname__': 'klass'}) {}
        MM __new__


        Moreover



        As you can see from this, with a meta-meta class it is possible to customize the call order and parameters to the metaclass __init__ and __new__, but there are still steps that can't be customized from pure-Python code, and would require native calls to API's (and possibly raw object structure manipulation) - that are:




        • One can't control the call to __prepare__

        • One can't control the call to __init_subclass__ on the created classes

        • One can control when descriptors' __set_name__ are called


        The last two items take place after meta-meta's __call__ return, and before resuming the flow to the module where the class module is.






        share|improve this answer














        Despite @torek's lenghty answer, with a lot of other details on class creation, what you brought together to this question is mostly correct.



        The only thing that is wrong in your code, which propably puzzled you is that te class you call Meta have to be itself the metaclass from SubMeta and not its parent.



        Simply change Submeta declaration to:



        class SubMeta(type, metaclass=Meta):
        ...


        (No need for it to inherit from "Meta" as well - it can derive only from type. It is otherwise though to think of a customization to type.__call__ that would be usefull at the same time for creating instances of your classes (that is when SubMeta.__call__ is called), and your classes themselves (Meta.__call__ called))



        Here is another, shorter example I just typed at the terminal. Sorry for the naming inconsistencies, and for being less complete - but it shows the main point:



        class M(type):
        def __call__(mmcls, *args, **kwargs):
        print("M's call", args, kwargs)
        return super().__call__(*args, **kwargs)

        class MM(type, metaclass=M):
        def __prepare__(cls, *args, **kw):
        print("MM Prepare")
        return {}
        def __new__(mcls, *args, **kw):
        print("MM __new__")
        return super().__new__(mcls, *args, **kw)

        class klass(metaclass=MM):
        pass


        Upon processing the klass body, Python output was:



        MM Prepare
        M's call ('klass', (), {'__module__': '__main__', '__qualname__': 'klass'}) {}
        MM __new__


        Moreover



        As you can see from this, with a meta-meta class it is possible to customize the call order and parameters to the metaclass __init__ and __new__, but there are still steps that can't be customized from pure-Python code, and would require native calls to API's (and possibly raw object structure manipulation) - that are:




        • One can't control the call to __prepare__

        • One can't control the call to __init_subclass__ on the created classes

        • One can control when descriptors' __set_name__ are called


        The last two items take place after meta-meta's __call__ return, and before resuming the flow to the module where the class module is.







        share|improve this answer














        share|improve this answer



        share|improve this answer








        edited Nov 12 at 12:39

























        answered Nov 12 at 3:51









        jsbueno

        54.4k673124




        54.4k673124
























            up vote
            3
            down vote













            The trick, identified



            Update 2: Based on behavior, the fact that M0.__call__ is called below must be a side effect of this line in builtin__build_class in the CPython source (Python/bltinmodule.c).



            In order to define a class that has a metaclass, we call the metaclass's __prepare__, __new__, and __init__ as usual. This creates a class—in the example below, Meta—that is callable, but its internal PyFunction_GET_CODE slot points not to its own __call__ but rather to its metaclass's __call__. Hence if we call Meta() (the metaclass object), we invoke M0.__call__:



            print("call Meta")
            print("Meta returns:", Meta('name', (), {}))
            print("finished calling Meta")


            produces:



            call Meta
            M0 __call__: mmcls=<class '__main__.Meta'>, args=('name', (), {}), kwargs={}
            Meta __new__: mcs=<class '__main__.Meta'>, name='name', bases=(), attrs={}, kwargs={}
            Meta __init__: mcs=<class '__main__.name'>, name='name', bases=(), attrs={}, kwargs={}
            Meta returns: <class '__main__.name'>
            finished calling Meta


            In other words, we see that Meta acts like type, but it (rather magically and not very well documented) invokes M0.__call__. This is no doubt due to looking up __call__ in the class's type, rather than in an instance of the class (and indeed there is no instance except for the one we're creating). This is in fact the general case: it falls out of the fact that we call __call__ on the type of Meta, and the type of Meta is M0:



            print("type(Meta) =", type(Meta))


            prints:



            type(Meta) = <class '__main__.M0'>


            which explains where this comes from. (I still think this should be emphasized in the documentation, which also should describe the constraints on metaclass typing—these are enforced in _calculate_winner in Lib/types.py and, as C code, in _PyType_CalculateMetaclass in Objects/typeobject.c.)



            Updated original answer



            I don't know where your diagram came from, but it's wrong. UPDATE: You can in fact have a metaclass for your metaclass; see jsbueno's answer, and I've updated the example below. New sentences / text are in bold, except for the final section describing my puzzlement at the apparent lack of documentation.



            Your existing metaclass code has at least one error. Most significantly, its __prepare__ needs to be a class-method. See also Using the __call__ method of a metaclass instead of __new__? and PEP 3115. And, to use a meta-meta-class, your metaclass needs to have a metaclass of its own, not a base class.



            Chris's answer contains correct definitions. But there are some unfortunate asymmetries between metaclass method arguments and class method arguments, which I'll illustrate below.



            One other thing that may help: note that the metaclass __prepare__ method is called before creating any instances of class B: it is called when class B itself is being defined. To show this, here is a corrected metaclass-and-class. I have also added a few more illustrators. I've added a meta-metaclass as well, based on jsbueno's answer. I cannot find formal Python documentation on this, but I've updated the output below.



            class M0(type):
            def __call__(mmcls, *args, **kwargs):
            print("M0 __call__: mmcls={!r}, "
            "args={!r}, kwargs={!r}".format(mmcls, args, kwargs))
            return super().__call__(*args, **kwargs)

            class Meta(type, metaclass=M0):
            def __call__(cls, *args, **kwargs):
            print("Meta __call__: cls={!r}, "
            "args={!r}, kwargs={!r}".format(cls, args, kwargs))
            return super().__call__(*args, **kwargs)

            def __new__(mcs, name, bases, attrs, **kwargs):
            print("Meta __new__: mcs={!r}, name={!r}, bases={!r}, "
            "attrs={!r}, kwargs={!r}".format(mcs, name, bases, attrs, kwargs))
            return super().__new__(mcs, name, bases, attrs)

            def __init__(mcs, name, bases, attrs, **kwargs):
            print("Meta __init__: mcs={!r}, name={!r}, bases={!r}, "
            "attrs={!r}, kwargs={!r}".format(mcs, name, bases, attrs, kwargs))
            super().__init__(name, bases, attrs, **kwargs)

            @classmethod
            def __prepare__(cls, name, bases, **kwargs):
            print("Meta __prepare__: name={!r}, "
            "bases={!r}, kwargs={!r}".format(name, bases, kwargs))
            return {}

            print("about to create class A")
            class A(metaclass=Meta): pass
            print("finished creating class A")

            print("about to create class B")

            class B(A, metaclass=Meta, foo=3):
            @staticmethod
            def __new__(cls, *args, **kwargs):
            print("B __new__: cls={!r}, "
            "args={!r}, kwargs={!r}".format(cls, args, kwargs))
            return super().__new__(cls)

            def __init__(self, *args, **kwargs):
            print("B __init__: args={!r}, kwargs={!r}, ".format(args, kwargs))

            print("finished creating class B")

            print("about to create instance b = B()")
            b = B('hello', bar=7)
            print("finished creating instance b")


            Now, let's observe what happens when I run this, and take each piece apart:



            $ python3.6 meta.py
            about to create class A
            Meta __prepare__: name='A', bases=(), kwargs={}
            M0 __call__: mmcls=<class '__main__.Meta'>, args=('A', (), {'__module__': '__main__', '__qualname__': 'A'}), kwargs={}
            Meta __new__: mcs=<class '__main__.Meta'>, name='A', bases=(), attrs={'__module__': '__main__', '__qualname__': 'A'}, kwargs={}
            Meta __init__: mcs=<class '__main__.A'>, name='A', bases=(), attrs={'__module__': '__main__', '__qualname__': 'A'}, kwargs={}
            finished creating class A


            To create class A itself, Python first calls the metaclass's __prepare__, delivering it the name of the class (A), the list of base classes (an empty tuple—it's called a list but is actually a tuple), and any keyword arguments (none). As PEP 3115 notes, the metaclass needs to return a dictionary or dict-like object; this one does by just returning an empty dictionary, so we are good here.



            (I don't print cls itself here, but if you do, you will see it is just <class '__main__.Meta'>.)



            Next, having gotten a dictionary from __prepare__, Python first calls the meta-meta __call__, i.e., M0.__call__, passing the entire set of arguments as the args tuple. It then populates the __prepare__-supplied dictionary with all the attributes for the class, passing this as the attrs to the metaclass __new__ and __init__. If you print the id of the dictionary returned from __prepare__ and passed to __new__ and __init__ you will see they all match.



            Since class A has no methods or data members, we see only the magic __module__ and __qualname__ attributes here. We also see no keyword arguments, so now let's move on to creating class B:



            about to create class B
            Meta __prepare__: name='B', bases=(<class '__main__.A'>,), kwargs={'foo': 3}
            M0 __call__: mmcls=<class '__main__.Meta'>, args=('B', (<class '__main__.A'>,), {'__module__': '__main__', '__qualname__': 'B', '__new__': <staticmethod object at 0x800ad0a58>, '__init__': <function B.__init__ at 0x800ad2840>, '__classcell__': <cell at 0x800a749d8: empty>}), kwargs={'foo': 3}
            Meta __new__: mcs=<class '__main__.Meta'>, name='B', bases=(<class '__main__.A'>,), attrs={'__module__': '__main__', '__qualname__': 'B', '__new__': <staticmethod object at 0x800ad0940>, '__init__': <function B.__init__ at 0x800ad27b8>, '__classcell__': <cell at 0x800a745b8: empty>}, kwargs={'foo': 3}
            Meta __init__: mcs=<class '__main__.B'>, name='B', bases=(<class '__main__.A'>,), attrs={'__module__': '__main__', '__qualname__': 'B', '__new__': <staticmethod object at 0x800ad0940>, '__init__': <function B.__init__ at 0x800ad27b8>, '__classcell__': <cell at 0x800a745b8: Meta object at 0x802047018>}, kwargs={'foo': 3}
            finished creating class B


            This one is rather more interesting. Now we have one base class, namely __main__.A. Class B also defines several methods (__new__ and __init__) and we see them in the attrs dictionaries passed to the metaclass __new__ and __init__ methods (which, remember, are just the now-populated dictionary returned by the metaclass's __prepare__). As before, the passing-on happens through the meta-meta-class M0.__call__. We also see one keyword argument throughout, {'foo': 3}. In the attribute dictionary, we can also observe the magic __classcell__ entry: see Provide __classcell__ example for Python 3.6 metaclass for a short description as to what this is about, but to be, er, super-short, it's for making super() work.



            The keyword argument is passed to all three metaclass methods, plus that of the meta-meta-class. (I'm not quite sure why. Note that modifying the dictionary in any metaclass method does not affect it in any other, as it's a copy each time of the original keyword arguments. However, we can modify it in the meta-meta-class: add kwargs.pop('foo', None) to M0.__call__ to observe this.)



            Now that we have our classes A and B, we can get on to the process of creating an actual instance of class B. Now we see the metaclass's __call__ invoked (not the meta-meta-class's):



            about to create instance b = B()
            Meta __call__: cls=<class '__main__.B'>, args=('hello',), kwargs={'bar': 7}


            It's possible to change the args or kwargs passed on, but I don't; the sample code above winds up calling type.__call__(cls, *args, **kwargs) (through the magic of super().__call__). This in turn calls B.__new__ and B.__init__:



            B __new__: cls=<class '__main__.B'>, args=('hello',), kwargs={'bar': 7}
            B __init__: args=('hello',), kwargs={'bar': 7},
            finished creating instance b


            which finishes the realization of the new instance of class B, which we then bind to the name b.



            Note that B.__new__ says:



            return super().__new__(cls)


            so we invoke object.__new__ to create the instance—this is more or less a requirement of all versions of Python; you can only "cheat" when you return a singleton instance (ideally, one that's non-modifiable). It's type.__call__ that calls B.__init__ on this object, passing the arguments and keyword-arguments we passed it. If we replace Meta's __call__ with:



                def __call__(cls, *args, **kwargs):
            print("Meta __call__: cls={!r}, "
            "args={!r}, kwargs={!r}".format(cls, args, kwargs))
            return object.__new__(cls)


            we will see that B.__new__ and B.__init__ are never called:



            about to create instance b = B()
            Meta __call__: cls=<class '__main__.B'>, args=('hello',), kwargs={'bar': 7}
            finished creating instance b


            This would, in effect, create a useless/uninitialized instance b. It's therefore critical that the metaclass __call__ method call the underlying class's __init__, usually by invoking type.__call__ via super().__call__. If the underlying class has a __new__, the metaclass should call that first, again usually by invoking type.__call__.



            Side note: what the documentation says



            To quote section 3.3.3.6:




            Once the class namespace has been populated by executing the class body, the class object is created by calling metaclass(name, bases, namespace, **kwds) (the additional keywords passed here are the same as those passed to __prepare__).




            This explains the call to Meta.__call__ when creating b as an instance of class B, but not the fact that Python first calls M0.__call__ before calling Meta.__new__ and Meta.__init__ when creating classes A and B themselves.



            The next paragraph mentions the __classcell__ entry; the one after that goes on to describe the use of __set_name__ and __init_subclass__ hooks. Nothing here tells us how or why Python calls M0.__call__ at this point.



            Earlier, in sections 3.3.3.3 through 3.3.3.5, the documentation describes the process of determining the metaclass, preparing the class namespace, and executing the class body. This is where the meta-metaclass action should be described, but isn't.



            Several additional sections describe a few additional constraints. One important one is 3.3.10, which talks about how special methods are found via the object type, bypassing both regular member attribute lookups and even (sometimes) a metaclass getattribute, saying:




            Bypassing the __getattribute__() machinery in this fashion provides significant scope for speed optimisations within the interpreter, at the cost of some flexibility in the handling of special methods (the special method must be set on the class object itself in order to be consistently invoked by the interpreter).




            Update 2: This is really the secret of the trick: the special __call__ method is found via the type's type. If the metaclass has a metaclass, the meta-meta-class provides the __call__ slot; otherwise the type of the metaclass is type, so that the __call__ slot is type.__call__.






            share|improve this answer























            • Despite your lenghty answer, and rather usefull information on the process of class creation, you did not touch the subject of the "meta-meta-class" call method, the OP is in doubt about. In fact, there is only a simple mistake in the OP's code - and the diagram posted in the question is actually correct.
              – jsbueno
              Nov 12 at 3:53










            • While I'm not sure whether I agree with jsbueno about what the mistake in the questioner's code is, I agree that you seem to have missed the meta-metaclass aspect, and that the diagram in the question is correct.
              – user2357112
              Nov 12 at 3:58










            • @jsbueno: interesting. I haven't seen this described elsewhere. Indeed, docs.python.org/3/reference/datamodel.html makes no mention of it while describing how regular metaclasses work.
              – torek
              Nov 12 at 7:06










            • @user2357112: indeed, testing shows it works—but there seems to be no hint of this in any documentation I can find.
              – torek
              Nov 12 at 8:01










            • Thanks a million for your detailed answer, I learned a lot. The diagram comes from the blog — understanding-python-metaclasses. Just like mentioned by jsbueno, Meta-metaclass should class SubMeta(type, metaclass=Meta): rather than class SubMeta(Meta):.
              – return long
              Nov 12 at 8:17















            up vote
            3
            down vote













            The trick, identified



            Update 2: Based on behavior, the fact that M0.__call__ is called below must be a side effect of this line in builtin__build_class in the CPython source (Python/bltinmodule.c).



            In order to define a class that has a metaclass, we call the metaclass's __prepare__, __new__, and __init__ as usual. This creates a class—in the example below, Meta—that is callable, but its internal PyFunction_GET_CODE slot points not to its own __call__ but rather to its metaclass's __call__. Hence if we call Meta() (the metaclass object), we invoke M0.__call__:



            print("call Meta")
            print("Meta returns:", Meta('name', (), {}))
            print("finished calling Meta")


            produces:



            call Meta
            M0 __call__: mmcls=<class '__main__.Meta'>, args=('name', (), {}), kwargs={}
            Meta __new__: mcs=<class '__main__.Meta'>, name='name', bases=(), attrs={}, kwargs={}
            Meta __init__: mcs=<class '__main__.name'>, name='name', bases=(), attrs={}, kwargs={}
            Meta returns: <class '__main__.name'>
            finished calling Meta


            In other words, we see that Meta acts like type, but it (rather magically and not very well documented) invokes M0.__call__. This is no doubt due to looking up __call__ in the class's type, rather than in an instance of the class (and indeed there is no instance except for the one we're creating). This is in fact the general case: it falls out of the fact that we call __call__ on the type of Meta, and the type of Meta is M0:



            print("type(Meta) =", type(Meta))


            prints:



            type(Meta) = <class '__main__.M0'>


            which explains where this comes from. (I still think this should be emphasized in the documentation, which also should describe the constraints on metaclass typing—these are enforced in _calculate_winner in Lib/types.py and, as C code, in _PyType_CalculateMetaclass in Objects/typeobject.c.)



            Updated original answer



            I don't know where your diagram came from, but it's wrong. UPDATE: You can in fact have a metaclass for your metaclass; see jsbueno's answer, and I've updated the example below. New sentences / text are in bold, except for the final section describing my puzzlement at the apparent lack of documentation.



            Your existing metaclass code has at least one error. Most significantly, its __prepare__ needs to be a class-method. See also Using the __call__ method of a metaclass instead of __new__? and PEP 3115. And, to use a meta-meta-class, your metaclass needs to have a metaclass of its own, not a base class.



            Chris's answer contains correct definitions. But there are some unfortunate asymmetries between metaclass method arguments and class method arguments, which I'll illustrate below.



            One other thing that may help: note that the metaclass __prepare__ method is called before creating any instances of class B: it is called when class B itself is being defined. To show this, here is a corrected metaclass-and-class. I have also added a few more illustrators. I've added a meta-metaclass as well, based on jsbueno's answer. I cannot find formal Python documentation on this, but I've updated the output below.



            class M0(type):
            def __call__(mmcls, *args, **kwargs):
            print("M0 __call__: mmcls={!r}, "
            "args={!r}, kwargs={!r}".format(mmcls, args, kwargs))
            return super().__call__(*args, **kwargs)

            class Meta(type, metaclass=M0):
            def __call__(cls, *args, **kwargs):
            print("Meta __call__: cls={!r}, "
            "args={!r}, kwargs={!r}".format(cls, args, kwargs))
            return super().__call__(*args, **kwargs)

            def __new__(mcs, name, bases, attrs, **kwargs):
            print("Meta __new__: mcs={!r}, name={!r}, bases={!r}, "
            "attrs={!r}, kwargs={!r}".format(mcs, name, bases, attrs, kwargs))
            return super().__new__(mcs, name, bases, attrs)

            def __init__(mcs, name, bases, attrs, **kwargs):
            print("Meta __init__: mcs={!r}, name={!r}, bases={!r}, "
            "attrs={!r}, kwargs={!r}".format(mcs, name, bases, attrs, kwargs))
            super().__init__(name, bases, attrs, **kwargs)

            @classmethod
            def __prepare__(cls, name, bases, **kwargs):
            print("Meta __prepare__: name={!r}, "
            "bases={!r}, kwargs={!r}".format(name, bases, kwargs))
            return {}

            print("about to create class A")
            class A(metaclass=Meta): pass
            print("finished creating class A")

            print("about to create class B")

            class B(A, metaclass=Meta, foo=3):
            @staticmethod
            def __new__(cls, *args, **kwargs):
            print("B __new__: cls={!r}, "
            "args={!r}, kwargs={!r}".format(cls, args, kwargs))
            return super().__new__(cls)

            def __init__(self, *args, **kwargs):
            print("B __init__: args={!r}, kwargs={!r}, ".format(args, kwargs))

            print("finished creating class B")

            print("about to create instance b = B()")
            b = B('hello', bar=7)
            print("finished creating instance b")


            Now, let's observe what happens when I run this, and take each piece apart:



            $ python3.6 meta.py
            about to create class A
            Meta __prepare__: name='A', bases=(), kwargs={}
            M0 __call__: mmcls=<class '__main__.Meta'>, args=('A', (), {'__module__': '__main__', '__qualname__': 'A'}), kwargs={}
            Meta __new__: mcs=<class '__main__.Meta'>, name='A', bases=(), attrs={'__module__': '__main__', '__qualname__': 'A'}, kwargs={}
            Meta __init__: mcs=<class '__main__.A'>, name='A', bases=(), attrs={'__module__': '__main__', '__qualname__': 'A'}, kwargs={}
            finished creating class A


            To create class A itself, Python first calls the metaclass's __prepare__, delivering it the name of the class (A), the list of base classes (an empty tuple—it's called a list but is actually a tuple), and any keyword arguments (none). As PEP 3115 notes, the metaclass needs to return a dictionary or dict-like object; this one does by just returning an empty dictionary, so we are good here.



            (I don't print cls itself here, but if you do, you will see it is just <class '__main__.Meta'>.)



            Next, having gotten a dictionary from __prepare__, Python first calls the meta-meta __call__, i.e., M0.__call__, passing the entire set of arguments as the args tuple. It then populates the __prepare__-supplied dictionary with all the attributes for the class, passing this as the attrs to the metaclass __new__ and __init__. If you print the id of the dictionary returned from __prepare__ and passed to __new__ and __init__ you will see they all match.



            Since class A has no methods or data members, we see only the magic __module__ and __qualname__ attributes here. We also see no keyword arguments, so now let's move on to creating class B:



            about to create class B
            Meta __prepare__: name='B', bases=(<class '__main__.A'>,), kwargs={'foo': 3}
            M0 __call__: mmcls=<class '__main__.Meta'>, args=('B', (<class '__main__.A'>,), {'__module__': '__main__', '__qualname__': 'B', '__new__': <staticmethod object at 0x800ad0a58>, '__init__': <function B.__init__ at 0x800ad2840>, '__classcell__': <cell at 0x800a749d8: empty>}), kwargs={'foo': 3}
            Meta __new__: mcs=<class '__main__.Meta'>, name='B', bases=(<class '__main__.A'>,), attrs={'__module__': '__main__', '__qualname__': 'B', '__new__': <staticmethod object at 0x800ad0940>, '__init__': <function B.__init__ at 0x800ad27b8>, '__classcell__': <cell at 0x800a745b8: empty>}, kwargs={'foo': 3}
            Meta __init__: mcs=<class '__main__.B'>, name='B', bases=(<class '__main__.A'>,), attrs={'__module__': '__main__', '__qualname__': 'B', '__new__': <staticmethod object at 0x800ad0940>, '__init__': <function B.__init__ at 0x800ad27b8>, '__classcell__': <cell at 0x800a745b8: Meta object at 0x802047018>}, kwargs={'foo': 3}
            finished creating class B


            This one is rather more interesting. Now we have one base class, namely __main__.A. Class B also defines several methods (__new__ and __init__) and we see them in the attrs dictionaries passed to the metaclass __new__ and __init__ methods (which, remember, are just the now-populated dictionary returned by the metaclass's __prepare__). As before, the passing-on happens through the meta-meta-class M0.__call__. We also see one keyword argument throughout, {'foo': 3}. In the attribute dictionary, we can also observe the magic __classcell__ entry: see Provide __classcell__ example for Python 3.6 metaclass for a short description as to what this is about, but to be, er, super-short, it's for making super() work.



            The keyword argument is passed to all three metaclass methods, plus that of the meta-meta-class. (I'm not quite sure why. Note that modifying the dictionary in any metaclass method does not affect it in any other, as it's a copy each time of the original keyword arguments. However, we can modify it in the meta-meta-class: add kwargs.pop('foo', None) to M0.__call__ to observe this.)



            Now that we have our classes A and B, we can get on to the process of creating an actual instance of class B. Now we see the metaclass's __call__ invoked (not the meta-meta-class's):



            about to create instance b = B()
            Meta __call__: cls=<class '__main__.B'>, args=('hello',), kwargs={'bar': 7}


            It's possible to change the args or kwargs passed on, but I don't; the sample code above winds up calling type.__call__(cls, *args, **kwargs) (through the magic of super().__call__). This in turn calls B.__new__ and B.__init__:



            B __new__: cls=<class '__main__.B'>, args=('hello',), kwargs={'bar': 7}
            B __init__: args=('hello',), kwargs={'bar': 7},
            finished creating instance b


            which finishes the realization of the new instance of class B, which we then bind to the name b.



            Note that B.__new__ says:



            return super().__new__(cls)


            so we invoke object.__new__ to create the instance—this is more or less a requirement of all versions of Python; you can only "cheat" when you return a singleton instance (ideally, one that's non-modifiable). It's type.__call__ that calls B.__init__ on this object, passing the arguments and keyword-arguments we passed it. If we replace Meta's __call__ with:



                def __call__(cls, *args, **kwargs):
            print("Meta __call__: cls={!r}, "
            "args={!r}, kwargs={!r}".format(cls, args, kwargs))
            return object.__new__(cls)


            we will see that B.__new__ and B.__init__ are never called:



            about to create instance b = B()
            Meta __call__: cls=<class '__main__.B'>, args=('hello',), kwargs={'bar': 7}
            finished creating instance b


            This would, in effect, create a useless/uninitialized instance b. It's therefore critical that the metaclass __call__ method call the underlying class's __init__, usually by invoking type.__call__ via super().__call__. If the underlying class has a __new__, the metaclass should call that first, again usually by invoking type.__call__.



            Side note: what the documentation says



            To quote section 3.3.3.6:




            Once the class namespace has been populated by executing the class body, the class object is created by calling metaclass(name, bases, namespace, **kwds) (the additional keywords passed here are the same as those passed to __prepare__).




            This explains the call to Meta.__call__ when creating b as an instance of class B, but not the fact that Python first calls M0.__call__ before calling Meta.__new__ and Meta.__init__ when creating classes A and B themselves.



            The next paragraph mentions the __classcell__ entry; the one after that goes on to describe the use of __set_name__ and __init_subclass__ hooks. Nothing here tells us how or why Python calls M0.__call__ at this point.



            Earlier, in sections 3.3.3.3 through 3.3.3.5, the documentation describes the process of determining the metaclass, preparing the class namespace, and executing the class body. This is where the meta-metaclass action should be described, but isn't.



            Several additional sections describe a few additional constraints. One important one is 3.3.10, which talks about how special methods are found via the object type, bypassing both regular member attribute lookups and even (sometimes) a metaclass getattribute, saying:




            Bypassing the __getattribute__() machinery in this fashion provides significant scope for speed optimisations within the interpreter, at the cost of some flexibility in the handling of special methods (the special method must be set on the class object itself in order to be consistently invoked by the interpreter).




            Update 2: This is really the secret of the trick: the special __call__ method is found via the type's type. If the metaclass has a metaclass, the meta-meta-class provides the __call__ slot; otherwise the type of the metaclass is type, so that the __call__ slot is type.__call__.






            share|improve this answer























            • Despite your lenghty answer, and rather usefull information on the process of class creation, you did not touch the subject of the "meta-meta-class" call method, the OP is in doubt about. In fact, there is only a simple mistake in the OP's code - and the diagram posted in the question is actually correct.
              – jsbueno
              Nov 12 at 3:53










            • While I'm not sure whether I agree with jsbueno about what the mistake in the questioner's code is, I agree that you seem to have missed the meta-metaclass aspect, and that the diagram in the question is correct.
              – user2357112
              Nov 12 at 3:58










            • @jsbueno: interesting. I haven't seen this described elsewhere. Indeed, docs.python.org/3/reference/datamodel.html makes no mention of it while describing how regular metaclasses work.
              – torek
              Nov 12 at 7:06










            • @user2357112: indeed, testing shows it works—but there seems to be no hint of this in any documentation I can find.
              – torek
              Nov 12 at 8:01










            • Thanks a million for your detailed answer, I learned a lot. The diagram comes from the blog — understanding-python-metaclasses. Just like mentioned by jsbueno, Meta-metaclass should class SubMeta(type, metaclass=Meta): rather than class SubMeta(Meta):.
              – return long
              Nov 12 at 8:17













            up vote
            3
            down vote










            up vote
            3
            down vote









            The trick, identified



            Update 2: Based on behavior, the fact that M0.__call__ is called below must be a side effect of this line in builtin__build_class in the CPython source (Python/bltinmodule.c).



            In order to define a class that has a metaclass, we call the metaclass's __prepare__, __new__, and __init__ as usual. This creates a class—in the example below, Meta—that is callable, but its internal PyFunction_GET_CODE slot points not to its own __call__ but rather to its metaclass's __call__. Hence if we call Meta() (the metaclass object), we invoke M0.__call__:



            print("call Meta")
            print("Meta returns:", Meta('name', (), {}))
            print("finished calling Meta")


            produces:



            call Meta
            M0 __call__: mmcls=<class '__main__.Meta'>, args=('name', (), {}), kwargs={}
            Meta __new__: mcs=<class '__main__.Meta'>, name='name', bases=(), attrs={}, kwargs={}
            Meta __init__: mcs=<class '__main__.name'>, name='name', bases=(), attrs={}, kwargs={}
            Meta returns: <class '__main__.name'>
            finished calling Meta


            In other words, we see that Meta acts like type, but it (rather magically and not very well documented) invokes M0.__call__. This is no doubt due to looking up __call__ in the class's type, rather than in an instance of the class (and indeed there is no instance except for the one we're creating). This is in fact the general case: it falls out of the fact that we call __call__ on the type of Meta, and the type of Meta is M0:



            print("type(Meta) =", type(Meta))


            prints:



            type(Meta) = <class '__main__.M0'>


            which explains where this comes from. (I still think this should be emphasized in the documentation, which also should describe the constraints on metaclass typing—these are enforced in _calculate_winner in Lib/types.py and, as C code, in _PyType_CalculateMetaclass in Objects/typeobject.c.)



            Updated original answer



            I don't know where your diagram came from, but it's wrong. UPDATE: You can in fact have a metaclass for your metaclass; see jsbueno's answer, and I've updated the example below. New sentences / text are in bold, except for the final section describing my puzzlement at the apparent lack of documentation.



            Your existing metaclass code has at least one error. Most significantly, its __prepare__ needs to be a class-method. See also Using the __call__ method of a metaclass instead of __new__? and PEP 3115. And, to use a meta-meta-class, your metaclass needs to have a metaclass of its own, not a base class.



            Chris's answer contains correct definitions. But there are some unfortunate asymmetries between metaclass method arguments and class method arguments, which I'll illustrate below.



            One other thing that may help: note that the metaclass __prepare__ method is called before creating any instances of class B: it is called when class B itself is being defined. To show this, here is a corrected metaclass-and-class. I have also added a few more illustrators. I've added a meta-metaclass as well, based on jsbueno's answer. I cannot find formal Python documentation on this, but I've updated the output below.



            class M0(type):
            def __call__(mmcls, *args, **kwargs):
            print("M0 __call__: mmcls={!r}, "
            "args={!r}, kwargs={!r}".format(mmcls, args, kwargs))
            return super().__call__(*args, **kwargs)

            class Meta(type, metaclass=M0):
            def __call__(cls, *args, **kwargs):
            print("Meta __call__: cls={!r}, "
            "args={!r}, kwargs={!r}".format(cls, args, kwargs))
            return super().__call__(*args, **kwargs)

            def __new__(mcs, name, bases, attrs, **kwargs):
            print("Meta __new__: mcs={!r}, name={!r}, bases={!r}, "
            "attrs={!r}, kwargs={!r}".format(mcs, name, bases, attrs, kwargs))
            return super().__new__(mcs, name, bases, attrs)

            def __init__(mcs, name, bases, attrs, **kwargs):
            print("Meta __init__: mcs={!r}, name={!r}, bases={!r}, "
            "attrs={!r}, kwargs={!r}".format(mcs, name, bases, attrs, kwargs))
            super().__init__(name, bases, attrs, **kwargs)

            @classmethod
            def __prepare__(cls, name, bases, **kwargs):
            print("Meta __prepare__: name={!r}, "
            "bases={!r}, kwargs={!r}".format(name, bases, kwargs))
            return {}

            print("about to create class A")
            class A(metaclass=Meta): pass
            print("finished creating class A")

            print("about to create class B")

            class B(A, metaclass=Meta, foo=3):
            @staticmethod
            def __new__(cls, *args, **kwargs):
            print("B __new__: cls={!r}, "
            "args={!r}, kwargs={!r}".format(cls, args, kwargs))
            return super().__new__(cls)

            def __init__(self, *args, **kwargs):
            print("B __init__: args={!r}, kwargs={!r}, ".format(args, kwargs))

            print("finished creating class B")

            print("about to create instance b = B()")
            b = B('hello', bar=7)
            print("finished creating instance b")


            Now, let's observe what happens when I run this, and take each piece apart:



            $ python3.6 meta.py
            about to create class A
            Meta __prepare__: name='A', bases=(), kwargs={}
            M0 __call__: mmcls=<class '__main__.Meta'>, args=('A', (), {'__module__': '__main__', '__qualname__': 'A'}), kwargs={}
            Meta __new__: mcs=<class '__main__.Meta'>, name='A', bases=(), attrs={'__module__': '__main__', '__qualname__': 'A'}, kwargs={}
            Meta __init__: mcs=<class '__main__.A'>, name='A', bases=(), attrs={'__module__': '__main__', '__qualname__': 'A'}, kwargs={}
            finished creating class A


            To create class A itself, Python first calls the metaclass's __prepare__, delivering it the name of the class (A), the list of base classes (an empty tuple—it's called a list but is actually a tuple), and any keyword arguments (none). As PEP 3115 notes, the metaclass needs to return a dictionary or dict-like object; this one does by just returning an empty dictionary, so we are good here.



            (I don't print cls itself here, but if you do, you will see it is just <class '__main__.Meta'>.)



            Next, having gotten a dictionary from __prepare__, Python first calls the meta-meta __call__, i.e., M0.__call__, passing the entire set of arguments as the args tuple. It then populates the __prepare__-supplied dictionary with all the attributes for the class, passing this as the attrs to the metaclass __new__ and __init__. If you print the id of the dictionary returned from __prepare__ and passed to __new__ and __init__ you will see they all match.



            Since class A has no methods or data members, we see only the magic __module__ and __qualname__ attributes here. We also see no keyword arguments, so now let's move on to creating class B:



            about to create class B
            Meta __prepare__: name='B', bases=(<class '__main__.A'>,), kwargs={'foo': 3}
            M0 __call__: mmcls=<class '__main__.Meta'>, args=('B', (<class '__main__.A'>,), {'__module__': '__main__', '__qualname__': 'B', '__new__': <staticmethod object at 0x800ad0a58>, '__init__': <function B.__init__ at 0x800ad2840>, '__classcell__': <cell at 0x800a749d8: empty>}), kwargs={'foo': 3}
            Meta __new__: mcs=<class '__main__.Meta'>, name='B', bases=(<class '__main__.A'>,), attrs={'__module__': '__main__', '__qualname__': 'B', '__new__': <staticmethod object at 0x800ad0940>, '__init__': <function B.__init__ at 0x800ad27b8>, '__classcell__': <cell at 0x800a745b8: empty>}, kwargs={'foo': 3}
            Meta __init__: mcs=<class '__main__.B'>, name='B', bases=(<class '__main__.A'>,), attrs={'__module__': '__main__', '__qualname__': 'B', '__new__': <staticmethod object at 0x800ad0940>, '__init__': <function B.__init__ at 0x800ad27b8>, '__classcell__': <cell at 0x800a745b8: Meta object at 0x802047018>}, kwargs={'foo': 3}
            finished creating class B


            This one is rather more interesting. Now we have one base class, namely __main__.A. Class B also defines several methods (__new__ and __init__) and we see them in the attrs dictionaries passed to the metaclass __new__ and __init__ methods (which, remember, are just the now-populated dictionary returned by the metaclass's __prepare__). As before, the passing-on happens through the meta-meta-class M0.__call__. We also see one keyword argument throughout, {'foo': 3}. In the attribute dictionary, we can also observe the magic __classcell__ entry: see Provide __classcell__ example for Python 3.6 metaclass for a short description as to what this is about, but to be, er, super-short, it's for making super() work.



            The keyword argument is passed to all three metaclass methods, plus that of the meta-meta-class. (I'm not quite sure why. Note that modifying the dictionary in any metaclass method does not affect it in any other, as it's a copy each time of the original keyword arguments. However, we can modify it in the meta-meta-class: add kwargs.pop('foo', None) to M0.__call__ to observe this.)



            Now that we have our classes A and B, we can get on to the process of creating an actual instance of class B. Now we see the metaclass's __call__ invoked (not the meta-meta-class's):



            about to create instance b = B()
            Meta __call__: cls=<class '__main__.B'>, args=('hello',), kwargs={'bar': 7}


            It's possible to change the args or kwargs passed on, but I don't; the sample code above winds up calling type.__call__(cls, *args, **kwargs) (through the magic of super().__call__). This in turn calls B.__new__ and B.__init__:



            B __new__: cls=<class '__main__.B'>, args=('hello',), kwargs={'bar': 7}
            B __init__: args=('hello',), kwargs={'bar': 7},
            finished creating instance b


            which finishes the realization of the new instance of class B, which we then bind to the name b.



            Note that B.__new__ says:



            return super().__new__(cls)


            so we invoke object.__new__ to create the instance—this is more or less a requirement of all versions of Python; you can only "cheat" when you return a singleton instance (ideally, one that's non-modifiable). It's type.__call__ that calls B.__init__ on this object, passing the arguments and keyword-arguments we passed it. If we replace Meta's __call__ with:



                def __call__(cls, *args, **kwargs):
            print("Meta __call__: cls={!r}, "
            "args={!r}, kwargs={!r}".format(cls, args, kwargs))
            return object.__new__(cls)


            we will see that B.__new__ and B.__init__ are never called:



            about to create instance b = B()
            Meta __call__: cls=<class '__main__.B'>, args=('hello',), kwargs={'bar': 7}
            finished creating instance b


            This would, in effect, create a useless/uninitialized instance b. It's therefore critical that the metaclass __call__ method call the underlying class's __init__, usually by invoking type.__call__ via super().__call__. If the underlying class has a __new__, the metaclass should call that first, again usually by invoking type.__call__.



            Side note: what the documentation says



            To quote section 3.3.3.6:




            Once the class namespace has been populated by executing the class body, the class object is created by calling metaclass(name, bases, namespace, **kwds) (the additional keywords passed here are the same as those passed to __prepare__).




            This explains the call to Meta.__call__ when creating b as an instance of class B, but not the fact that Python first calls M0.__call__ before calling Meta.__new__ and Meta.__init__ when creating classes A and B themselves.



            The next paragraph mentions the __classcell__ entry; the one after that goes on to describe the use of __set_name__ and __init_subclass__ hooks. Nothing here tells us how or why Python calls M0.__call__ at this point.



            Earlier, in sections 3.3.3.3 through 3.3.3.5, the documentation describes the process of determining the metaclass, preparing the class namespace, and executing the class body. This is where the meta-metaclass action should be described, but isn't.



            Several additional sections describe a few additional constraints. One important one is 3.3.10, which talks about how special methods are found via the object type, bypassing both regular member attribute lookups and even (sometimes) a metaclass getattribute, saying:




            Bypassing the __getattribute__() machinery in this fashion provides significant scope for speed optimisations within the interpreter, at the cost of some flexibility in the handling of special methods (the special method must be set on the class object itself in order to be consistently invoked by the interpreter).




            Update 2: This is really the secret of the trick: the special __call__ method is found via the type's type. If the metaclass has a metaclass, the meta-meta-class provides the __call__ slot; otherwise the type of the metaclass is type, so that the __call__ slot is type.__call__.






            share|improve this answer














            The trick, identified



            Update 2: Based on behavior, the fact that M0.__call__ is called below must be a side effect of this line in builtin__build_class in the CPython source (Python/bltinmodule.c).



            In order to define a class that has a metaclass, we call the metaclass's __prepare__, __new__, and __init__ as usual. This creates a class—in the example below, Meta—that is callable, but its internal PyFunction_GET_CODE slot points not to its own __call__ but rather to its metaclass's __call__. Hence if we call Meta() (the metaclass object), we invoke M0.__call__:



            print("call Meta")
            print("Meta returns:", Meta('name', (), {}))
            print("finished calling Meta")


            produces:



            call Meta
            M0 __call__: mmcls=<class '__main__.Meta'>, args=('name', (), {}), kwargs={}
            Meta __new__: mcs=<class '__main__.Meta'>, name='name', bases=(), attrs={}, kwargs={}
            Meta __init__: mcs=<class '__main__.name'>, name='name', bases=(), attrs={}, kwargs={}
            Meta returns: <class '__main__.name'>
            finished calling Meta


            In other words, we see that Meta acts like type, but it (rather magically and not very well documented) invokes M0.__call__. This is no doubt due to looking up __call__ in the class's type, rather than in an instance of the class (and indeed there is no instance except for the one we're creating). This is in fact the general case: it falls out of the fact that we call __call__ on the type of Meta, and the type of Meta is M0:



            print("type(Meta) =", type(Meta))


            prints:



            type(Meta) = <class '__main__.M0'>


            which explains where this comes from. (I still think this should be emphasized in the documentation, which also should describe the constraints on metaclass typing—these are enforced in _calculate_winner in Lib/types.py and, as C code, in _PyType_CalculateMetaclass in Objects/typeobject.c.)



            Updated original answer



            I don't know where your diagram came from, but it's wrong. UPDATE: You can in fact have a metaclass for your metaclass; see jsbueno's answer, and I've updated the example below. New sentences / text are in bold, except for the final section describing my puzzlement at the apparent lack of documentation.



            Your existing metaclass code has at least one error. Most significantly, its __prepare__ needs to be a class-method. See also Using the __call__ method of a metaclass instead of __new__? and PEP 3115. And, to use a meta-meta-class, your metaclass needs to have a metaclass of its own, not a base class.



            Chris's answer contains correct definitions. But there are some unfortunate asymmetries between metaclass method arguments and class method arguments, which I'll illustrate below.



            One other thing that may help: note that the metaclass __prepare__ method is called before creating any instances of class B: it is called when class B itself is being defined. To show this, here is a corrected metaclass-and-class. I have also added a few more illustrators. I've added a meta-metaclass as well, based on jsbueno's answer. I cannot find formal Python documentation on this, but I've updated the output below.



            class M0(type):
            def __call__(mmcls, *args, **kwargs):
            print("M0 __call__: mmcls={!r}, "
            "args={!r}, kwargs={!r}".format(mmcls, args, kwargs))
            return super().__call__(*args, **kwargs)

            class Meta(type, metaclass=M0):
            def __call__(cls, *args, **kwargs):
            print("Meta __call__: cls={!r}, "
            "args={!r}, kwargs={!r}".format(cls, args, kwargs))
            return super().__call__(*args, **kwargs)

            def __new__(mcs, name, bases, attrs, **kwargs):
            print("Meta __new__: mcs={!r}, name={!r}, bases={!r}, "
            "attrs={!r}, kwargs={!r}".format(mcs, name, bases, attrs, kwargs))
            return super().__new__(mcs, name, bases, attrs)

            def __init__(mcs, name, bases, attrs, **kwargs):
            print("Meta __init__: mcs={!r}, name={!r}, bases={!r}, "
            "attrs={!r}, kwargs={!r}".format(mcs, name, bases, attrs, kwargs))
            super().__init__(name, bases, attrs, **kwargs)

            @classmethod
            def __prepare__(cls, name, bases, **kwargs):
            print("Meta __prepare__: name={!r}, "
            "bases={!r}, kwargs={!r}".format(name, bases, kwargs))
            return {}

            print("about to create class A")
            class A(metaclass=Meta): pass
            print("finished creating class A")

            print("about to create class B")

            class B(A, metaclass=Meta, foo=3):
            @staticmethod
            def __new__(cls, *args, **kwargs):
            print("B __new__: cls={!r}, "
            "args={!r}, kwargs={!r}".format(cls, args, kwargs))
            return super().__new__(cls)

            def __init__(self, *args, **kwargs):
            print("B __init__: args={!r}, kwargs={!r}, ".format(args, kwargs))

            print("finished creating class B")

            print("about to create instance b = B()")
            b = B('hello', bar=7)
            print("finished creating instance b")


            Now, let's observe what happens when I run this, and take each piece apart:



            $ python3.6 meta.py
            about to create class A
            Meta __prepare__: name='A', bases=(), kwargs={}
            M0 __call__: mmcls=<class '__main__.Meta'>, args=('A', (), {'__module__': '__main__', '__qualname__': 'A'}), kwargs={}
            Meta __new__: mcs=<class '__main__.Meta'>, name='A', bases=(), attrs={'__module__': '__main__', '__qualname__': 'A'}, kwargs={}
            Meta __init__: mcs=<class '__main__.A'>, name='A', bases=(), attrs={'__module__': '__main__', '__qualname__': 'A'}, kwargs={}
            finished creating class A


            To create class A itself, Python first calls the metaclass's __prepare__, delivering it the name of the class (A), the list of base classes (an empty tuple—it's called a list but is actually a tuple), and any keyword arguments (none). As PEP 3115 notes, the metaclass needs to return a dictionary or dict-like object; this one does by just returning an empty dictionary, so we are good here.



            (I don't print cls itself here, but if you do, you will see it is just <class '__main__.Meta'>.)



            Next, having gotten a dictionary from __prepare__, Python first calls the meta-meta __call__, i.e., M0.__call__, passing the entire set of arguments as the args tuple. It then populates the __prepare__-supplied dictionary with all the attributes for the class, passing this as the attrs to the metaclass __new__ and __init__. If you print the id of the dictionary returned from __prepare__ and passed to __new__ and __init__ you will see they all match.



            Since class A has no methods or data members, we see only the magic __module__ and __qualname__ attributes here. We also see no keyword arguments, so now let's move on to creating class B:



            about to create class B
            Meta __prepare__: name='B', bases=(<class '__main__.A'>,), kwargs={'foo': 3}
            M0 __call__: mmcls=<class '__main__.Meta'>, args=('B', (<class '__main__.A'>,), {'__module__': '__main__', '__qualname__': 'B', '__new__': <staticmethod object at 0x800ad0a58>, '__init__': <function B.__init__ at 0x800ad2840>, '__classcell__': <cell at 0x800a749d8: empty>}), kwargs={'foo': 3}
            Meta __new__: mcs=<class '__main__.Meta'>, name='B', bases=(<class '__main__.A'>,), attrs={'__module__': '__main__', '__qualname__': 'B', '__new__': <staticmethod object at 0x800ad0940>, '__init__': <function B.__init__ at 0x800ad27b8>, '__classcell__': <cell at 0x800a745b8: empty>}, kwargs={'foo': 3}
            Meta __init__: mcs=<class '__main__.B'>, name='B', bases=(<class '__main__.A'>,), attrs={'__module__': '__main__', '__qualname__': 'B', '__new__': <staticmethod object at 0x800ad0940>, '__init__': <function B.__init__ at 0x800ad27b8>, '__classcell__': <cell at 0x800a745b8: Meta object at 0x802047018>}, kwargs={'foo': 3}
            finished creating class B


            This one is rather more interesting. Now we have one base class, namely __main__.A. Class B also defines several methods (__new__ and __init__) and we see them in the attrs dictionaries passed to the metaclass __new__ and __init__ methods (which, remember, are just the now-populated dictionary returned by the metaclass's __prepare__). As before, the passing-on happens through the meta-meta-class M0.__call__. We also see one keyword argument throughout, {'foo': 3}. In the attribute dictionary, we can also observe the magic __classcell__ entry: see Provide __classcell__ example for Python 3.6 metaclass for a short description as to what this is about, but to be, er, super-short, it's for making super() work.



            The keyword argument is passed to all three metaclass methods, plus that of the meta-meta-class. (I'm not quite sure why. Note that modifying the dictionary in any metaclass method does not affect it in any other, as it's a copy each time of the original keyword arguments. However, we can modify it in the meta-meta-class: add kwargs.pop('foo', None) to M0.__call__ to observe this.)



            Now that we have our classes A and B, we can get on to the process of creating an actual instance of class B. Now we see the metaclass's __call__ invoked (not the meta-meta-class's):



            about to create instance b = B()
            Meta __call__: cls=<class '__main__.B'>, args=('hello',), kwargs={'bar': 7}


            It's possible to change the args or kwargs passed on, but I don't; the sample code above winds up calling type.__call__(cls, *args, **kwargs) (through the magic of super().__call__). This in turn calls B.__new__ and B.__init__:



            B __new__: cls=<class '__main__.B'>, args=('hello',), kwargs={'bar': 7}
            B __init__: args=('hello',), kwargs={'bar': 7},
            finished creating instance b


            which finishes the realization of the new instance of class B, which we then bind to the name b.



            Note that B.__new__ says:



            return super().__new__(cls)


            so we invoke object.__new__ to create the instance—this is more or less a requirement of all versions of Python; you can only "cheat" when you return a singleton instance (ideally, one that's non-modifiable). It's type.__call__ that calls B.__init__ on this object, passing the arguments and keyword-arguments we passed it. If we replace Meta's __call__ with:



                def __call__(cls, *args, **kwargs):
            print("Meta __call__: cls={!r}, "
            "args={!r}, kwargs={!r}".format(cls, args, kwargs))
            return object.__new__(cls)


            we will see that B.__new__ and B.__init__ are never called:



            about to create instance b = B()
            Meta __call__: cls=<class '__main__.B'>, args=('hello',), kwargs={'bar': 7}
            finished creating instance b


            This would, in effect, create a useless/uninitialized instance b. It's therefore critical that the metaclass __call__ method call the underlying class's __init__, usually by invoking type.__call__ via super().__call__. If the underlying class has a __new__, the metaclass should call that first, again usually by invoking type.__call__.



            Side note: what the documentation says



            To quote section 3.3.3.6:




            Once the class namespace has been populated by executing the class body, the class object is created by calling metaclass(name, bases, namespace, **kwds) (the additional keywords passed here are the same as those passed to __prepare__).




            This explains the call to Meta.__call__ when creating b as an instance of class B, but not the fact that Python first calls M0.__call__ before calling Meta.__new__ and Meta.__init__ when creating classes A and B themselves.



            The next paragraph mentions the __classcell__ entry; the one after that goes on to describe the use of __set_name__ and __init_subclass__ hooks. Nothing here tells us how or why Python calls M0.__call__ at this point.



            Earlier, in sections 3.3.3.3 through 3.3.3.5, the documentation describes the process of determining the metaclass, preparing the class namespace, and executing the class body. This is where the meta-metaclass action should be described, but isn't.



            Several additional sections describe a few additional constraints. One important one is 3.3.10, which talks about how special methods are found via the object type, bypassing both regular member attribute lookups and even (sometimes) a metaclass getattribute, saying:




            Bypassing the __getattribute__() machinery in this fashion provides significant scope for speed optimisations within the interpreter, at the cost of some flexibility in the handling of special methods (the special method must be set on the class object itself in order to be consistently invoked by the interpreter).




            Update 2: This is really the secret of the trick: the special __call__ method is found via the type's type. If the metaclass has a metaclass, the meta-meta-class provides the __call__ slot; otherwise the type of the metaclass is type, so that the __call__ slot is type.__call__.







            share|improve this answer














            share|improve this answer



            share|improve this answer








            edited Nov 12 at 9:50

























            answered Nov 11 at 8:33









            torek

            178k16227303




            178k16227303












            • Despite your lenghty answer, and rather usefull information on the process of class creation, you did not touch the subject of the "meta-meta-class" call method, the OP is in doubt about. In fact, there is only a simple mistake in the OP's code - and the diagram posted in the question is actually correct.
              – jsbueno
              Nov 12 at 3:53










            • While I'm not sure whether I agree with jsbueno about what the mistake in the questioner's code is, I agree that you seem to have missed the meta-metaclass aspect, and that the diagram in the question is correct.
              – user2357112
              Nov 12 at 3:58










            • @jsbueno: interesting. I haven't seen this described elsewhere. Indeed, docs.python.org/3/reference/datamodel.html makes no mention of it while describing how regular metaclasses work.
              – torek
              Nov 12 at 7:06










            • @user2357112: indeed, testing shows it works—but there seems to be no hint of this in any documentation I can find.
              – torek
              Nov 12 at 8:01










            • Thanks a million for your detailed answer, I learned a lot. The diagram comes from the blog — understanding-python-metaclasses. Just like mentioned by jsbueno, Meta-metaclass should class SubMeta(type, metaclass=Meta): rather than class SubMeta(Meta):.
              – return long
              Nov 12 at 8:17


















            • Despite your lenghty answer, and rather usefull information on the process of class creation, you did not touch the subject of the "meta-meta-class" call method, the OP is in doubt about. In fact, there is only a simple mistake in the OP's code - and the diagram posted in the question is actually correct.
              – jsbueno
              Nov 12 at 3:53










            • While I'm not sure whether I agree with jsbueno about what the mistake in the questioner's code is, I agree that you seem to have missed the meta-metaclass aspect, and that the diagram in the question is correct.
              – user2357112
              Nov 12 at 3:58










            • @jsbueno: interesting. I haven't seen this described elsewhere. Indeed, docs.python.org/3/reference/datamodel.html makes no mention of it while describing how regular metaclasses work.
              – torek
              Nov 12 at 7:06










            • @user2357112: indeed, testing shows it works—but there seems to be no hint of this in any documentation I can find.
              – torek
              Nov 12 at 8:01










            • Thanks a million for your detailed answer, I learned a lot. The diagram comes from the blog — understanding-python-metaclasses. Just like mentioned by jsbueno, Meta-metaclass should class SubMeta(type, metaclass=Meta): rather than class SubMeta(Meta):.
              – return long
              Nov 12 at 8:17
















            Despite your lenghty answer, and rather usefull information on the process of class creation, you did not touch the subject of the "meta-meta-class" call method, the OP is in doubt about. In fact, there is only a simple mistake in the OP's code - and the diagram posted in the question is actually correct.
            – jsbueno
            Nov 12 at 3:53




            Despite your lenghty answer, and rather usefull information on the process of class creation, you did not touch the subject of the "meta-meta-class" call method, the OP is in doubt about. In fact, there is only a simple mistake in the OP's code - and the diagram posted in the question is actually correct.
            – jsbueno
            Nov 12 at 3:53












            While I'm not sure whether I agree with jsbueno about what the mistake in the questioner's code is, I agree that you seem to have missed the meta-metaclass aspect, and that the diagram in the question is correct.
            – user2357112
            Nov 12 at 3:58




            While I'm not sure whether I agree with jsbueno about what the mistake in the questioner's code is, I agree that you seem to have missed the meta-metaclass aspect, and that the diagram in the question is correct.
            – user2357112
            Nov 12 at 3:58












            @jsbueno: interesting. I haven't seen this described elsewhere. Indeed, docs.python.org/3/reference/datamodel.html makes no mention of it while describing how regular metaclasses work.
            – torek
            Nov 12 at 7:06




            @jsbueno: interesting. I haven't seen this described elsewhere. Indeed, docs.python.org/3/reference/datamodel.html makes no mention of it while describing how regular metaclasses work.
            – torek
            Nov 12 at 7:06












            @user2357112: indeed, testing shows it works—but there seems to be no hint of this in any documentation I can find.
            – torek
            Nov 12 at 8:01




            @user2357112: indeed, testing shows it works—but there seems to be no hint of this in any documentation I can find.
            – torek
            Nov 12 at 8:01












            Thanks a million for your detailed answer, I learned a lot. The diagram comes from the blog — understanding-python-metaclasses. Just like mentioned by jsbueno, Meta-metaclass should class SubMeta(type, metaclass=Meta): rather than class SubMeta(Meta):.
            – return long
            Nov 12 at 8:17




            Thanks a million for your detailed answer, I learned a lot. The diagram comes from the blog — understanding-python-metaclasses. Just like mentioned by jsbueno, Meta-metaclass should class SubMeta(type, metaclass=Meta): rather than class SubMeta(Meta):.
            – return long
            Nov 12 at 8:17


















             

            draft saved


            draft discarded



















































             


            draft saved


            draft discarded














            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53246017%2fthe-call-order-of-python3-metaclass%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            Popular posts from this blog

            Bressuire

            Vorschmack

            Quarantine