Skip to content

Difference in behavior between pure Python pickle and C extension _pickle when unpickling instance of instance of metaclass #105250

@jamesmurphy-mc

Description

@jamesmurphy-mc

Bug report

When pickling and unpickling an instance of an instance of a metaclass, _pickle reaches into the class object directly and calls tp_new from the C side, whereas pickle uses cls.__new__, which triggers a custom __getattribute__ if defined. Whether this is a bug or just an intentional optimization I'm not sure but at the very least one there is an observable difference in behavior in that unpickling from a distribution where _pickle is available does not call __getattribute__ but unpickling from a distribution where _pickle is not available does call __getattribute__.

MWE

(Works on all available CPythons on Compiler Explorer, 3.5 - 3.11)
https://godbolt.org/z/x76bs8dzv

To simulate different distributions having _pickle available or not, we can use a meta path entry to cause it to fail to import, causing pickle to fall back to the pure Python version.

import importlib.abc import sys class HideModuleFinder(importlib.abc.MetaPathFinder): def __init__(self, hidden): self.hidden = set(hidden) def find_spec(self, fullname, path, target=None): if fullname in self.hidden: raise ImportError("Module is hidden") return None # let next finder try def install(hidden): sys.meta_path.insert(0, HideModuleFinder(hidden)) hide_pickle = False # change me for testing different behavior if hide_pickle: install({"_pickle"}) import pickle # must be done after meta path hook class Meta(type): def __getattribute__(self, item): print("__getattribute__ called with", item) return type.__getattribute__(self, item) MyClass = Meta("MyClass", (), {}) obj = MyClass() print("PICKLING") obj_str = pickle.dumps(obj) print("UNPICKLING") new_obj = pickle.loads(obj_str)

Output with hide_pickle = True

PICKLING __getattribute__ called with __reduce__ __getattribute__ called with __dict__ __getattribute__ called with __slots__ __getattribute__ called with __new__ __getattribute__ called with __qualname__ __getattribute__ called with __module__ UNPICKLING __getattribute__ called with __new__ 

Output with hide_pickle = False

PICKLING __getattribute__ called with __reduce__ __getattribute__ called with __dict__ __getattribute__ called with __slots__ __getattribute__ called with __qualname__ __getattribute__ called with __module__ UNPICKLING 

Metadata

Metadata

Assignees

No one assigned

    Labels

    stdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

    Projects

    Status

    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions