Repository navigation
Difference in behavior between pure Python pickle and C extension _pickle when unpickling instance of instance of metaclass #105250
Description
Activity
- changed the title
[-]Difference in behavior between pure Python pickle and C extension _pickle when unpickling class instances.[/-][+]Difference in behavior between pure Python pickle and C extension _pickle when unpickling instance of instance of metaclass[/+]on Jun 2, 2023 Hi @corona10 @hugovk, I'd like to work on this issue as part of the PyCon KR 2026 sprint.
I confirmed it still reproduces on current main (70fdc96).
One subtlety I noticed while investigating: the C unpickler is not always "metaclass-free".
load_newobjinModules/_pickle.ccallstp_newdirectly (even though the comment above the call says the intent is to callcls.__new__(cls, *args, **kwargs)), but when the class defines__new__in Python,slot_tp_newitself looks__new__up on the type, so the metaclass hook does fire even under_pickle. So whether the hook runs currently depends on whether the class being unpickled happens to define__new__in Python.Reproducer and output on main
from test.support.import_helper import import_fresh_module c_pickle = import_fresh_module("pickle", fresh=["_pickle"]) py_pickle = import_fresh_module("pickle", blocked=["_pickle"]) LOG = [] class Meta(type): def __getattribute__(cls, name): LOG.append(name) return super().__getattribute__(name) class InheritsNew(metaclass=Meta): # tp_new is object.__new__ (a real C slot) def __init__(self, value): self.value = value class DefinesNew(metaclass=Meta): # tp_new is slot_tp_new def __new__(cls, *args): return super().__new__(cls) def __init__(self, value): self.value = value def lookups(mod, obj, protocol): """Attribute lookups the metaclass observes while unpickling.""" data = mod.dumps(obj, protocol=protocol) LOG.clear() mod.loads(data) return sorted(set(LOG)) for cls in (InheritsNew, DefinesNew): for protocol in (2, 5): c = lookups(c_pickle, cls(42), protocol) py = lookups(py_pickle, cls(42), protocol) print(f"{cls.__name__} / protocol {protocol}: C={c!s:<13} " f"py={py!s:<13} {'DIVERGES' if c != py else 'same'}")
InheritsNew / protocol 2: C=[] py=['__new__'] DIVERGES InheritsNew / protocol 5: C=[] py=['__new__'] DIVERGES DefinesNew / protocol 2: C=['__new__'] py=['__new__'] same DefinesNew / protocol 5: C=['__new__'] py=['__new__'] sameI'm planning to keep the fix as minimal as possible (mirroring the pure-Python
cls.__new__lookup on the_pickle.cside). If you think this issue isn't a good fit for the sprint, please let me know and I'll pick another one.Reacted by Donghee Na, Hugo van Kemenade and James Murphy@chaerrypick01 I think that it's good to go
Reacted by Elly Chaewon KimReacted by Elly Chaewon Kim, Hugo van Kemenade and James Murphy
Metadata
Metadata
Assignees
Projects
- Status
Bug report
When pickling and unpickling an instance of an instance of a metaclass,
_picklereaches into the class object directly and callstp_newfrom the C side, whereaspickleusescls.__new__, which triggers a custom__getattribute__if defined. Whether this is a bug or just an intentional optimization I'm not sure but at the very least one there is an observable difference in behavior in that unpickling from a distribution where_pickleis available does not call__getattribute__but unpickling from a distribution where_pickleis not available does call__getattribute__.MWE
(Works on all available CPythons on Compiler Explorer, 3.5 - 3.11)
https://godbolt.org/z/x76bs8dzv
To simulate different distributions having
_pickleavailable or not, we can use a meta path entry to cause it to fail to import, causingpickleto fall back to the pure Python version.Output with
hide_pickle = TrueOutput with
hide_pickle = FalseLinked PRs