Deprecate pickle support for itertools

Pickle support was long ago added to some itertools. It was done mostly to support an atypical use case for a single company. It was implemented in a very inefficient manner, essentially replaying iteration from the beginning to the mid-stream state where it was frozen. The implementation was of low quality and had many bugs. Also, it was not a documented or advertised feature. Newer itertools don't support pickling and no one has noticed or cared. The popular third-party package more-itertools is implemented with generators which do not have pickle support — again, none of their users seems to have noticed or cared.

IMO, this is just cruft that has made maintenance more difficult and we should get rid of it. As an undocumented feature, we could just remove it directly. But to be on the safe side, we can go through a deprecation cycle.

Linked PRs

@rhettinger

Hi @rhettinger, would you like a hand on this?

@JosephSBoyle

@JosephSBoyle I am already working on a patch for this issue. Please find a different issue to work on.

Thanks for working on this. When the PR is ready for review, please assign to me :-)

@JosephSBoyle

@JosephSBoyle Do you pick up this task?

I still plan to create a pull request for this issue (within the next two weeks, hopefully).

@rhettinger

Hi @rhettinger, I'd be happy to work on this.

@ZackerySpytz, perhaps we could work together somehow? Up to you - I appreciate that you were working on this before I volunteered to help.

I was about to use the pickling feature on my end, and I wouldn't have noticed this had I not checked in the doc, noticed it wasn't there and propose it to be in the issue above.
So, I wouldn't be surprised if others than me use the feature as well, and while I fully understand and support the rationale for deprecating, I think making it follow a deprecation cycle wouldn't hurt.
In my experience and opinion, iterators in python generally fail spectacularly when you try to pickle them, so having the itertools one not fail may have felt for many to be a low-key guarantee on that being a supported feature, even if it was not explicitly documented (and even though it's not a very good practice).

@JosephSBoyle

@JosephSBoyle You no longer need to wait. There is no particular reason for @ZackerySpytz to reserve this one. There are plenty of other open bugs.

@rhettinger

Sure, @rhettinger. Where would you recommend starting on this?

Thomas, can this deprecation still go into 3.12? It deprecates an undocumented feature.

It may be undocumented but pickle use is so viral, it's very likely depended on without people even knowing :) Let's get the deprecation into beta 2 if we can (it's scheduled for next week), but given that it's not actually problematic and just something we don't want to support, I would suggest holding back on removing it until we have a clearer picture of what it would break.

GH-101588: Deprecate pickle/copy/deepcopy support in itertools (GH-10…

pythonGH-101588: Deprecate pickle/copy/deepcopy support in itertools (p…

[3.12] GH-101588: Deprecate pickle/copy/deepcopy support in itertools (…

pythonGH-101588: Deprecate pickle/copy/deepcopy support in itertools

GH-101588: Remove deprecated pickle/copy/deepcopy from itertools (gh-…

I missed when this was deprecated, but I support this. It was inefficient, it leaked internal state, and it often not worked as expected for complex iterators like chain or tee.

Should not this also be deprecated for builtins zip, map and filter, and for iterators of builtin collections?

A little late to the party, but exposing this surface makes for a relatively easy crash in Py3.13.1:

Python 3.13.1 (main, Dec 28 2024, 13:25:56) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import itertools
>>> tdo = itertools._tee_dataobject(object(), [1], None)
>>> tob = itertools._tee(())
>>> tob.__setstate__((tdo, 0))
<python-input-3>:1: DeprecationWarning: Pickle, copy, and deepcopy support will be removed from itertools in Python 3.14.
  tob.__setstate__((tdo, 0))
>>> next(tob)
1
>>> next(tob)
Segmentation fault

This seems to work on any non-iterable object in the first argument to the TDO constructor. I suppose it's because the TDO calls PyIter_Next on this object without ever having done a PyIter_Check.

Arguably that's a separate bug, but removing this support should probably see the interface to _tee_dataobject from user code go away. TDOs seem to be a fragile internal detail in any case.

Navigation Menu

Repository navigation

Deprecate pickle support for itertools #101588

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Deprecate pickle support for itertools #101588

Description

Linked PRs

Activity

JosephSBoyle commented on Feb 21, 2023

ZackerySpytz commented on Feb 21, 2023

rhettinger commented on Feb 21, 2023

rhettinger commented on Apr 4, 2023

ZackerySpytz commented on Apr 5, 2023

JosephSBoyle commented on Apr 5, 2023

Gouvernathor commented on Apr 18, 2023

rhettinger commented on Apr 24, 2023

JosephSBoyle commented on Apr 24, 2023

rhettinger commented on May 24, 2023

Yhg1s commented on May 24, 2023

serhiy-storchaka commented on May 9, 2024

Grissess commented on Apr 11, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions