Defining iterator in a separate class no longer works in 3.13

Bug report

Bug description:

Defining an interator for a class in a separate class no longer works properly in 3.13. With the following test_iter.py:

class list2(list):
    def __iter__(self):
        return list2iterator(self)

class list2iterator:
    def __init__(self, X):
        self._X = X
        self._pointer = -1

    def __next__(self):
        self._pointer += 1
        if self._pointer == len(self._X):
            self._pointer = -1
            raise StopIteration
        return self._X[self._pointer]

With Python 3.13.1 one gets:

>>> from test_iter import list2
>>> X=list2([1,2,3])
>>> [x for x in X]
Traceback (most recent call last):
  File "<python-input-2>", line 1, in <module>
    [x for x in X]
                ^
TypeError: 'list2iterator' object is not iterable

With Python 3.12.7 it works:

>>> from test_iter import list2
>>> X=list2([1,2,3])
>>> [x for x in X]
[1, 2, 3]

Bisected to bcc7227e

CPython versions tested on:

3.12, 3.13

Operating systems tested on:

Linux

Linked PRs

@efimov-mikhail

cc @efimov-mikhail @markshannon @JelleZijlstra

https://docs.python.org/3/library/stdtypes.html#iterator-types says that iterators must also implement __iter__ (returning self). That's a long standing requirement on iterators, and not one introduced in 3.13.

Alright, thanks

@ronaldoussoren

@ronaldoussoren You're right, and this code is definitely doing something weird, but it works in 3.12:
example.py:

class list2(list):
    def __iter__(self):
        return list2iterator(self)

class list2iterator:
    def __init__(self, X):
        self._X = X
        self._pointer = -1

    def __next__(self):
        self._pointer += 1
        if self._pointer == len(self._X):
            self._pointer = -1
            raise StopIteration
        return self._X[self._pointer]


target = list2([1, 2, 3])
for i in target:
    print(i)


[print(i) for i in target]

3.12.8:

eclips4@suffering ~/tmp> python3.12 example.py
1
2
3
1
2
3

3.13.1:

1                                                                                  
2
3
Traceback (most recent call last):
  File "/Users/eclips4/tmp/example.py", line 23, in <module>
    [print(i) for i in target]
                       ^^^^^^
TypeError: 'list2iterator' object is not iterable

Though it probably shouldn't work, it is definitely a regression.

Bytecode for [x for x in y] 3.12.8:

  0           0 RESUME                   0

  1           2 LOAD_NAME                0 (y)
              4 GET_ITER
              6 LOAD_FAST_AND_CLEAR      0 (x)
              8 SWAP                     2
             10 BUILD_LIST               0
             12 SWAP                     2
        >>   14 FOR_ITER                 4 (to 26)
             18 STORE_FAST               0 (x)
             20 LOAD_FAST                0 (x)
             22 LIST_APPEND              2
             24 JUMP_BACKWARD            6 (to 14)
        >>   26 END_FOR
             28 SWAP                     2
             30 STORE_FAST               0 (x)
             32 RETURN_VALUE
        >>   34 SWAP                     2
             36 POP_TOP
             38 SWAP                     2
             40 STORE_FAST               0 (x)
             42 RERAISE                  0
ExceptionTable:
  10 to 26 -> 34 [2]

3.13.1:

   0           RESUME                   0

   1           LOAD_NAME                0 (y)
               GET_ITER
               LOAD_FAST_AND_CLEAR      0 (x)
               SWAP                     2
       L1:     BUILD_LIST               0
               SWAP                     2
               GET_ITER  - problematic instruction
       L2:     FOR_ITER                 4 (to L3)
               STORE_FAST_LOAD_FAST     0 (x, x)
               LIST_APPEND              2
               JUMP_BACKWARD            6 (to L2)
       L3:     END_FOR
               POP_TOP
       L4:     SWAP                     2
               STORE_FAST               0 (x)
               RETURN_VALUE

  --   L5:     SWAP                     2
               POP_TOP

   1           SWAP                     2
               STORE_FAST               0 (x)
               RERAISE                  0
ExceptionTable:
  L1 to L4 -> L5 [2]

@ronaldoussoren

@ronaldoussoren You're right, and this code is definitely doing something weird, but it works in 3.12: example.py:

Right, the code from the OP is broken according to the data model, but we still did a change that's not backward compatible. I'm not deep enough into this to fully understand the ramifications.

This behavior change is a direct consequence of change in bytecode generation in my PR. I don't think this is a bug.

The exact example worked in the CPython implementation of 3.12, but that may be an accident, as it should not have by the language definition. It might have already failed in 3.12 in other implementations. In any case, adding the following exposes the defect in list2iterable even in 3.12.

target2 = iter(target)  # A list2iterator object
target3 = iter(target2)  # Fails in 3.12.8 
# TypeError: 'list2iterator' object is not iterable

If there is still a question about changing anything, Guido might remember the original design discussions.

Relevant bit from glossary: iterator:

CPython implementation detail: CPython does not consistently apply the requirement that an iterator define __iter__().

Adding a __iter__ method to skia.Iter

@picnixz

I think we should decide what to do with this issue.
FYI, on current CPython 3.12 branch this code snippet works, but on 3.13, 3.14 and main branches it doesn't.

@picnixz @Eclips4 @markshannon

Basically, there are two ways which I can see as reliable:

Just close the issue w/o any changes in codebase.
Provide changes to 3.13 branch only just to fix some "very big change for minor release".

And some links on related issues and PRs:
#125038
#125178
#132351

@hugovk

And maybe it's reasonable to do some changes in docs?
Some clear mention about generator expressions like (x for x in None), for example.
They aren't correct in Python 3.13.3 and before, an exception will be raised on this expression in code.
But on current 3.13 branch this code is correct.

Maybe it's better to remove all related changes in 3.13 branch?
And have exactly the same behavior as on 3.12?

@hugovk @terryjreedy @JelleZijlstra

The issue title could perhaps be improved.

Current title: "Defining iterator in a separate class no longer works in 3.13"
Suggested new title: "Iterator without __iter__ no longer works in 3.13.1"

Details:

If I understand correctly, the iterator being defined in a separate class isn't really relevant; what's very relevant is that it doesn't have an __iter__ member function
it's significant that the version in which this broke is minor release 3.13.1, not 3.13 (I observe the example code succeeds in 3.13.0)

The title and text of committed #132351 seems to imply that maybe it fixes this, i.e. reverts the behavior to pre-3.13.1 in which the example code works (because the generated bytcode has no redundant GET_ITER on something already known to be an iterator).

Is that correct?

That would make me very happy, since I would very much like to go back to happily ignoring the IMO ill-advised "requirement" that iterators be iterable (in fact it is my hope that some day that requirement might be officially removed, for a brighter less-confusing future in which Iterator and Iterable are two clearly distinct non-overlapping concepts).

@efimov-mikhail

@efimov-mikhail wrote:

And maybe it's reasonable to do some changes in docs?
Some clear mention about generator expressions like (x for x in None), for example.
They aren't correct in Python 3.13.3 and before, an exception will be raised on this expression in code.
But on current 3.13 branch this code is correct.

(Unfortunately I don't currently have anything higher than 3.13.0 to try this out on...)

In 3.13.0 I see that (x for x in None) throws TypeError: 'NoneType' object is not iterable which seems reasonable to me.
Are you saying that, in later versions, it returns something instead of throwing, and that it's correct? Why?

@markshannon

The title and text of committed #132351 seems to imply that maybe it fixes this, i.e. reverts the behavior to pre-3.13.1 in which the example code works (because the generated bytcode has no redundant GET_ITER on something already known to be an iterator).

Is that correct?

No, this is not correct.
On current main branch

-> % cat test_list2.py
class list2(list):
    def __iter__(self):
        return list2iterator(self)

class list2iterator:
    def __init__(self, X):
        self._X = X
        self._pointer = -1

    def __next__(self):
        self._pointer += 1
        if self._pointer == len(self._X):
            self._pointer = -1
            raise StopIteration
        return self._X[self._pointer]

[print(i) for i in target]

-> % ./python test_list2.py
Traceback (most recent call last):
  File "/home/sikko/projects/cpython/test_iter.py", line 21, in <module>
    [print(i) for i in target]
                       ^^^^^^
TypeError: 'list2iterator' object is not iterable

>>> dis.dis('[print(i) for i in target]')
   0           RESUME                   0

   1           LOAD_NAME                0 (target)
               GET_ITER
               LOAD_FAST_AND_CLEAR      0 (i)
               SWAP                     2
       L1:     BUILD_LIST               0
               SWAP                     2
               GET_ITER
       L2:     FOR_ITER                11 (to L3)
               STORE_FAST               0 (i)
               LOAD_NAME                1 (print)
               PUSH_NULL
               LOAD_FAST_BORROW         0 (i)
               CALL                     1
               LIST_APPEND              2
               JUMP_BACKWARD           13 (to L2)
       L3:     END_FOR
               POP_ITER
       L4:     SWAP                     2
               STORE_FAST               0 (i)
               RETURN_VALUE

  --   L5:     SWAP                     2
               POP_TOP

   1           SWAP                     2
               STORE_FAST               0 (i)
               RERAISE                  0
ExceptionTable:
  L1 to L4 -> L5 [2]

This is because of changes in bytecode.
You can see two GET_ITER bytecodes for list comprehension here.

Suggested new title: "Iterator without __iter__ no longer works in 3.13.1"

It sounds reasonable for me. Although, this is not the only change.

(Unfortunately I don't currently have anything higher than 3.13.0 to try this out on...)

In 3.13.0 I see that (x for x in None) throws TypeError: 'NoneType' object is not iterable which seems reasonable to me. Are you saying that, in later versions, it returns something instead of throwing, and that it's correct? Why?

Yes, on current main, 3.14 and 3.13 branches we can see the following result:

>>> g = (x for x in None)
>>> g
<generator object <genexpr> at 0x7f923bb48130>
>>> list(g)
Traceback (most recent call last):
  File "<python-input-4>", line 1, in <module>
    list(g)
    ~~~~^^^
  File "<python-input-2>", line 1, in <genexpr>
    g = (x for x in None)
                    ^^^^
TypeError: 'NoneType' object is not iterable

You can see some motivation here:
#125178 (comment)
#132351 (comment)

cc @markshannon @sobolevn

[3.14] GH-128161: Remove redundant GET_ITER from list comprehension c…

[3.14] pythonGH-128161: Remove redundant GET_ITER from list comprehen…

[3.13] gh-128161: Remove redundant GET_ITER from list comprehension c…

[3.13] GH-128161: Fix refleak introduced in GH-134788 (GH-134799)

@picnixz

This is fixed in 3.13.4:

Python 3.13.4 (main, Jun  4 2025, 17:14:13) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from test_iter import list2
>>> X=list2([1,2,3])
>>> [x for x in X]
[1, 2, 3]

But in 3.14.0b2 it sill does not work.

However it could be reintroduced again, related issue #135171 and fix #135225

If there's a decision how this should work, do we need some additional tests to make the behavior stable?
cc: @picnixz @markshannon

Whatever we decide on the main branch, you're better off adding an __iter__ method returning self to your iterator. There's no guarantee that this will always work in CPython or other implementations.

I've started related d.p.o. thread here: https://discuss.python.org/t/generator-expressions-behavior-with-non-iterable-objects/96329.

Navigation Menu

Defining iterator in a separate class no longer works in 3.13 #128161

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

7 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Defining iterator in a separate class no longer works in 3.13 #128161

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

Eclips4 commented on Dec 21, 2024

ronaldoussoren commented on Dec 21, 2024

antonio-rojas commented on Dec 21, 2024

Eclips4 commented on Dec 22, 2024

ronaldoussoren commented on Dec 22, 2024

efimov-mikhail commented on Dec 22, 2024

terryjreedy commented on Dec 22, 2024

pochmann3 commented on Dec 23, 2024

7 remaining items

efimov-mikhail commented on May 26, 2025

efimov-mikhail commented on May 26, 2025

donhatch commented on May 27, 2025

donhatch commented on May 27, 2025

donhatch commented on May 27, 2025

efimov-mikhail commented on May 27, 2025

Roman513 commented on Jun 8, 2025

serhiy-storchaka commented on Jun 10, 2025

efimov-mikhail commented on Jun 22, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions