Repository navigation
inspect.getsourcelines() is buggy on 3.12 #104866
Description
% cat gsl.py
import inspect
def test_list_add(self):
def capybara() -> None:
assert_is_value(
[x] + [y],
z,
)
assert_is_value(
# in the binop implementation
a,
)
print(inspect.getsourcelines(test_list_add))
% python3.11 gsl.py
(['def test_list_add(self):\n', ' def capybara() -> None:\n', ' assert_is_value(\n', ' [x] + [y],\n', ' z,\n', ' )\n', ' assert_is_value(\n', ' # in the binop implementation\n', ' a,\n', ' )\n'], 3)
% ~/py/cpython/python.exe gsl.py
(['def test_list_add(self):\n', ' def capybara() -> None:\n', ' assert_is_value(\n', ' [x] + [y],\n', ' z,\n', ' )\n', ' assert_is_value(\n', ' # in the binop implementation\n'], 3)
% ~/py/cpython/python.exe -V
Python 3.12.0b1+
On 3.12, the last two lines (a, )) are not returned as part of the source lines. When I tried to minify by removing the first assert_is_value() call, it instead returned the print() line as part of the function.
This looks related to the tokenizer, cc @pablogsal @lysnikolaou.
Activity
lysnikolaou commentedon May 24, 2023
This is a bug in the implementation of the new tokenize module. The tokenizer checks whether something is in a parenthesized block in order to emit a
NLtoken instead of aNEWLINEtoken. It also sets and unsets a flag about a comment-only line, in order to do the same. When both are true (a comment inside a parenthesized block), it nevers resets the comment flag. Here's a minimal example:I'm opening a PR shortly.
pythongh-104866: Tokenize should emit NEWLINE after exiting block wit…
gh-104866: Tokenize should emit NEWLINE after exiting block with comm…
pythongh-104866: Tokenize should emit NEWLINE after exiting block wit…
[3.12] gh-104866: Tokenize should emit NEWLINE after exiting block wi…