Repository navigation
RuntimeError after ctrl-C interrupt when asyncio is closing the threadpool #96827
Description
Activity
What does it do in Python 3.11 (e.g. 3.11rc2, now available from python.org)?
I tried, and it seems to give a similar traceback.
Details
~/3.11$ ./python.exe intr.py coro start thread start coro stop ^Cinterrupted! thread stop exception calling callback for <Future at 0x10d283450 state=finished returned NoneType> Traceback (most recent call last): File "/Users/guido/3.11/Lib/concurrent/futures/_base.py", line 340, in _invoke_callbacks callback(self) File "/Users/guido/3.11/Lib/asyncio/futures.py", line 407, in _call_set_state dest_loop.call_soon_threadsafe(_set_state, destination, source) File "/Users/guido/3.11/Lib/asyncio/base_events.py", line 803, in call_soon_threadsafe self._check_closed() File "/Users/guido/3.11/Lib/asyncio/base_events.py", line 519, in _check_closed raise RuntimeError('Event loop is closed') RuntimeError: Event loop is closed Exception in thread Thread-1 (_do_shutdown): Traceback (most recent call last): File "/Users/guido/3.11/Lib/asyncio/base_events.py", line 580, in _do_shutdown self.call_soon_threadsafe(future.set_result, None) File "/Users/guido/3.11/Lib/asyncio/base_events.py", line 803, in call_soon_threadsafe self._check_closed() File "/Users/guido/3.11/Lib/asyncio/base_events.py", line 519, in _check_closed raise RuntimeError('Event loop is closed') RuntimeError: Event loop is closed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Users/guido/3.11/Lib/threading.py", line 1038, in _bootstrap_inner self.run() File "/Users/guido/3.11/Lib/threading.py", line 975, in run self._target(*self._args, **self._kwargs) File "/Users/guido/3.11/Lib/asyncio/base_events.py", line 582, in _do_shutdown self.call_soon_threadsafe(future.set_exception, ex) File "/Users/guido/3.11/Lib/asyncio/base_events.py", line 803, in call_soon_threadsafe self._check_closed() File "/Users/guido/3.11/Lib/asyncio/base_events.py", line 519, in _check_closed raise RuntimeError('Event loop is closed') RuntimeError: Event loop is closed ~/3.11$Ouch, this is tricky.
I think I understand the last part of the traceback. When a loop is closed, it tries to shut down its default executor. This is done by creating a helper thread (in
shutdown_default_executor())) that runs_do_shutdown(). The helper thread blocks until the executor is actually shut down, and then usescall_soon_threadsafe()to report back to the event loop that it's done. Because the thread sleeps for 3 seconds, the by the time this happens the loop's_closedflag has already been set, andcall_soon_threadsafe()raises an exception because of this.Interestingly, the traceback has the form
<traceback 1> <traceback 2> During handling of the above exception, another exception occurred: <traceback 3>Now both traceback 2 and traceback 3 are reporting an error in
_do_shutdown(). The line numbers suggest that the firstcall_soon_tnreadsafe()call fails, its exception is caught, and then the second call fails in a similar manner.So what about traceback 1? This is happening in
_call_set_state(), an inner function of_chain_future()which is used to chain the executor's Future (which is not anasyncio.Futurebut aconcurrent.futures.Future) to the future that wraps therun_in_executor()call.Still with me? It looks like
_call_set_state()could simply check whether the destination loop isn't closed, which it does a few lines earlier when it finds the destination future is cancelled.I'll try to come up with a repro that doesn't require one to hit ^C, and take it from there.
That sounds right, you might be able to create reproducer with
_thread.interrupt_main.I have a fix, but not yet a test.
I spent 30-60 minutes trying to come up with a test case, but I'm pretty unhappy with it. The key thing is that we need to aim the keyboard interrupt to happen once
shutdown_default_executor()has entered theawait futureline. I tried various ways to do this but wasn't very successful. The best thing I can think of is to have a loop in the executor thread that sleeps 1 msec at a time and then checks if theloop._executor_shutdown_calledflag is set yet. Once that is set, sleep a little longer (another msec?) and then call_thread.interrupt_main(), in the hope that the extra bit of sleeping is enough to make us reach theawait future.However, when I try this, it doesn't always trigger the bug with Python 3.11rc2 or Python 3.10.
Details
```py import asyncio import _thread import timeasync def main():
print("coro start")
loop = asyncio.get_running_loop()
loop.run_in_executor(None, blocking, loop)
print("coro stop")def blocking(loop):
print("thread start")
while not loop._executor_shutdown_called:
time.sleep(0.001)
print("Got it")
time.sleep(0.001)
_thread.interrupt_main()
print("thread stop")try:
asyncio.run(main())
except KeyboardInterrupt:
print("interrupted!")</details>- added a commit that references this issue
on Sep 30, 2022 5 remaining items
I'm still seeing this issue on Python 3.11.0 on Windows. That is, if I run the test program in the original comment here and hit Ctrl-C after "coro stop" is printed, I get the same exception traceback:
❯ py .\bug.py coro start thread start coro stop interrupted! thread stop exception calling callback for <Future at 0x250844fd1d0 state=finished returned NoneType> Traceback (most recent call last): File "C:\Users\Gustav\AppData\Local\Programs\Python\Python311\Lib\concurrent\futures\_base.py", line 340, in _invoke_callbacks callback(self) File "C:\Users\Gustav\AppData\Local\Programs\Python\Python311\Lib\asyncio\futures.py", line 401, in _call_set_state dest_loop.call_soon_threadsafe(_set_state, destination, source) File "C:\Users\Gustav\AppData\Local\Programs\Python\Python311\Lib\asyncio\base_events.py", line 803, in call_soon_threadsafe self._check_closed() File "C:\Users\Gustav\AppData\Local\Programs\Python\Python311\Lib\asyncio\base_events.py", line 519, in _check_closed raise RuntimeError('Event loop is closed') RuntimeError: Event loop is closed Exception in thread Thread-1 (_do_shutdown): Traceback (most recent call last): File "C:\Users\Gustav\AppData\Local\Programs\Python\Python311\Lib\asyncio\base_events.py", line 580, in _do_shutdown self.call_soon_threadsafe(future.set_result, None) File "C:\Users\Gustav\AppData\Local\Programs\Python\Python311\Lib\asyncio\base_events.py", line 803, in call_soon_threadsafe self._check_closed() File "C:\Users\Gustav\AppData\Local\Programs\Python\Python311\Lib\asyncio\base_events.py", line 519, in _check_closed raise RuntimeError('Event loop is closed') RuntimeError: Event loop is closed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "C:\Users\Gustav\AppData\Local\Programs\Python\Python311\Lib\threading.py", line 1038, in _bootstrap_inner self.run() File "C:\Users\Gustav\AppData\Local\Programs\Python\Python311\Lib\threading.py", line 975, in run self._target(*self._args, **self._kwargs) File "C:\Users\Gustav\AppData\Local\Programs\Python\Python311\Lib\asyncio\base_events.py", line 582, in _do_shutdown self.call_soon_threadsafe(future.set_exception, ex) File "C:\Users\Gustav\AppData\Local\Programs\Python\Python311\Lib\asyncio\base_events.py", line 803, in call_soon_threadsafe self._check_closed() File "C:\Users\Gustav\AppData\Local\Programs\Python\Python311\Lib\asyncio\base_events.py", line 519, in _check_closed raise RuntimeError('Event loop is closed') RuntimeError: Event loop is closedDid the fix make it into the 3.11.0 release?
(I found this issue when trying to debug the same error in a program running a subprocess - I assume that
create_subprocess_execuses the threadpool in the background).I guess we'll have to reopen this. It definitely made it into 3.11 (based on the date it was committed). I am out of bandwidth to personally research this -- do you have time to look into it more? Maybe we only fixed it for Selector event loops and not for the Proactor event loop?
I'm happy to help, but I don't understand enough about asyncio in general, or the proactor/process implementation in particular, to diagnose what the issue might be.
I checked on Linux. It appears fixed on 3.10, but not on 3.11. I will try to find out what makes the difference.
I can't believe. I have installed Python 3.11.0 on Fedora 35 from the Fedora RPMs and the patch for this issue is missing! At least on the
asyncio/base_events.pyfile.On Github:
https://github.com/python/cpython/blob/3.11/Lib/asyncio/base_events.py#L577def _do_shutdown(self, future): try: self._default_executor.shutdown(wait=True) if not self.is_closed(): self.call_soon_threadsafe(future.set_result, None) except Exception as ex: if not self.is_closed(): self.call_soon_threadsafe(future.set_exception, ex)On Fedora, RPM python3.11-libs-3.11.0-1.fc35.x86_64:
def _do_shutdown(self, future): try: self._default_executor.shutdown(wait=True) self.call_soon_threadsafe(future.set_result, None) except Exception as ex: self.call_soon_threadsafe(future.set_exception, ex)The people from Fedora/Redhat used the source tarball to build their RPMs. Here's the thing: I do not see the patched asyncio code in the source tarballs (both xz and gz) on the official download page https://www.python.org/downloads/release/python-3110/ !!
Could somebody please check and confirm that, because I still can't believe it. But if I'm not wrong, it explains the report from @pfmoore.
I have checked that code in my installed copy of Python 3.11.0 on Windows, and I see
def _do_shutdown(self, future): try: self._default_executor.shutdown(wait=True) self.call_soon_threadsafe(future.set_result, None) except Exception as ex: self.call_soon_threadsafe(future.set_exception, ex)So it looks like the patch didn't make it into the Windows 3.11.0 release either 🙁
@pablogsal @zooba is there any possibility that the 3.11 Windows release was done from somewhere other than what's currently on the 3.11 branch?
Edit: Never mind, the fix is not on the 3.11.0 tag - see https://github.com/python/cpython/blob/v3.11.0/Lib/asyncio/base_events.py#L577
So this should be fixed in 3.11.1. That's cool, thanks for the investigation!
Metadata
Metadata
Assignees
Projects
- Status
Test program based on a SO question; press Ctrl-C after "coro stop" and before "thread stop":
The output is as expected:
but a stack trace is printed to stderr:
The interrupt occurs in
asyncio.runat line 49 in the fileasyncio/runners.pyThe control returns to the main program (
except KeyboardInterrupt), but the cleanup waitsuntil the spawned thread terminates and at that time the event loop is closed and the
RuntimeErroroccurs.
In my opinion, no
RuntimeErrorshould happen. Given that the entire user's program is guarded bytry-exceptwhich successfully catches theKeyboardInterrupt, there is very little a programmercan do to avoid this situation maybe except suppressing the output by redirecting the stderr to /dev/null.
Your environment
Python 3.10