Message339991
This problem was identified in https://bugs.python.org/issue34970 but I think the fix might have been incorrect. The theory in issue34970 was that GC was causing the weakrefset for `all_tasks` to change during iteration. However Weakset provides an `_IterationGuard` class to prevent GC from changing the set during iteration and hence preventing this problem in a single thread. My thoughts on this problem are: - `asyncio.tasks._all_tasks` is shared for all users of asyncio (https://github.com/python/cpython/blob/3.7/Lib/asyncio/tasks.py#L818) - Any new Task constructed mutates `_all_tasks` (https://github.com/python/cpython/blob/3.7/Lib/asyncio/tasks.py#L117) - _IterationGuard won't protect iterations in this case because calls to Weakset.add will always commit changes even if there is something iterating (https://github.com/python/cpython/blob/3.6/Lib/_weakrefset.py#L83) - calls to `asyncio.all_tasks` or `asyncio.tasks.Task.all_tasks` crash if any task is started on any thread during iteration. Repro code: ``` import asyncio from threading import Thread async def do_nothing(): await asyncio.sleep(0) async def loop_tasks(): loop = asyncio.get_event_loop() while True: loop.create_task(do_nothing()) await asyncio.sleep(0.01) def old_thread(): loop = asyncio.new_event_loop() while True: asyncio.tasks.Task.all_tasks(loop=loop) def new_thread(): loop = asyncio.new_event_loop() while True: asyncio.all_tasks(loop=loop) old_t = Thread(target=old_thread) new_t = Thread(target=new_thread) old_t.start() new_t.start() asyncio.run(loop_tasks()) ``` Output: ``` Exception in thread Thread-2: Traceback (most recent call last): File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner self.run() File "/usr/lib/python3.7/threading.py", line 865, in run self._target(*self._args, **self._kwargs) File "tmp/test_asyncio.py", line 25, in new_thread asyncio.all_tasks(loop=loop) File "/usr/lib/python3.7/asyncio/tasks.py", line 40, in all_tasks return {t for t in list(_all_tasks) File "/usr/lib/python3.7/_weakrefset.py", line 60, in __iter__ for itemref in self.data: RuntimeError: Set changed size during iteration Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner self.run() File "/usr/lib/python3.7/threading.py", line 865, in run self._target(*self._args, **self._kwargs) File "tmp/test_asyncio.py", line 19, in old_thread asyncio.tasks.Task.all_tasks(loop=loop) File "/usr/lib/python3.7/asyncio/tasks.py", line 52, in _all_tasks_compat return {t for t in list(_all_tasks) if futures._get_loop(t) is loop} File "/usr/lib/python3.7/_weakrefset.py", line 60, in __iter__ for itemref in self.data: RuntimeError: Set changed size during iteration ``` | |
| Date | User | Action | Args | | 2019-04-11 18:12:48 | Nick Davies | set | recipients: + Nick Davies, asvetlov, yselivanov | | 2019-04-11 18:12:48 | Nick Davies | set | messageid: <1555006368.14.0.78767463153.issue36607@roundup.psfhosted.org> | | 2019-04-11 18:12:48 | Nick Davies | link | issue36607 messages | | 2019-04-11 18:12:47 | Nick Davies | create | | |