-
- Notifications
You must be signed in to change notification settings - Fork 33.4k
Open
Labels
stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directorytopic-multiprocessingtype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error
Description
Bug report
Bug description:
Code:
from concurrent.futures import ThreadPoolExecutor data = [ list(range(0, 5)), list(range(5, 10)), ] def _f(x): print(f"Processing {x}") return True print("=== 1 - No consumption from the iterator ===") executor = ThreadPoolExecutor(max_workers=1) for ints in data: executor.map(_f, ints) executor.shutdown(wait=True) print("=== 2 - Consume all values from the iterator ===") executor = ThreadPoolExecutor(max_workers=1) for ints in data: futures = executor.map(_f, ints) results = list(futures) executor.shutdown(wait=True) print("=== 3 - Consume one value from the iterator ===") executor = ThreadPoolExecutor(max_workers=1) for ints in data: futures = executor.map(_f, ints) first = next(futures) executor.shutdown(wait=True) print("=== 4 - Dropping iterator cancels remaining futures ===") executor = ThreadPoolExecutor(max_workers=1) futures = executor.map(_f, range(0, 5)) first = next(futures) del futures executor.shutdown(wait=True)Result:
=== 1 - No consumption from the iterator === Processing 0 Processing 1 Processing 2 Processing 3 Processing 4 Processing 5 Processing 6 Processing 7 Processing 8 Processing 9 === 2 - Consume all values from the iterator === Processing 0 Processing 1 Processing 2 Processing 3 Processing 4 Processing 5 Processing 6 Processing 7 Processing 8 Processing 9 === 3 - Consume one value from the iterator === Processing 0 Processing 1 Processing 5 Processing 6 Processing 7 Processing 8 Processing 9 === 4 - Dropping iterator cancels remaining futures === Processing 0 Processing 1 The behaviour seems to be:
- If the iterator returned from
mapis never used (case 1), futures are not cancelled - If the iterator returned from
mapis exhausted (case 2), futures are not cancelled - If the iterator returned from
mapis partially consumed and then dropped (cases 3 & 4), the remaining futures are cancelled
We hit this doing a version of case 3, calling any on the iterator, which short-circuited, causing the remaining futures to not execute. This tripped us up and seems like quite a confusing behaviour that is not flagged in the docs.
It looks like this is caused by this code: https://github.com/python/cpython/blob/main/Lib/concurrent/futures/_base.py#L669-L671
Possibly related to #108518
CPython versions tested on:
3.12
Operating systems tested on:
Linux
Metadata
Metadata
Assignees
Labels
stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directorytopic-multiprocessingtype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error