Skip to content

Dropping concurrent.futures.Executor.map result cancels pending futures #136578

@mirober

Description

@mirober

Bug report

Bug description:

Code:

from concurrent.futures import ThreadPoolExecutor data = [ list(range(0, 5)), list(range(5, 10)), ] def _f(x): print(f"Processing {x}") return True print("=== 1 - No consumption from the iterator ===") executor = ThreadPoolExecutor(max_workers=1) for ints in data: executor.map(_f, ints) executor.shutdown(wait=True) print("=== 2 - Consume all values from the iterator ===") executor = ThreadPoolExecutor(max_workers=1) for ints in data: futures = executor.map(_f, ints) results = list(futures) executor.shutdown(wait=True) print("=== 3 - Consume one value from the iterator ===") executor = ThreadPoolExecutor(max_workers=1) for ints in data: futures = executor.map(_f, ints) first = next(futures) executor.shutdown(wait=True) print("=== 4 - Dropping iterator cancels remaining futures ===") executor = ThreadPoolExecutor(max_workers=1) futures = executor.map(_f, range(0, 5)) first = next(futures) del futures executor.shutdown(wait=True)

Result:

=== 1 - No consumption from the iterator === Processing 0 Processing 1 Processing 2 Processing 3 Processing 4 Processing 5 Processing 6 Processing 7 Processing 8 Processing 9 === 2 - Consume all values from the iterator === Processing 0 Processing 1 Processing 2 Processing 3 Processing 4 Processing 5 Processing 6 Processing 7 Processing 8 Processing 9 === 3 - Consume one value from the iterator === Processing 0 Processing 1 Processing 5 Processing 6 Processing 7 Processing 8 Processing 9 === 4 - Dropping iterator cancels remaining futures === Processing 0 Processing 1 

The behaviour seems to be:

  • If the iterator returned from map is never used (case 1), futures are not cancelled
  • If the iterator returned from map is exhausted (case 2), futures are not cancelled
  • If the iterator returned from map is partially consumed and then dropped (cases 3 & 4), the remaining futures are cancelled

We hit this doing a version of case 3, calling any on the iterator, which short-circuited, causing the remaining futures to not execute. This tripped us up and seems like quite a confusing behaviour that is not flagged in the docs.

It looks like this is caused by this code: https://github.com/python/cpython/blob/main/Lib/concurrent/futures/_base.py#L669-L671

Possibly related to #108518

CPython versions tested on:

3.12

Operating systems tested on:

Linux

Metadata

Metadata

Assignees

No one assigned

    Labels

    stdlibStandard Library Python modules in the Lib/ directorytopic-multiprocessingtype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions