Skip to content

gh-136380: Fix import behavior for concurrent.futures.InterpreterPoolExecutor #136381

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 15 commits into from
Jul 8, 2025

Conversation

aisk
Copy link
Contributor

@aisk aisk commented Jul 7, 2025

It's hard to write the test, but on my local machine, when remove the _interpreters module or raise a ModuleNotFoundError in the try branch, the behavior is correct compare to a normal non-exist field:

>>> from concurrent.futures import InterpreterPoolExecutor Traceback (most recent call last): File "<python-input-0>", line 1, in <module> from concurrent.futures import InterpreterPoolExecutor ImportError: cannot import name 'InterpreterPoolExecutor' from 'concurrent.futures' (C:\Users\xxxxx\Source\cpython\Lib\concurrent\futures\__init__.py) >>> from concurrent import futures >>> futures.InterpreterPoolExecutor Traceback (most recent call last): File "<python-input-2>", line 1, in <module> futures.InterpreterPoolExecutor File "C:\Users\xxxxx\Source\cpython\Lib\concurrent\futures\__init__.py", line 69, in __getattr__ raise AttributeError(f"module {__name__!r} has no attribute {name!r}") AttributeError: module 'concurrent.futures' has no attribute 'InterpreterPoolExecutor' 
>>> from concurrent.futures import NotExist Traceback (most recent call last): File "<python-input-0>", line 1, in <module> from concurrent.futures import NotExist ImportError: cannot import name 'NotExist' from 'concurrent.futures' (C:\Users\xxxxx\Source\cpython\Lib\concurrent\futures\__init__.py) >>> from concurrent import futures >>> futures.NotExist Traceback (most recent call last): File "<python-input-2>", line 1, in <module> futures.NotExist File "C:\Users\xxxxx\Source\cpython\Lib\concurrent\futures\__init__.py", line 69, in __getattr__ raise AttributeError(f"module {__name__!r} has no attribute {name!r}") AttributeError: module 'concurrent.futures' has no attribute 'NotExist' 

We must import _interpreters in __init__.py to determine whether to append the InterpreterPoolExecutor to __all__. This may have a negative impact on importing performance.

@aisk aisk changed the title Fix import behavior for concurrent.futures.InterpreterPoolExecutor gh-136380: Fix import behavior for concurrent.futures.InterpreterPoolExecutor Jul 7, 2025
Copy link
Member

@serhiy-storchaka serhiy-storchaka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The downside of this is that _interpreters is now always imported.

Could you add tests? It is not easy, the tests should remove concurrent and all its submodules recursively from sys.module, and restore sys.module at the end. There are helpers for this somewhere, they can be private.

Copy link
Member

@ZeroIntensity ZeroIntensity left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

aisk and others added 4 commits July 8, 2025 01:20
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
…b_nXl.rst Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
@aisk
Copy link
Contributor Author

aisk commented Jul 7, 2025

The downside of this is that _interpreters is now always imported.

Could you add tests? It is not easy, the tests should remove concurrent and all its submodules recursively from sys.module, and restore sys.module at the end. There are helpers for this somewhere, they can be private.

Found a way to test it by running the test within a subprocess.

Copy link
Member

@serhiy-storchaka serhiy-storchaka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. 👍

@ZeroIntensity
Copy link
Member

I didn't mean to hit the update button, sorry about that.

aisk and others added 3 commits July 8, 2025 22:02
Co-authored-by: sobolevn <mail@sobolevn.me>
Co-authored-by: sobolevn <mail@sobolevn.me>
Copy link
Member

@sobolevn sobolevn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@sobolevn sobolevn enabled auto-merge (squash) July 8, 2025 13:18
@sobolevn sobolevn added the needs backport to 3.14 bugs and security fixes label Jul 8, 2025
@sobolevn sobolevn merged commit 490eea0 into python:main Jul 8, 2025
42 checks passed
@miss-islington-app
Copy link

Thanks @aisk for the PR, and @sobolevn for merging it 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jul 8, 2025
…terPoolExecutor` (pythonGH-136381) (cherry picked from commit 490eea0) Co-authored-by: AN Long <aisk@users.noreply.github.com> Co-authored-by: Serhiy Storchaka <storchaka@gmail.com> Co-authored-by: Peter Bierma <zintensitydev@gmail.com> Co-authored-by: sobolevn <mail@sobolevn.me>
@bedevere-app
Copy link

bedevere-app bot commented Jul 8, 2025

GH-136420 is a backport of this pull request to the 3.14 branch.

@bedevere-app bedevere-app bot removed the needs backport to 3.14 bugs and security fixes label Jul 8, 2025
@aisk aisk deleted the import-interpreter-pool-executor branch July 8, 2025 13:36
sobolevn added a commit that referenced this pull request Jul 8, 2025
…eterPoolExecutor` (GH-136381) (#136420) gh-136380: Fix import behavior for `concurrent.futures.InterpreterPoolExecutor` (GH-136381) (cherry picked from commit 490eea0) Co-authored-by: AN Long <aisk@users.noreply.github.com> Co-authored-by: Serhiy Storchaka <storchaka@gmail.com> Co-authored-by: Peter Bierma <zintensitydev@gmail.com> Co-authored-by: sobolevn <mail@sobolevn.me>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
4 participants