Improved Errors for Missing Standard Library Modules

Ideas
1

Python is often being built or distributed without its full standard library. It would be nice if the error messages hinted at why a particular module is missing, and perhaps what to do about it.

Myself and @encukou propose a mechanism to improve error messages by default, and for allowing distributors to provide more informative error messages to users when removed standard library
modules fail to import. (Note that this supersedes the remaining parts of PEP 534 that didn’t already get implemented since the PEP was proposed.)

For example, if CPython is built without the third-party zlib library:

 >>> import zlib Traceback (most recent call last): File "<python-input-0>", line 1, in <module> import zlib ModuleNotFoundError: No module named 'zlib'

An example from Fedora 42 and Python 3.14, when the python3.14-tkinter
package is not installed:

 >>> import tkinter Traceback (most recent call last): File "<python-input-0>", line 1, in <module> import tkinter ModuleNotFoundError: No module named 'tkinter'

Proposed changes

At the end of the build, the check_extension_modules.py script will compile information about why modules are missing to a _stdlib_module_info module. This will then be used in the traceback module for nicer error messages.

To allow distributors to add platform-specific information, let’s add a new configure option, --with-missing-stdlib-config=FILE. This will take the location of a JSON file that maps module names
to error messages. For example:

 { "_tkinter": "Install the python-tkinter package to use tkinter", }

This would provide a cleaner solution than the patching of specific modules (e.g. Debian patch) or
sys.excepthook as is currently done by several distributors.

A draft implementation can be found on here.

What do you think?

4 Likes
2

ccing Fedora team: @hroncok @ksurma. … And @torsava – remember PEP 534? :‍)

3 Likes
3

As I think you know, tkinter is a seperate RPM sudo dnf install python3-tkinter should fix that.

Not everyone needs tkinter so it is not installed by default on Fedora, however it is installed for me (maybe its a weak dependency?).

I remember that I needed to install a lot more .deb on debian/ubuntu to get things like venv and pip. For Fedora most modules are installed by default.

This cannot detect how the built modules are broken up into packages. It will only detect that a module was not built.

Typically a build is broken into logical packages based on the distro’s rules. For example tkinter being a seperate package.

4

This seems to be factored in already? The contents of the JSON file passed to --with-missing-stdlib-config is propagated into _stdlib_modules_info.py even if, as far as the build is concerned, the referenced modules “aren’t missing” (at build time). Then at runtime, the criteria for using the custom messages is just that the module is not found – not that it was not compiled in at build time so whether the module is missing due to not being compiled in or because it was sliced off into a subpackage by the distro shouldn’t matter.

> cat missing.json {"_tkinter": "Install the python-tkinter package to use tkinter"} > ./configure --with-missing-stdlib-config=missing.json > make > ./python -c 'import tkinter' # no error )> ./python -c 'import sys; sys.modules["_tkinter"] = None; import tkinter' Traceback (most recent call last): File "<string>", line 1, in <module> import sys; sys.modules["_tkinter"] = None; import tkinter ^^^^^^^^^^^^^^ File "/e/notebooks/cpython/Lib/tkinter/__init__.py", line 38, in <module> import _tkinter # If this fails your Python may not be configured for Tk ^^^^^^^^^^^^^^^ ModuleNotFoundError: Install the python-tkinter package to use tkinter # Mimic a distro scenario with python3 but not python3-tkinter > rm ./build/lib.linux-x86_64-3.15/_tkinter.cpython-315-x86_64-linux-gnu.so > ./python -c 'import tkinter' Traceback (most recent call last): File "<string>", line 1, in <module> import tkinter File "/e/notebooks/cpython/Lib/tkinter/__init__.py", line 38, in <module> import _tkinter # If this fails your Python may not be configured for Tk ^^^^^^^^^^^^^^^ ModuleNotFoundError: Install the python-tkinter package to use tkinter

One thing this does highlight is that it makes the vague # If this fails your Python may not be configured for Tk comment kind of distracting. Maybe that should be removed?

3 Likes
5

Thank you Brénainn for your well explained answer!

As for the comment in tkinter, that is a future discussion, we need to get this implemented first:-)

6

This looks nice and simple, and we would use it, thanks!

Suppose in Fedora, both _tkinter and tkinter modules are part of the python3-tkinter RPM package. That means we would need to list them both here, because otherwise Python doesn’t know tkinter uses _tkinter (when neither of them is installed). Perhaps the example could be extended in this way to be more useful?

That would also solve the “confusing comment” part mentioned by @bwoodsend.

2 Likes
7

Yes. The docs should mention that you need to list all modules that are removed or packaged separately.
(You might want to autogenerate the file.)

Yes. And this proposal is for a mechanism that Fedora packagers can use to put sudo dnf install python3-tkinter in the error message that users see.

(Fun fact: on Fedora it is a conditional weak dependency, you get the bindings by default if you install both Python and Tk.)

1 Like
8

The proposal seems sound. We could certainly use it in Debian.

We’d probably use it for gdbm, tk, ensurepip, as well as the bits of stdlib that aren’t available from python3-minimal.

@doko42 also wondered, when this came up on our list:

That could be extended to all available modules in a distro, hinting to the installation of the corresponding package.
In this case, you could build up a database of module → package mappings like it is done for the command-not-found command.
Instead of having a single json file, maybe have an approach to scan a dir for multiple json files?

I wondered the same thing myself, when I saw this proposal. It would be a usability improvement for our users. And (at least in Debian’s case), we have the mapping available to provide to cPython.

2 Likes
9

Thank you for the feedback!

As I noted my email reply, providing such a mechanism for all modules is out-of-scope for this proposal, we may return to it in the future, but we would like to get this done first.

11

The problem here is that the mapping of module names to installable packages would need to be dynamic. The standard library can safely contain a description of itself, but repository contents (and the set of enabled repositories) and aren’t synced with Python updates.
For your use case, I think a custom, package-manager-specific excepthook would work better.

1 Like
12

If you accept an optional mapping of python-package name to distro-package name at build time I’d assume that could fix the UX issue.

Then debian and Fedora can provide their own disto specific mappings.

13

Yeah, we don’t need to support all modules, we were just querying the scope of this.

And yes, if we did try to support all of them, the JSON table would have to be loaded at runtime, not compile-time.

1 Like