In Fedora, when building Python 3.11.0a2 with Python 3.11.0a2 for regen, we found that regen-frozen fails. It can be reproduced either on the v3.11.0a2 tag or on the main branch. On tag v3.11.0a2 with 3.11.0a2 installed in $PATH: # in root of cpython local clone: $ git clean -fdx $ mkdir -p build/optimized $ cd build/optimized $ ../../configure $ make regen-all PYTHON_FOR_REGEN=python3.11 ... ERROR: missing _freeze_module On the main branch: # in root of cpython local clone: $ git clean -fdx $ mkdir -p build/optimized $ cd build/optimized $ ../../configure $ make $ make regen-all PYTHON_FOR_REGEN=./python (success) However, the working tree is dirty: $ git diff diff --git a/Parser/parser.c b/Parser/parser.c index b3aa35989ed..c26cd6eeb05 100644 --- a/Parser/parser.c +++ b/Parser/parser.c @@ -1,4 +1,4 @@ -// @generated by pegen from ./Grammar/python.gram +// @generated by pegen from ../../Grammar/python.gram #include "pegen.h" #if defined(Py_DEBUG) && defined(Py_BUILD_CORE) diff --git a/Tools/peg_generator/pegen/grammar_parser.py b/Tools/peg_generator/pegen/grammar_parser.py index 6e9f7d3d11d..fbbbfad76b0 100644 --- a/Tools/peg_generator/pegen/grammar_parser.py +++ b/Tools/peg_generator/pegen/grammar_parser.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3.8 -# @generated by pegen from ./Tools/peg_generator/pegen/metagrammar.gram +# @generated by pegen from ../../Tools/peg_generator/pegen/metagrammar.gram import ast import sys And if we install Python somewhere: $ make $ make install DESTDIR=/tmp/python And use that one again for regen: $ cd ../.. $ git clean -fdx $ mkdir -p build/optimized $ cd build/optimized $ ../../configure $ make regen-all PYTHON_FOR_REGEN=/tmp/python/usr/local/bin/python3.11 ... # Regenerate Lib/keyword.py from Grammar/python.gram and Grammar/Tokens # using Tools/peg_generator/pegen PYTHONPATH=../../Tools/peg_generator /tmp/python/usr/local/bin/python3.11 -m pegen.keywordgen \	../../Grammar/python.gram \	../../Grammar/Tokens \	../../Lib/keyword.py.new /tmp/python/usr/local/bin/python3.11 ../../Tools/scripts/update_file.py ../../Lib/keyword.py ../../Lib/keyword.py.new /tmp/python/usr/local/bin/python3.11 ../../Tools/scripts/freeze_modules.py ERROR: missing _freeze_module make: *** [Makefile:1259: regen-frozen] Error 1 It fails. I've isolated the failure to: $ make regen-frozen PYTHON_FOR_REGEN=/tmp/python/usr/local/bin/python3.11 /tmp/python/usr/local/bin/python3.11 ../../Tools/scripts/freeze_modules.py ERROR: missing _freeze_module make: *** [Makefile:1259: regen-frozen] Error 1

git bisect says: ff8859d965815e8b5af346bd90299cfa5568c855 is the first new commit commit ff8859d965815e8b5af346bd90299cfa5568c855 Author: Victor Stinner <vstinner@python.org> Date: Thu Oct 7 21:19:13 2021 +0200 bpo-45402: Fix test_tools.test_sundry() (GH-28786) Fix test_tools.test_sundry() when Python is built out of tree: fix how the freeze_modules.py tool locates the _freeze_module program. bpo-45402.jlQvep.rst">Misc/NEWS.d/next/Tests/2021-10-07-13-43-01.bpo-45402.jlQvep.rst | 3 +++ Tools/scripts/freeze_modules.py | 7 ++++--- 2 files changed, 7 insertions(+), 3 deletions(-) create mode 100644 bpo-45402.jlQvep.rst">Misc/NEWS.d/next/Tests/2021-10-07-13-43-01.bpo-45402.jlQvep.rst It seems the fix for an out-of-tree build bug introduced another one.

It also seems that it is *not* necessary to use Python 3.11 for regen-frozen to get the failure: $ make regen-frozen PYTHON_FOR_REGEN=python3.10 python3.10 ../../Tools/scripts/freeze_modules.py ERROR: missing _freeze_module make: *** [Makefile:1259: regen-frozen] Error 1 $ make regen-frozen PYTHON_FOR_REGEN=python3.9 python3.9 ../../Tools/scripts/freeze_modules.py ERROR: missing _freeze_module make: *** [Makefile:1259: regen-frozen] Error 1 My guess is that the fix for test_tools.test_sundry() actually broke the real usage. It has: # When building out of the source tree, get the tool from directory # of the Python executable TOOL = os.path.dirname(sys.executable) TOOL = os.path.join(TOOL, 'Programs', '_freeze_module') But sys.executable is *not* the Python we have built, but the Python we use for regen. E.g. when Python for regen is /usr/bin/python3.X, it tries to use /usr/bin/Programs/_freeze_module which obviously is not there. The assumption that sys.executable is the freshly built Python is only correct in the tests, but not in reality.

 New changeset 253b7a0a9fef1d72a4cb87b837885576e68e917c by Victor Stinner in branch 'main': bpo-45866: pegen strips directory of "generated from" header (GH-29777) https://github.com/python/cpython/commit/253b7a0a9fef1d72a4cb87b837885576e68e917c 

 New changeset b6defde2afe656db830d6fedf74ca5f6225f5928 by Victor Stinner in branch '3.10': bpo-45866: pegen strips directory of "generated from" header (GH-29777) (GH-29792) https://github.com/python/cpython/commit/b6defde2afe656db830d6fedf74ca5f6225f5928 

 New changeset 93a540d74c654819ad04d72fcdcf827d0e259999 by Victor Stinner in branch '3.9': bpo-45866: pegen strips directory of "generated from" header (GH-29777) (GH-29792) (GH-29797) https://github.com/python/cpython/commit/93a540d74c654819ad04d72fcdcf827d0e259999 

It seems like all bugs reported in this issue has been fixed, so I close the issue. Thanks for the bug report ;-) > -# @generated by pegen from ./Tools/peg_generator/pegen/metagrammar.gram > +# @generated by pegen from ../../Tools/peg_generator/pegen/metagrammar.gram This issue is now fixed in 3.9, 3.10 and main branches. > ERROR: missing _freeze_module I'm no longer able to reproduce this error on an up-to-date main branch. The issue has been fixed by this change: commit 5be98e57b3c3b36d1a1176b49c73b8822c6380e7 Author: Guido van Rossum <guido@python.org> Date: Tue Nov 23 08:56:06 2021 -0800 bpo-45873: Get rid of bootstrap_python (#29717) Instead we use $(PYTHON_FOR_REGEN) .../deepfreeze.py with the frozen .h file as input, as we did for Windows in bpo-45850. We also get rid of the code that generates the .h files when make regen-frozen is run (i.e., .../make_frozen.py), and the MANIFEST file. Restore Python 3.8 and 3.9 as Windows host Python again Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>

 New changeset 9a7611a7c4da6406383f8ffcea272ded6f890f1e by Victor Stinner in branch 'main': bpo-45866: Fix typo in the NEWS entry (GH-29798) https://github.com/python/cpython/commit/9a7611a7c4da6406383f8ffcea272ded6f890f1e 

 New changeset fc16ea9c8b8769af8a4c0c16fed7eba2e8bf4019 by Miss Islington (bot) in branch '3.10': bpo-45866: Fix typo in the NEWS entry (GH-29798) https://github.com/python/cpython/commit/fc16ea9c8b8769af8a4c0c16fed7eba2e8bf4019 

 New changeset b3f14dacfea54332e2ddde792142d818b3554dbc by Miss Islington (bot) in branch '3.9': bpo-45866: Fix typo in the NEWS entry (GH-29798) https://github.com/python/cpython/commit/b3f14dacfea54332e2ddde792142d818b3554dbc