Don't fail entirely when can't install requirements for a specific benchmark

[mdboom]

When running multiple benchmarks, pyperformance first creates a venv with the dependencies for each benchmark. More specifically, it actually first tries to install the dependencies into a "global" venv (to reduce the number of total venvs created), and if that fails installs into a new venv specific to the benchmark. If something is broken in one of the dependencies (usually it's because upstream CPython broke backward compatibility on purpose and the dependency no longer compiles), it fails on both attempts, and pyperformance just moves along to the next benchmark.

However, a recent change to setuptools now leaves the package cache in a bad state on the first attempt such that installing wheel on the second attempt fails. This creates an exception where one wasn't expected and the whole pyperformance run comes to a halt.

I think it's safe to expect that creating the venv fails in this way on the second attempt and we should handle it gracefully.

I also plan to investigate the setuptools change further and report there, but in the meantime, I think it makes sense to make pyperformance more robust to that.

================================================== (58/81) creating venv for benchmark (pytorch_alexnet_inference) (trying common venv first) Installing requirements into the virtual environment C:\actions-runner\_work\benchmarking\benchmarking\venv\cpython3.14-2a0759990ce7-compat-af30061f91bc # 'C:\actions-runner\_work\benchmarking\benchmarking\venv\cpython3.14-2a0759990ce7-compat-af30061f91bc\Scripts\python.exe' -m pip install future==0.18.2 numpy==1.19.0 Pillow==8.0.0 torch==1.5.1 torchvision==0.6.1 'pyperf @ git+https://github.com/psf/pyperf@7b9a23adb3ceeb182f58a03e453158dbe081a7e7' Collecting pyperf@ git+https://github.com/psf/pyperf@7b9a23adb3ceeb182f58a03e453158dbe081a7e7 Using cached pyperf-2.7.0-py3-none-any.whl Collecting future==0.18.2 Using cached future-0.18.2.tar.gz (829 kB) Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Collecting numpy==1.19.0 Using cached numpy-1.19.0.zip (7.3 MB) Installing build dependencies: started Installing build dependencies: finished with status 'done' Getting requirements to build wheel: started Getting requirements to build wheel: finished with status 'done' Preparing metadata (pyproject.toml): started Preparing metadata (pyproject.toml): finished with status 'error' error: subprocess-exited-with-error Preparing metadata (pyproject.toml) did not run successfully. exit code: 1 [102 lines of output] Running from numpy source directory. <string>:460: UserWarning: Unrecognized setuptools command, proceeding with generating Cython sources and expanding templates C:\Windows\ServiceProfiles\NetworkService\AppData\Local\Temp\pip-install-6s5v8bw7\numpy_987489534d5543d08fa6076fdcc0d42d\tools\cythonize.py:73: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. required_version = LooseVersion('0.29.14') C:\Windows\ServiceProfiles\NetworkService\AppData\Local\Temp\pip-install-6s5v8bw7\numpy_987489534d5543d08fa6076fdcc0d42d\tools\cythonize.py:75: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. if LooseVersion(cython_version) < required_version: performance hint: _common.pyx:275:19: Exception check after calling 'random_func' will always require the GIL to be acquired. Declare 'random_func' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions. performance hint: _common.pyx:299:19: Exception check after calling 'random_func' will always require the GIL to be acquired. Declare 'random_func' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions. performance hint: _common.pyx:322:50: Exception check after calling 'random_func' will always require the GIL to be acquired. Declare 'random_func' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions. performance hint: _common.pyx:426:31: Exception check after calling 'f' will always require the GIL to be acquired. Declare 'f' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions. performance hint: _common.pyx:465:31: Exception check after calling 'f' will always require the GIL to be acquired. Declare 'f' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions. performance hint: _common.pyx:509:31: Exception check after calling 'f' will always require the GIL to be acquired. Declare 'f' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions. performance hint: _common.pyx:592:36: Exception check after calling 'f0' will always require the GIL to be acquired. Declare 'f0' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions. performance hint: _common.pyx:596:36: Exception check after calling 'f1' will always require the GIL to be acquired. Declare 'f1' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions. performance hint: _common.pyx:600:36: Exception check after calling 'f2' will always require the GIL to be acquired. Declare 'f2' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions. performance hint: _common.pyx:604:36: Exception check after calling 'f3' will always require the GIL to be acquired. Declare 'f3' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions. performance hint: _common.pyx:638:31: Exception check after calling 'f' will always require the GIL to be acquired. Declare 'f' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions. performance hint: _common.pyx:675:31: Exception check after calling 'f' will always require the GIL to be acquired. Declare 'f' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions. performance hint: _common.pyx:712:63: Exception check after calling 'f' will always require the GIL to be acquired. Declare 'f' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions. performance hint: _common.pyx:754:31: Exception check after calling 'f' will always require the GIL to be acquired. Declare 'f' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions. performance hint: _common.pyx:785:31: Exception check after calling 'f' will always require the GIL to be acquired. Declare 'f' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions. performance hint: _common.pyx:903:40: Exception check after calling 'f0' will always require the GIL to be acquired. Declare 'f0' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions. performance hint: _common.pyx:907:40: Exception check after calling 'fd' will always require the GIL to be acquired. Declare 'fd' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions. performance hint: _common.pyx:911:41: Exception check after calling 'fdd' will always require the GIL to be acquired. Declare 'fdd' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions. performance hint: _common.pyx:916:40: Exception check after calling 'fi' will always require the GIL to be acquired. Declare 'fi' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions. performance hint: _common.pyx:920:41: Exception check after calling 'fdi' will always require the GIL to be acquired. Declare 'fdi' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions. performance hint: _common.pyx:924:38: Exception check after calling 'fiii' will always require the GIL to be acquired. Declare 'fiii' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions. performance hint: _common.pyx:960:31: Exception check after calling 'f' will always require the GIL to be acquired. Declare 'f' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions. performance hint: _common.pyx:1002:32: Exception check after calling 'f1' will always require the GIL to be acquired. Declare 'f1' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions. performance hint: _generator.pyx:706:41: Exception check after calling '_shuffle_int' will always require the GIL to be acquired. Possible solutions: 1. Declare '_shuffle_int' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions. 2. Use an 'int' return type on '_shuffle_int' to allow an error code to be returned. performance hint: _generator.pyx:735:45: Exception check after calling '_shuffle_int' will always require the GIL to be acquired. Possible solutions: 1. Declare '_shuffle_int' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions. 2. Use an 'int' return type on '_shuffle_int' to allow an error code to be returned. Error compiling Cython file: ------------------------------------------------------------ ... for i in range(1, RK_STATE_LEN): self.rng_state.key[i] = val[i] self.rng_state.pos = i self._bitgen.state = &self.rng_state self._bitgen.next_uint64 = &mt19937_uint64 ^ ------------------------------------------------------------ _mt19937.pyx:138:35: Cannot assign type 'uint64_t (*)(void *) except? -1 nogil' to 'uint64_t (*)(void *) noexcept nogil'. Exception values are incompatible. Suggest adding 'noexcept' to the type of the value being assigned. Processing numpy/random\_bounded_integers.pxd.in Processing numpy/random\bit_generator.pyx Processing numpy/random\mtrand.pyx Processing numpy/random\_bounded_integers.pyx.in Processing numpy/random\_common.pyx Processing numpy/random\_generator.pyx Processing numpy/random\_mt19937.pyx Traceback (most recent call last): File "C:\Windows\ServiceProfiles\NetworkService\AppData\Local\Temp\pip-install-6s5v8bw7\numpy_987489534d5543d08fa6076fdcc0d42d\tools\cythonize.py", line 235, in <module> main() ~~~~^^ File "C:\Windows\ServiceProfiles\NetworkService\AppData\Local\Temp\pip-install-6s5v8bw7\numpy_987489534d5543d08fa6076fdcc0d42d\tools\cythonize.py", line 231, in main find_process_files(root_dir) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^ File "C:\Windows\ServiceProfiles\NetworkService\AppData\Local\Temp\pip-install-6s5v8bw7\numpy_987489534d5543d08fa6076fdcc0d42d\tools\cythonize.py", line 222, in find_process_files process(root_dir, fromfile, tofile, function, hash_db) ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Windows\ServiceProfiles\NetworkService\AppData\Local\Temp\pip-install-6s5v8bw7\numpy_987489534d5543d08fa6076fdcc0d42d\tools\cythonize.py", line 188, in process processor_function(fromfile, tofile) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^ File "C:\Windows\ServiceProfiles\NetworkService\AppData\Local\Temp\pip-install-6s5v8bw7\numpy_987489534d5543d08fa6076fdcc0d42d\tools\cythonize.py", line 77, in process_pyx subprocess.check_call( ~~~~~~~~~~~~~~~~~~~~~^ [sys.executable, '-m', 'cython'] + flags + ["-o", tofile, fromfile]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\actions-runner\_work\benchmarking\benchmarking\cpython\Lib\subprocess.py", line 419, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['C:\\actions-runner\\_work\\benchmarking\\benchmarking\\venv\\cpython3.14-2a0759990ce7-compat-af30061f91bc\\Scripts\\python.exe', '-m', 'cython', '-3', '--fast-fail', '-o', '_mt19937.c', '_mt19937.pyx']' returned non-zero exit status 1. Cythonizing sources Traceback (most recent call last): File "C:\actions-runner\_work\benchmarking\benchmarking\venv\cpython3.14-2a0759990ce7-compat-af30061f91bc\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 353, in <module> main() ~~~~^^ File "C:\actions-runner\_work\benchmarking\benchmarking\venv\cpython3.14-2a0759990ce7-compat-af30061f91bc\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 335, in main json_out['return_val'] = hook(**hook_input['kwargs']) ~~~~^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\actions-runner\_work\benchmarking\benchmarking\venv\cpython3.14-2a0759990ce7-compat-af30061f91bc\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 149, in prepare_metadata_for_build_wheel return hook(metadata_directory, config_settings) File "C:\Windows\ServiceProfiles\NetworkService\AppData\Local\Temp\pip-build-env-cg0x_j4n\overlay\Lib\site-packages\setuptools\build_meta.py", line 368, in prepare_metadata_for_build_wheel self.run_setup() ~~~~~~~~~~~~~~^^ File "C:\Windows\ServiceProfiles\NetworkService\AppData\Local\Temp\pip-build-env-cg0x_j4n\overlay\Lib\site-packages\setuptools\build_meta.py", line 497, in run_setup super().run_setup(setup_script=setup_script) ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Windows\ServiceProfiles\NetworkService\AppData\Local\Temp\pip-build-env-cg0x_j4n\overlay\Lib\site-packages\setuptools\build_meta.py", line 313, in run_setup exec(code, locals()) ~~~~^^^^^^^^^^^^^^^^ File "<string>", line 489, in <module> File "<string>", line 469, in setup_package File "<string>", line 274, in generate_cython RuntimeError: Running cythonize failed! [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed Encountered error while generating package metadata. See above for output. note: This is an issue with the package mentioned above, not pip. hint: See above for details. Notice: A new release of pip is available: 24.1.1 -> 24.1.2 Notice: To update, run: C:\actions-runner\_work\benchmarking\benchmarking\venv\cpython3.14-2a0759990ce7-compat-af30061f91bc\Scripts\python.exe -m pip install --upgrade pip Command failed with exit code 1 (falling back to unique venv) Creating the virtual environment C:\actions-runner\_work\benchmarking\benchmarking\venv\cpython3.14-2a0759990ce7-compat-af30061f91bc-bm-pytorch_alexnet_inference # 'C:\actions-runner\_work\benchmarking\benchmarking\cpython\PCBuild\amd64\python.exe' -m venv --without-pip 'C:\actions-runner\_work\benchmarking\benchmarking\venv\cpython3.14-2a0759990ce7-compat-af30061f91bc-bm-pytorch_alexnet_inference' # 'C:\actions-runner\_work\benchmarking\benchmarking\venv\cpython3.14-2a0759990ce7-compat-af30061f91bc-bm-pytorch_alexnet_inference\Scripts\python.exe' -m ensurepip -v -U Using pip 24.1.1 from C:\WINDOWS\SERVIC~1\NETWOR~1\AppData\Local\Temp\tmptyeix318\pip-24.1.1-py3-none-any.whl\pip (python 3.14) Looking in links: c:\WINDOWS\SERVIC~1\NETWOR~1\AppData\Local\Temp\tmptyeix318 Processing c:\windows\servic~1\networ~1\appdata\local\temp\tmptyeix318\pip-24.1.1-py3-none-any.whl Installing collected packages: pip Successfully installed pip-24.1.1 # 'C:\actions-runner\_work\benchmarking\benchmarking\venv\cpython3.14-2a0759990ce7-compat-af30061f91bc-bm-pytorch_alexnet_inference\Scripts\python.exe' -m pip install -U 'setuptools>=18.5' wheel Collecting setuptools>=18.5 Using cached setuptools-71.0.3-py3-none-any.whl.metadata (6.5 kB) Collecting wheel Using cached wheel-0.43.0-py3-none-any.whl.metadata (2.2 kB) Using cached setuptools-71.0.3-py3-none-any.whl (2.3 MB) Using cached wheel-0.43.0-py3-none-any.whl (65 kB) Installing collected packages: wheel, setuptools ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: 'C:\\actions-runner\\_work\\benchmarking\\benchmarking\\venv\\cpython3.14-2a0759990ce7-compat-af30061f91bc-bm-pytorch_alexnet_inference\\Lib\\site-packages\\pkg_resources\\tests\\data\\my-test-package_unpacked-egg\\my_test_package-1.0-py3.7.egg\\EGG-INFO\\dependency_links.txt' Notice: A new release of pip is available: 24.1.1 -> 24.1.2 Notice: To update, run: C:\actions-runner\_work\benchmarking\benchmarking\venv\cpython3.14-2a0759990ce7-compat-af30061f91bc-bm-pytorch_alexnet_inference\Scripts\python.exe -m pip install --upgrade pip Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "C:\actions-runner\_work\benchmarking\benchmarking\venv\Lib\site-packages\pyperformance\__main__.py", line 2, in <module> pyperformance.cli.main() File "C:\actions-runner\_work\benchmarking\benchmarking\venv\Lib\site-packages\pyperformance\cli.py", line 328, in main _main() File "C:\actions-runner\_work\benchmarking\benchmarking\venv\Lib\site-packages\pyperformance\cli.py", line 312, in _main cmd_run(options, benchmarks) File "C:\actions-runner\_work\benchmarking\benchmarking\venv\Lib\site-packages\pyperformance\commands.py", line 180, in cmd_run suite, errors = run_benchmarks(benchmarks, executable, options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\actions-runner\_work\benchmarking\benchmarking\venv\Lib\site-packages\pyperformance\run.py", line 111, in run_benchmarks venv = VenvForBenchmarks.ensure( ^^^^^^^^^^^^^^^^^^^^^^^^^ Command failed with exit code 1 File "C:\actions-runner\_work\benchmarking\benchmarking\venv\Lib\site-packages\pyperformance\venv.py", line 197, in ensure return cls.create( ^^^^^^^^^^^ File "C:\actions-runner\_work\benchmarking\benchmarking\venv\Lib\site-packages\pyperformance\venv.py", line 164, in create self.ensure_pip(upgrade=upgrade) File "C:\actions-runner\_work\benchmarking\benchmarking\venv\Lib\site-packages\pyperformance\_venv.py", line 230, in ensure_pip raise RequirementsInstallationFailedError('wheel') pyperformance._venv.RequirementsInstallationFailedError: wheel Remove directory C:\actions-runner\_work\benchmarking\benchmarking\venv\cpython3.14-2a0759990ce7-compat-af30061f91bc-bm-pytorch_alexnet_inference 

[ericsnowcurrently]

LGTM