python -m test test_unitttest -R3:3 fails #96624
Description
For the last couple months, it seems buildbots have been failing -m test test_unittest -R3:3, then re-trying the 3 failed test cases and passing on the re-try. Look at https://buildbot.python.org/all/#/builders/123 for example. The tests consistently pass the first warmup run, then fail on the second warmup run.
Example stdout from running -m test test_unittest -R3:3 -v:
====================================================================== FAIL: test_loadTestsFromName__module_not_loaded (test.test_unittest.test_loader.Test_TestLoader.test_loadTestsFromName__module_not_loaded) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Users\sween\Source\Repos\cpython2\cpython\Lib\test\test_unittest\test_loader.py", line 594, in test_loadTestsFromName__module_not_loaded self.assertEqual(list(suite), []) AssertionError: Lists differ: [<unittest.loader._FailedTest testMethod=dummy>] != [] First list contains 1 additional elements. First extra element 0: <unittest.loader._FailedTest testMethod=dummy> - [<unittest.loader._FailedTest testMethod=dummy>] + [] ====================================================================== FAIL: test_loadTestsFromNames__module_not_loaded (test.test_unittest.test_loader.Test_TestLoader.test_loadTestsFromNames__module_not_loaded) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Users\sween\Source\Repos\cpython2\cpython\Lib\test\test_unittest\test_loader.py", line 1019, in test_loadTestsFromNames__module_not_loaded self.assertEqual(list(suite), [unittest.TestSuite()]) AssertionError: Lists differ: [<uni[15 chars]tSuite tests=[<unittest.loader._FailedTest testMethod=dummy>]>] != [<uni[15 chars]tSuite tests=[]>] First differing element 0: <unit[13 chars]stSuite tests=[<unittest.loader._FailedTest testMethod=dummy>]> <unit[13 chars]stSuite tests=[]> - [<unittest.suite.TestSuite tests=[<unittest.loader._FailedTest testMethod=dummy>]>] + [<unittest.suite.TestSuite tests=[]>] ====================================================================== FAIL: test_loadTestsFromNames__unknown_attr_name (test.test_unittest.test_loader.Test_TestLoader.test_loadTestsFromNames__unknown_attr_name) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Users\sween\Source\Repos\cpython2\cpython\Lib\test\test_unittest\test_loader.py", line 715, in test_loadTestsFromNames__unknown_attr_name error, test = self.check_deferred_error(loader, list(suite)[0]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\sween\Source\Repos\cpython2\cpython\Lib\test\test_unittest\test_loader.py", line 620, in check_deferred_error self.assertEqual(1, len(loader.errors)) AssertionError: 1 != 2 ---------------------------------------------------------------------- Ran 1001 tests in 8.861s FAILED (failures=3, skipped=55) test_unittest failed (3 failures) == Tests result: FAILURE == 1 test failed: test_unittest Total duration: 18.3 sec Tests result: FAILURE I bisected to here:
commit c735d54
Author: Victor Stinner vstinner@python.org
Date: Tue Jun 21 10:27:59 2022 +0200gh-93839: Move Lib/unttest/test/ to Lib/test/test_unittest/ (#94043) * Move Lib/unittest/test/ to Lib/test/test_unittest/ * Remove Lib/test/test_unittest.py * Replace unittest.test with test.test_unittest * Remove unittest.load_tests() * Rewrite unittest __init__.py and __main__.py * Update build system, CODEOWNERS, and wasm_assets.py
cc @vstinner