Generator finalization is slower in 3.11 vs 3.10 #100762
Description
I found that the Python 3.11.1 implementation of all() is 30% slower compared to Python 3.10.9.
any() also seems to be around 4% slower on my device
Environment
- CPython versions tested on: Python 3.10.9 and Python 3.11.1
- Operating system and architecture: Windows 10 22H2 (Build 19045) 64 bit
Code to test all():
import timeit from functools import partial import platform def find_by_keys( keys: list[str], table: list[dict[str, str | int]], match_data: dict[str, str | int], ) -> dict[str, str | int] | None: for item in table: if all(item[k] == match_data[k] for k in keys): return item return None def main(): keys: list[str] = ["id", "key_1", "key_2"] table: list[dict[str, str | int]] = [ { "id": i, "key_1": "val_1", "key_2": "val_2", } for i in range(1, 5001) ] match_data: dict[str, str | int] = { "id": 3000, "key_1": "val_1", "key_2": "val_2", } timeit_output = timeit.repeat( partial(find_by_keys, keys, table, match_data), repeat=10000, number=1 ) average_time = sum(timeit_output) / len(timeit_output) best_time = min(timeit_output) tps_output = 1 / average_time print(f"Python version = {platform.python_version()}") print(f"Average time = {average_time}") print(f"Best time = {best_time}") print(f"Average transactions per second = {tps_output}") if __name__ == "__main__": main()Results for all()
Console output using Python 3.10.9:
Python version = 3.10.9 Average time = 0.0008256170657486655 Best time = 0.0007106999401003122 Average transactions per second = 1211.2152733824669 Console output using Python 3.11.1:
Python version = 3.11.1 Average time = 0.0011819988898839802 Best time = 0.001033599954098463 Average transactions per second = 846.0244832363215 Code to test any():
import timeit from functools import partial import platform def find_by_keys( keys: list[str], table: list[dict[str, str | int]], match_data: dict[str, str | int], ) -> dict[str, str | int] | None: for item in table: if any(item[k] == match_data[k] for k in keys): return item return None def main(): keys: list[str] = ["id", "key_1", "key_2"] table: list[dict[str, str | int]] = [ { "id": i, "key_1": "foo_1", "key_2": "foo_2", } for i in range(1, 5001) ] match_data: dict[str, str | int] = { "id": 3000, "key_1": "val_1", "key_2": "val_2", } timeit_output = timeit.repeat( partial(find_by_keys, keys, table, match_data), repeat=10000, number=1 ) average_time = sum(timeit_output) / len(timeit_output) best_time = min(timeit_output) tps_output = 1 / average_time print(f"Python version = {platform.python_version()}") print(f"Average time = {average_time}") print(f"Best time = {best_time}") print(f"Average transactions per second = {tps_output}") if __name__ == "__main__": main()Results for any()
Console output using Python 3.10.9:
Python version = 3.10.9 Average time = 0.0009300541202770546 Best time = 0.00090230000205338 Average transactions per second = 1075.206246817238 Console output using Python 3.11.1:
Python version = 3.11.1 Average time = 0.0009645268900785595 Best time = 0.000930099980905652 Average transactions per second = 1036.7777303943815 Linked PRs
- gh-100762: Switch order of exception match in gen_close #101011
- GH-100762: Don't call
gen.throw()ingen.close(), unless necessary. #101013 - [3.11] GH-100762: Don't call gen.throw() in gen.close(), unless necessary #101316
- gh-100762: Fix optimization in gen_close #111069
- [3.12] gh-100762: Fix optimization in gen_close (GH-111069) #115818