-  
 -   Notifications  
You must be signed in to change notification settings  - Fork 19.2k
 
Description
Pandas version checks
-  
I have checked that this issue has not already been reported.
 -  
I have confirmed this bug exists on the latest version of pandas.
 -  
I have confirmed this bug exists on the main branch of pandas.
 
Reproducible Example
from pandas._libs import hashtable as ht class testkey: def __init__(self, val): self._val = val; self.throw = False def __hash__(self): if self.throw: import sys print("exiting") sys.exit(0) return self._val.__hash__() def __eq__(self, other): return self._val.__eq__(other._val) def __repr__(self): return self._val.__repr__() y1 = testkey("hello") y2 = testkey("hello2") x = ht.PyObjectHashTable() x.set_item(y1, 123) x.set_item(y2, 456) print(f"val1: {x.get_item(y2)}") y2.throw = True try: print(f"val2: {x.get_item(y2)}") except KeyError: print("missed key") y2.throw = False print(f"val3: {x.get_item(y2)}")Prints:
val1: 456 exiting missed key val3: 456 or (I guess this depends on some random hash value used when the __hash__ throws)
val1: 456 exiting val2: 456 val3: 456 While I would expect it to print
val1: 456 exiting Issue Description
I'm running to an issue in prod where I have a custom signal handler for SIGINT, which, when raised, will raise a SystemExit exception. Now, sometimes this SystemExit does not result in the application stopping, and the application keeps running.
Debugging it so far, it seems to be this always happens when we're running some pandas code. It's a bit hard to debug, but I think the issue happens due to PyObjectHashTable's handling of exceptions during __hash__ and __eq__ calls. It looks like these are swallowed (or replaced by another exception) under some conditions. I'm not too familiar with the Pandas code, so it was a bit hard to follow where it exactly goes wrong. The example attached, however, tries to show this behavior when we throw in the __hash__ during PyObjectHashTable.get_item(). In this case the SystemExit exception gets replaced by a KeyError.
Expected Behavior
I expect exceptions thrown during __hash__ or __eq__ to not be swallowed.
Installed Versions
>>> import pandas pand>>> pandas.show_versions( ... ) INSTALLED VERSIONS ------------------ commit : fd3f57170aa1af588ba877e8e28c158a20a4886d python : 3.9.6.final.0 python-bits : 64 OS : Darwin OS-release : 22.1.0 Version : Darwin Kernel Version 22.1.0: Sun Oct 9 20:19:12 PDT 2022; root:xnu-8792.41.9~3/RELEASE_ARM64_T6020 machine : arm64 processor : arm byteorder : little LC_ALL : None LANG : None LOCALE : None.UTF-8 pandas : 2.2.0 numpy : 1.26.3 pytz : 2023.3.post1 dateutil : 2.8.2 setuptools : 58.0.4 pip : 23.3.2 Cython : None pytest : None hypothesis : None sphinx : None blosc : None feather : None xlsxwriter : None lxml.etree : None html5lib : None pymysql : None psycopg2 : None jinja2 : None IPython : None pandas_datareader : None adbc-driver-postgresql: None adbc-driver-sqlite : None bs4 : None bottleneck : None dataframe-api-compat : None fastparquet : None fsspec : None gcsfs : None matplotlib : None numba : None numexpr : None odfpy : None openpyxl : None pandas_gbq : None pyarrow : 15.0.0 pyreadstat : None python-calamine : None pyxlsb : None s3fs : None scipy : None sqlalchemy : None tables : None tabulate : None xarray : None xlrd : None zstandard : None tzdata : 2023.4 qtpy : None pyqt5 : None