BUG: testing.assert_frame_equal unhelpful error message for string[pyarrow] #54190
Description
Pandas version checks
-
I have checked that this issue has not already been reported.
-
I have confirmed this bug exists on the latest version of pandas.
-
I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
import pandas as pd df = ( pd.DataFrame({"a": [1, 2, 3], "b": [1.0, 2.0, None], "c": [None, "x", "y"]}) .astype({"a": "int64[pyarrow]", "b": "float64[pyarrow]", "c": "string[pyarrow]"}) .set_index("a") ) df.to_parquet("demo.parquet") df2 = pd.read_parquet("demo.parquet", dtype_backend="pyarrow") pd.testing.assert_frame_equal(df, df2)results in the error message
AssertionError: Attributes of DataFrame.iloc[:, 1] (column name="c") are different Attribute "dtype" are different [left]: string[pyarrow] [right]: string[pyarrow] Issue Description
The problem is that
print(type(df.dtypes[-1])) # string_.StringDtype' print(type(df2.dtypes[-1])) # arrow.dtype.ArrowDtype have the same string representation
import pyarrow as pa import pandas as pd print(repr(pd.ArrowDtype(pa.string()))) # string[pyarrow] print(repr(pd.StringDtype("pyarrow"))) # string[pyarrow]Expected Behavior
StringDtype("pyarrow")andArrowDtype("string")need differentreprtesting.assert_frame_equalshould consider printing not justrepr(dtype), but alsorepr(type(dtype)).pd.read_parquetshould round trip both index and column dtypes
Additional context
As per https://pandas.pydata.org/docs/user_guide/pyarrow.html
The string alias
"string[pyarrow]"maps topd.StringDtype("pyarrow")which is not equivalent to specifyingdtype=pd.ArrowDtype(pa.string()). Generally, operations on the data will behave similarly exceptpd.StringDtype("pyarrow")can return NumPy-backed nullable types whilepd.ArrowDtype(pa.string())will returnArrowDtype.
Installed Versions
INSTALLED VERSIONS ------------------ commit : 0f437949513225922d851e9581723d82120684a6 python : 3.11.3.final.0 python-bits : 64 OS : Linux OS-release : 5.19.0-46-generic Version : #47~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Jun 21 15:35:31 UTC 2 machine : x86_64 processor : x86_64 byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : en_US.UTF-8 pandas : 2.0.3 numpy : 1.24.3 pytz : 2023.3 dateutil : 2.8.2 setuptools : 68.0.0 pip : 23.2 Cython : 0.29.36 pytest : 7.4.0 hypothesis : None sphinx : 7.0.1 blosc : None feather : None xlsxwriter : None lxml.etree : 4.9.3 html5lib : None pymysql : 1.4.6 psycopg2 : None jinja2 : 3.1.2 IPython : 8.14.0 pandas_datareader: None bs4 : 4.12.2 bottleneck : None brotli : None fastparquet : 2023.7.0 fsspec : 2023.6.0 gcsfs : None matplotlib : 3.7.2 numba : 0.57.1 numexpr : 2.8.4 odfpy : None openpyxl : 3.1.2 pandas_gbq : None pyarrow : 12.0.1 pyreadstat : None pyxlsb : None s3fs : None scipy : 1.11.1 snappy : None sqlalchemy : 1.4.49 tables : 3.8.0 tabulate : 0.9.0 xarray : 2023.7.0 xlrd : None zstandard : None tzdata : 2023.3 qtpy : None pyqt5 : None