BUG: pd.concat() produces invalid index when inputs have Int64Dtype index with NAs #62903
Description
Pandas version checks
-
I have checked that this issue has not already been reported.
-
I have confirmed this bug exists on the latest version of pandas.
-
I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
Construct index using MultiIndex.from_product():
levels1 = ['a', 'b'] levels2 = pd.Series([1, 2, pd.NA], dtype=pd.Int64Dtype()) index1 = pd.MultiIndex.from_product([levels1, levels2], names=['one', 'two']) series1 = pd.Series([f'{i1}-{i2}' for i1, i2 in index1], index=index1) series1one two a 1 a-1 2 a-2 <NA> a-<NA> b 1 b-1 2 b-2 <NA> b-<NA> dtype: object Split series by first index level and recombine using pd.concat():
series2 = pd.concat([series1.loc[i1] for i1 in levels1], keys=levels1, names=['one']) series2one two a 1 a-1 2 a-2 <NA> a-<NA> b 1 b-1 2 b-2 <NA> b-<NA> dtype: object Series 1 ok:
def check(series): for ix in series.index: print(repr(ix), end=': ') print(repr(series.at[ix])) check(series1)('a', 1): 'a-1' ('a', 2): 'a-2' ('a', <NA>): 'a-<NA>' ('b', 1): 'b-1' ('b', 2): 'b-2' ('b', <NA>): 'b-<NA>' check(series2)('a', 1): 'a-1' ('a', 2): 'a-2' ('a', <NA>): --------------------------------------------------------------------------- KeyError Traceback (most recent call last) File ~/opt/mambaforge/envs/myenv/lib/python3.12/site-packages/pandas/core/indexes/multi.py:3072, in MultiIndex.get_loc(self, key) 3071 try: -> 3072 return self._engine.get_loc(key) 3073 except KeyError as err: File pandas/_libs/index.pyx:794, in pandas._libs.index.BaseMultiIndexCodesEngine.get_loc() File pandas/_libs/index.pyx:167, in pandas._libs.index.IndexEngine.get_loc() File pandas/_libs/index.pyx:196, in pandas._libs.index.IndexEngine.get_loc() File pandas/_libs/hashtable_class_helper.pxi:2152, in pandas._libs.hashtable.UInt64HashTable.get_item() File pandas/_libs/hashtable_class_helper.pxi:2176, in pandas._libs.hashtable.UInt64HashTable.get_item() KeyError: 17 The above exception was the direct cause of the following exception: KeyError Traceback (most recent call last) Cell In[210], line 1 ----> 1 check(series2) Cell In[208], line 4, in check(series) 2 for ix in series.index: 3 print(repr(ix), end=': ') ----> 4 print(repr(series.at[ix])) File ~/opt/mambaforge/envs/myenv/lib/python3.12/site-packages/pandas/core/indexing.py:2576, in _AtIndexer.__getitem__(self, key) 2573 raise ValueError("Invalid call for scalar access (getting)!") 2574 return self.obj.loc[key] -> 2576 return super().__getitem__(key) File ~/opt/mambaforge/envs/myenv/lib/python3.12/site-packages/pandas/core/indexing.py:2528, in _ScalarAccessIndexer.__getitem__(self, key) 2525 raise ValueError("Invalid call for scalar access (getting)!") 2527 key = self._convert_key(key) -> 2528 return self.obj._get_value(*key, takeable=self._takeable) File ~/opt/mambaforge/envs/myenv/lib/python3.12/site-packages/pandas/core/series.py:1249, in Series._get_value(self, label, takeable) 1246 return self._values[label] 1248 # Similar to Index.get_value, but we do not fall back to positional -> 1249 loc = self.index.get_loc(label) 1251 if is_integer(loc): 1252 return self._values[loc] File ~/opt/mambaforge/envs/myenv/lib/python3.12/site-packages/pandas/core/indexes/multi.py:3074, in MultiIndex.get_loc(self, key) 3072 return self._engine.get_loc(key) 3073 except KeyError as err: -> 3074 raise KeyError(key) from err 3075 except TypeError: 3076 # e.g. test_partial_slicing_with_multiindex partial string slicing 3077 loc, _ = self.get_loc_level(key, list(range(self.nlevels))) KeyError: ('a', <NA>) Issue Description
This seems like a weird corner case, but somehow pd.concat() creates an invalid MultiIndex when the concatenated Series (example shown above) or DataFrames have indices with Int64Dtype data type that contain NA values. When using at or loc with an index tuple containing NA a KeyError is raised. This doesn't happen with what should be an identical Series/DataFrame.
The two indices in the example do not compare equal according to .equals() but do have equal values according to ==:
>>> series1.index.equals(series2.index) False >>> series1.index == series2.index array([ True, True, True, True, True, True]) A difference can be seen in the levels and codes attributes:
>>> series1.index.levels FrozenList([['a', 'b'], [1, 2]]) >>> series2.index.levels FrozenList([['a', 'b'], [1, 2, <NA>]]) >>> series1.index.codes FrozenList([[0, 0, 0, 1, 1, 1], [0, 1, -1, 0, 1, -1]]) >>> series2.index.codes FrozenList([[0, 0, 0, 1, 1, 1], [0, 1, 2, 0, 1, 2]]) Expected Behavior
Lookup succeeds.
Installed Versions
INSTALLED VERSIONS
commit : 9c8bc3e
python : 3.12.3
python-bits : 64
OS : Linux
OS-release : 4.18.0-477.27.1.el8_8.x86_64
Version : #1 SMP Thu Aug 31 10:29:22 EDT 2023
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8
pandas : 2.3.3
numpy : 1.26.4
pytz : 2025.2
dateutil : 2.9.0.post0
pip : 25.2
Cython : 3.1.4
sphinx : 8.2.3
IPython : 9.6.0
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : None
blosc : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : 2025.9.0
html5lib : None
hypothesis : None
gcsfs : None
jinja2 : 3.1.6
lxml.etree : None
matplotlib : 3.10.1
numba : None
numexpr : 2.14.1
odfpy : None
openpyxl : 3.1.5
pandas_gbq : None
psycopg2 : 2.9.9
pymysql : None
pyarrow : 16.1.0
pyreadstat : None
pytest : 8.4.2
python-calamine : None
pyxlsb : None
s3fs : None
scipy : 1.16.2
sqlalchemy : 2.0.44
tables : 3.9.2
tabulate : 0.9.0
xarray : None
xlrd : None
xlsxwriter : 3.2.9
zstandard : 0.23.0
tzdata : 2025.2
qtpy : None
pyqt5 : None