Fix OverflowError: Python int too large to convert to C long raised during annotation label map creation with custom labels.

[briangow]

The patch makes the label‑store dtype explicit (int64) and replaces row‑wise .loc assignment with a keyed, vectorized update.

[bemoody]

Pandas is Greek to me. What is the issue this is trying to solve? What integer value is too large to convert to long? Annotation *type codes* can't be too large; those must be between 1 and 49.

[briangow]

@bemoody this is to fix the test error currently seen in #544. My understanding is that this is a known issue when Pandas tries to unify data types. Here is the explanation (along with the full test error):

WFDB’s own constraints keep label_store small (1–49), so it isn’t your annotation values that are “too big.” The overflow comes from pandas’ internal attempt to unify dtypes when a row assignment ends up behaving like an append under the hood. On some platforms/pandas versions, that path tries to convert Python ints through a C long and can raise OverflowError even if your data are reasonable.

=================================== FAILURES =================================== ____________________________ TestAnnotation.test_3 _____________________________ self = <tests.test_annotation.TestAnnotation testMethod=test_3> def test_3(self): """ Annotation file with custom annotation types Target file created with: rdann -r sample-data/1003 -a atr > ann-3 """ > annotation = wfdb.rdann("sample-data/1003", "atr") tests/test_annotation.py:183: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ wfdb/io/annotation.py:2028: in rdann annotation.set_label_elements(return_label_elements) wfdb/io/annotation.py:1408: in set_label_elements self.convert_label_attribute(contained_elements[0], e) wfdb/io/annotation.py:1471: in convert_label_attribute label_map = self.create_label_map(inplace=False) wfdb/io/annotation.py:911: in create_label_map label_map.loc[i] = self.custom_labels.loc[i] .venv/lib/python3.9/site-packages/pandas/core/indexing.py:912: in __setitem__ iloc._setitem_with_indexer(indexer, value, self.name) .venv/lib/python3.9/site-packages/pandas/core/indexing.py:1933: in _setitem_with_indexer self._setitem_with_indexer_missing(indexer, value) .venv/lib/python3.9/site-packages/pandas/core/indexing.py:2329: in _setitem_with_indexer_missing self.obj._mgr = self.obj._append(value)._mgr .venv/lib/python3.9/site-packages/pandas/core/frame.py:10581: in _append other = row_df.infer_objects(copy=False).rename_axis( .venv/lib/python3.9/site-packages/pandas/core/generic.py:6910: in infer_objects new_mgr = self._mgr.convert(copy=copy) .venv/lib/python3.9/site-packages/pandas/core/internals/managers.py:466: in convert return self.apply("convert", copy=copy, using_cow=using_copy_on_write()) .venv/lib/python3.9/site-packages/pandas/core/internals/managers.py:363: in apply applied = getattr(b, f)(**kwargs) .venv/lib/python3.9/site-packages/pandas/core/internals/blocks.py:656: in convert blocks = self.split_and_operate( .venv/lib/python3.9/site-packages/pandas/core/internals/blocks.py:473: in split_and_operate rbs = func(nb, *args, **kwargs) .venv/lib/python3.9/site-packages/pandas/core/internals/blocks.py:675: in convert res_values = lib.maybe_convert_objects( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ??? E OverflowError: Python int too large to convert to C long pandas/_libs/lib.pyx:2619: OverflowError =========================== short test summary info ============================ FAILED tests/test_annotation.py::TestAnnotation::test_3 - OverflowError: Python int too large to convert to C long =================== 1 failed, 69 passed in 281.48s (0:04:41) =================== Error: Process completed with exit code 1. 

[bemoody]

The problem seems to be related to NPY_PROMOTION_STATE=weak_and_warn; perhaps it is the same or similar to pandas-dev/pandas#60023 (which was closed, understandably, because this was never meant to be supported long-term.)

I'd suggest that we've gotten what benefit we can out of NPY_PROMOTION_STATE, and we can probably just remove it from test.yml. Or we could leave it set only for the test-deb11-i386 job.

As for create_label_map, without understanding this code I'd rather not change it. I'd much rather eliminate the use of pandas and rewrite the logic using standard Python data structures and/or numpy.

[briangow]

@bemoody , thanks for looking into this. It does seem like it is probably related to pandas-dev/pandas#60023. I'd be fine with removing NPY_PROMOTION_STATE but if you prefer to keep it only for test-deb11-i386 that is fine also. I can update this PR once you let me know your thoughts.