DOC: Add expanded index descriptors for specifying for RangeIndex-as-metadata in Parquet file schema (#25709)

Author: wesm

doc/source/development/developer.rst

@@ -37,12 +37,19 @@ So that a ``pandas.DataFrame`` can be faithfully reconstructed, we store a
 
 .. code-block:: text
 
- {'index_columns': ['__index_level_0__', '__index_level_1__', ...],
+ {'index_columns': [<descr0>, <descr1>, ...],
  'column_indexes': [<ci0>, <ci1>, ..., <ciN>],
  'columns': [<c0>, <c1>, ...],
- 'pandas_version': $VERSION}
+ 'pandas_version': $VERSION,
+ 'creator': {
+ 'library': $LIBRARY,
+ 'version': $LIBRARY_VERSION
+ }}
 
-Here, ``<c0>``/``<ci0>`` and so forth are dictionaries containing the metadata
+The "descriptor" values ``<descr0>`` in the ``'index_columns'`` field are
+strings (referring to a column) or dictionaries with values as described below.
+
+The ``<c0>``/``<ci0>`` and so forth are dictionaries containing the metadata
 for each column, *including the index columns*. This has JSON form:
 
 .. code-block:: text
@@ -53,26 +60,37 @@ for each column, *including the index columns*. This has JSON form:
  'numpy_type': numpy_type,
  'metadata': metadata}
 
-.. note::
+See below for the detailed specification for these.
+
+Index Metadata Descriptors
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``RangeIndex`` can be stored as metadata only, not requiring serialization. The
+descriptor format for these as is follows:
 
- Every index column is stored with a name matching the pattern
- ``__index_level_\d+__`` and its corresponding column information is can be
- found with the following code snippet.
+.. code-block:: python
 
- Following this naming convention isn't strictly necessary, but strongly
- suggested for compatibility with Arrow.
+ index = pd.RangeIndex(0, 10, 2)
+ {'kind': 'range',
+ 'name': index.name,
+ 'start': index.start,
+ 'stop': index.stop,
+ 'step': index.step}
 
- Here's an example of how the index metadata is structured in pyarrow:
+Other index types must be serialized as data columns along with the other
+DataFrame columns. The metadata for these is a string indicating the name of
+the field in the data columns, for example ``'__index_level_0__'``.
 
- .. code-block:: python
+If an index has a non-None ``name`` attribute, and there is no other column
+with a name matching that value, then the ``index.name`` value can be used as
+the descriptor. Otherwise (for unnamed indexes and ones with names colliding
+with other column names) a disambiguating name with pattern matching
+``__index_level_\d+__`` should be used. In cases of named indexes as data
+columns, ``name`` attribute is always stored in the column descriptors as
+above.
 
- # assuming there's at least 3 levels in the index
- index_columns = metadata['index_columns'] # noqa: F821
- columns = metadata['columns'] # noqa: F821
- ith_index = 2
- assert index_columns[ith_index] == '__index_level_2__'
- ith_index_info = columns[-len(index_columns):][ith_index]
- ith_index_level_name = ith_index_info['name']
+Column Metadata
+~~~~~~~~~~~~~~~
 
 ``pandas_type`` is the logical type of the column, and is one of:
 
@@ -161,4 +179,8 @@ As an example of fully-formed metadata:
  'numpy_type': 'int64',
  'metadata': None}
  ],
- 'pandas_version': '0.20.0'}
+ 'pandas_version': '0.20.0',
+ 'creator': {
+ 'library': 'pyarrow',
+ 'version': '0.13.0'
+ }}