feat(core): Add support for _file column

[gbrgr]

Which issue does this PR close?

• Closes Support for _file metadata column #1766.

What changes are included in this PR?

This PR adds support for the _file reserved column in Iceberg table scans, allowing users to retrieve the file path for each row in their query results. This is useful for debugging, auditing, and tracking data lineage.

Core Implementation

1. Reserved column constants (reader.rs):

   • Added RESERVED_FIELD_ID_FILE = 2147483646 - reserved field ID per Iceberg spec
   • Added RESERVED_COL_NAME_FILE = "_file" - column name per Iceberg spec

2. Public API (scan/mod.rs):

   • Exposed RESERVED_COL_NAME_FILE as a public constant in the iceberg::scan module

Key Features

• Column ordering preserved: If a user selects ["col1", "_file", "col2"], the _file column appears in the correct position (second)
• Memory efficient: Uses Run-End Encoding (REE) to store the repeated file path value across all rows in a batch

Usage Example

use iceberg::scan::RESERVED_COL_NAME_FILE; let scan = table .scan() .select(["id", "name", RESERVED_COL_NAME_FILE]) .build()?; let batches = scan.to_arrow().await?; // Each batch will have 3 columns: id, name, and _file

Are these changes tested?

Yes, comprehensive tests have been added to verify the functionality:

New Tests (9 tests added)

Table Scan API Tests (7 tests)

1. test_select_with_file_column - Verifies basic functionality of selecting _file with regular columns
2. test_select_file_column_position - Verifies column ordering is preserved
3. test_select_file_column_only - Tests selecting only the _file column
4. test_file_column_with_multiple_files - Tests multiple data files scenario
5. test_file_column_at_start - Tests _file at position 0
6. test_file_column_at_end - Tests _file at the last position
7. test_select_with_repeated_column_names - Tests repeated column selection

Arrow Reader Helper Tests (2 tests)

8. test_add_file_path_column_ree - Tests the add_file_path_column_ree_at_position() helper
9. test_add_file_path_column_ree_empty_batch - Tests empty batch handling

[vustef]

How do we ask for _file column without having to explicitly list all the other columns? E.g. get me all columns + _file. There should be some shortcut for this.

[liurenjie1024]

Thanks @gbrgr for this pr. But I think we need to rethink how to compute the _file, _pos metadata column. While it's somehow trivial to compute _file, it's non trivial to compute _pos efficient, since when we read parquet files, we have filtered out some row groups. I think the best way is to push reading these two columns to arrow-rs.

[liurenjie1024]

I think we should add a metadata_columns module similar to what java did: https://github.com/apache/iceberg/blob/bb32b90c4ad63f037f0bda197cc53fb08c886934/core/src/main/java/org/apache/iceberg/MetadataColumns.java#L2

[vustef]

Thanks @gbrgr for this pr. But I think we need to rethink how to compute the _file, _pos metadata column. While it's somehow trivial to compute _file, it's non trivial to compute _pos efficient, since when we read parquet files, we have filtered out some row groups. I think the best way is to push reading these two columns to arrow-rs.

@liurenjie1024 I agree for _pos, and we have a PR there: apache/arrow-rs#8715
But _file seems like something that we don't need the arrow-rs to know about. Similarly, in future, for _row_id from V3 spec, we cannot expect arrow-rs to be responsible for computing that one.

How do we go forward with rethinking this, what would be the action items for us?