Experimental datumaro implementation

[AlbertvanHouten]

This pull request introduces the new experimental dataset into the dataset classes of OTX. A lot of logic has been migrated to datumaro such as management of image color channels and the tiling implementation.

Summary

How to test

Checklist

• The PR title and description are clear and descriptive
• I have manually tested the changes
• All changes are covered by automated tests
• All related issues are linked to this PR (if applicable)
• Documentation has been updated (if applicable)

[Copilot]

Pull Request Overview

This PR implements an experimental Datumaro dataset integration for OTX, transitioning from legacy Datumaro components to the new experimental Dataset API. The changes introduce a new sample-based architecture while maintaining compatibility with existing OTX functionality.

Key changes:

• Migration from legacy Datumaro components to experimental Dataset API with schema-based conversion
• Introduction of new OTXSample-based data entities with PyTree registration for TorchVision compatibility
• Replacement of legacy polygon handling with numpy ragged arrays for better performance
• Comprehensive test updates and new test implementations for the updated dataset architecture

Reviewed Changes

Copilot reviewed 59 out of 59 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
pyproject.toml	Updates Datumaro dependency to experimental branch version
library/src/otx/data/dataset/*.py	Implements new dataset classes using experimental Datumaro with sample-based architecture
library/src/otx/data/entity/sample.py	Adds new OTXSample classes with PyTree registration for transform compatibility
library/tests/unit/types/test_label.py	Updates label tests to use new hierarchical label categories
library/tests/unit/data/transform_libs/test_torchvision.py	Converts polygon handling from Datumaro objects to numpy arrays

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The polygon data structure has changed from Datumaro Polygon objects to numpy ragged arrays, but the test doesn't validate that the new format maintains the same geometric properties. Consider adding assertions to verify that the polygon areas and vertex coordinates are equivalent between the old and new formats.

[Copilot]

The function modifies the input polygon array in-place, which could cause issues if the original array is used elsewhere. The function should create a copy before modification to avoid unintended side effects.

Suggested change

	def revert_hflip(polygon: np.ndarray, width: int) -> np.ndarray:
	def revert_hflip(polygon: np.ndarray, width: int) -> np.ndarray:
	polygon = polygon.copy()

[Copilot]

[nitpick] Creating a dummy polygon with three identical points may not be the best approach for handling empty/invalid polygons. Consider using an empty array or a clearly marked invalid polygon structure that can be properly filtered out later in the pipeline.

Suggested change

	rescaled_polygons[i] = np.array([[0, 0], [0, 0], [0, 0]], dtype=np.float32)
	rescaled_polygons[i] = np.empty((0, 2), dtype=np.float32)

[Copilot]

[nitpick] Same issue as in rescale_polygons - using dummy points for invalid polygons. This pattern is repeated in multiple functions and should be standardized across the codebase for consistency.