Experimental datumaro implementation #4920

AlbertvanHouten · 2025-10-20T08:10:13Z

This pull request introduces the new experimental dataset into the dataset classes of OTX. A lot of logic has been migrated to datumaro such as management of image color channels and the tiling implementation.

Summary

How to test

Checklist

The PR title and description are clear and descriptive
I have manually tested the changes
All changes are covered by automated tests
All related issues are linked to this PR (if applicable)
Documentation has been updated (if applicable)

Signed-off-by: Albert van Houten <[email protected]> Co-authored-by: Leonardo Lai <[email protected]>

…ing_extensions into feature/datumaro

Signed-off-by: Albert van Houten <[email protected]>

…4751) Signed-off-by: Albert van Houten <[email protected]> Co-authored-by: Grégoire Payen de La Garanderie <[email protected]>

…ing_extensions into feature/datumaro # Conflicts: # library/src/otx/data/dataset/base_new.py # library/src/otx/data/dataset/classification_new.py # library/src/otx/data/dataset/detection_new.py # library/src/otx/data/dataset/instance_segmentation_new.py # library/src/otx/data/dataset/keypoint_detection_new.py # library/src/otx/data/dataset/segmentation_new.py # library/src/otx/data/entity/sample.py

Signed-off-by: Albert van Houten <[email protected]>

…ing_extensions into feature/datumaro

Signed-off-by: Albert van Houten <[email protected]>

Signed-off-by: Albert van Houten <[email protected]> Co-authored-by: Albert van Houten <[email protected]>

Signed-off-by: Albert van Houten <[email protected]>

…orm/training_extensions into feature/datumaro

Signed-off-by: Albert van Houten <[email protected]>

…ing_extensions into feature/datumaro Signed-off-by: Albert van Houten <[email protected]> # Conflicts: # library/pyproject.toml # library/src/otx/data/dataset/anomaly.py # library/src/otx/data/dataset/base.py # library/src/otx/data/dataset/classification.py # library/src/otx/data/dataset/detection.py # library/src/otx/data/dataset/instance_segmentation.py # library/src/otx/data/dataset/keypoint_detection.py # library/src/otx/data/dataset/segmentation.py # library/src/otx/data/dataset/tile.py # library/src/otx/data/factory.py # library/src/otx/data/module.py # library/src/otx/data/transform_libs/torchvision.py # library/tests/unit/data/samplers/test_class_incremental_sampler.py # library/tests/unit/data/utils/test_utils.py

Copilot

Pull Request Overview

This PR implements an experimental Datumaro dataset integration for OTX, transitioning from legacy Datumaro components to the new experimental Dataset API. The changes introduce a new sample-based architecture while maintaining compatibility with existing OTX functionality.

Key changes:

Migration from legacy Datumaro components to experimental Dataset API with schema-based conversion
Introduction of new OTXSample-based data entities with PyTree registration for TorchVision compatibility
Replacement of legacy polygon handling with numpy ragged arrays for better performance
Comprehensive test updates and new test implementations for the updated dataset architecture

Reviewed Changes

Copilot reviewed 59 out of 59 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
pyproject.toml	Updates Datumaro dependency to experimental branch version
library/src/otx/data/dataset/*.py	Implements new dataset classes using experimental Datumaro with sample-based architecture
library/src/otx/data/entity/sample.py	Adds new OTXSample classes with PyTree registration for transform compatibility
library/tests/unit/types/test_label.py	Updates label tests to use new hierarchical label categories
library/tests/unit/data/transform_libs/test_torchvision.py	Converts polygon handling from Datumaro objects to numpy arrays

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

library/tests/unit/data/transform_libs/test_torchvision.py

Copilot · 2025-10-29T09:25:49Z

library/src/otx/data/transform_libs/utils.py

+            rescaled_polygons[i] = scaled_points
+        else:
+            # Handle empty or invalid polygons
+            rescaled_polygons[i] = np.array([[0, 0], [0, 0], [0, 0]], dtype=np.float32)


[nitpick] Creating a dummy polygon with three identical points may not be the best approach for handling empty/invalid polygons. Consider using an empty array or a clearly marked invalid polygon structure that can be properly filtered out later in the pipeline.

Suggested change

rescaled_polygons[i] = np.array([[0, 0], [0, 0], [0, 0]], dtype=np.float32)

rescaled_polygons[i] = np.empty((0, 2), dtype=np.float32)

why exactly 3 points BTW? Why not just empty tensor or 0 tensor?

Not entirely sure why Greg applied it in this manner, but I guess there is a validation step somewhere that requires 3 points for a valid polygon.

library/src/otx/data/transform_libs/utils.py

# Conflicts: # library/tests/unit/data/dataset/test_base.py # library/tests/unit/data/test_tiling.py

AlbertvanHouten · 2025-12-17T12:42:21Z

library/src/otx/recipe/semantic_segmentation/dino_v2_tile.yaml

@kprokofi please review if these changes make sense. The old logic only worked when tiling with square images.

I think padding do nothing here. We resize and pad to the same size.

Without the padding, tiling only works if the image is square (which it was previously in the integration test). I have replaced it with another dataset, as the polygons in the dataset were self-intersecting, which isn't supported, and this new dataset does have rectangle images, which broke specifically this tiling recipe.

leoll2 · 2025-12-17T16:51:28Z

.github/workflows/lib-lint-and-test.yaml

+      # Unit test models separately using --forked flag so that each is run in a separate process.
+      # Running these models after each other from the same process can cause segfaults
+      - name: Unit testing models
        working-directory: library
-        run: uv run pytest tests/unit --cov --cov-report=xml
+        env:
+          OMP_NUM_THREADS: "1"
+        run: uv run pytest tests/unit/backend/native/models --cov --cov-report=xml --cov-append --forked


Do you have any idea what causes the segfaults?

Unclear to me. I've tried a few different things to see if it would fix it, like clearing up pytorch memory usage, but this was the only way that I could fix it.

library/src/otx/backend/native/callbacks/gpu_mem_monitor.py

library/src/otx/data/factory.py

library/src/otx/recipe/detection/rtmdet_tiny.yaml

kprokofi

First round of comments

library/src/otx/backend/native/models/detection/rtdetr.py

library/src/otx/backend/native/tools/tile_merge.py

library/src/otx/backend/openvino/models/segmentation.py

library/src/otx/data/entity/sample.py

library/src/otx/data/factory.py

library/src/otx/data/transform_libs/torchvision.py

library/src/otx/data/transform_libs/utils.py

kprokofi · 2025-12-19T14:04:13Z

library/src/otx/data/transform_libs/utils.py

+            rescaled_polygons[i] = scaled_points
+        else:
+            # Handle empty or invalid polygons
+            rescaled_polygons[i] = np.array([[0, 0], [0, 0], [0, 0]], dtype=np.float32)


why exactly 3 points BTW? Why not just empty tensor or 0 tensor?

kprokofi

Second round of comments

kprokofi · 2025-12-19T14:28:17Z

library/src/otx/recipe/semantic_segmentation/dino_v2_tile.yaml

I think padding do nothing here. We resize and pad to the same size.

library/src/otx/types/label.py

library/tests/conftest.py

kprokofi · 2025-12-19T14:44:41Z

library/tests/integration/api/test_augmentation.py

        "RandomPhotometricDistort",
        "RandomGaussianBlur",
        "RandomGaussianNoise",


Why did you remove TopdownAffine from configurable augs list? We actually provide possibility to turn on/off this augmentation in Geti

Signed-off-by: Albert van Houten <[email protected]>

…from dataset Signed-off-by: Albert van Houten <[email protected]>

Signed-off-by: Albert van Houten <[email protected]>

AlbertvanHouten · 2026-01-06T15:48:38Z

I believe all comments should be handled now @kprokofi. Please recheck the PR when you have time.

leoll2 · 2026-01-07T13:25:35Z

library/src/otx/data/entity/sample.py

+    """OTXDataItemSample is a base class for OTX data items."""
+
+    subset: Subset = subset_field()
+    image: np.ndarray | tv_tensors.Image | torch.Tensor = image_field(dtype=pl.UInt8(), channels_first=False)


All other samples have channels_first=False. Why does SegmentationSample differ?

I think its due to different transformations being called for the segmentation task. Without channels first, the predicted masks will have H/W channels swapped.

library/src/otx/backend/native/models/detection/rtdetr.py

Signed-off-by: Albert van Houten <[email protected]>

…ing_extensions into feature/datumaro

Signed-off-by: Albert van Houten <[email protected]>

kprokofi

Thank you for your work!

AlbertvanHouten and others added 16 commits September 4, 2025 10:10

Classification Dataset refactor (#4606)

9bd3fa7

Signed-off-by: Albert van Houten <[email protected]> Co-authored-by: Leonardo Lai <[email protected]>

Merge branch 'develop' of https://github.com/open-edge-platform/train…

373a046

…ing_extensions into feature/datumaro

Detection dataset updated to use new datumaro version (#4675)

9f26bb8

Signed-off-by: Albert van Houten <[email protected]>

Update the segmentation dataset to use new Datumaro dataset (#4676)

8b2fe8b

Instance segmentation support with the new Datumaro dataset (#4686)

3d84b50

Migrate OTX Keypoint detection to use the new experimental dataset (#…

d1ff0dc

…4751) Signed-off-by: Albert van Houten <[email protected]> Co-authored-by: Grégoire Payen de La Garanderie <[email protected]>

Move new dataset files to library folder

a3d82dd

Signed-off-by: Albert van Houten <[email protected]>

Update OTX classification task to use legacy converter (#4769)

c1b8084

Signed-off-by: Albert van Houten <[email protected]>

Merge branch 'develop' of https://github.com/open-edge-platform/train…

9fd7abc

…ing_extensions into feature/datumaro

Migrate anomaly task to use the new Datumaro dataset (#4770)

4751cb2

Merge branch 'develop' of https://github.com/open-edge-platform/train…

5ff3b59

…ing_extensions into feature/datumaro

Point datumaro deps to the dev branch

6088088

Signed-off-by: Albert van Houten <[email protected]>

Tilling support with the experimental Datumaro dataset (#4858)

7827ca5

Signed-off-by: Albert van Houten <[email protected]> Co-authored-by: Albert van Houten <[email protected]>

Use new datumaro for multi label and hierarchical classification (#4815)

64e58ca

Signed-off-by: Albert van Houten <[email protected]>

Merge branch 'feature/datumaro' of https://github.com/open-edge-platf…

fdb6afe

…orm/training_extensions into feature/datumaro

github-actions bot added DEPENDENCY Any changes in any dependencies (new dep or its version) should be produced via Change Request on PM TEST Any changes in tests BUILD labels Oct 20, 2025

AlbertvanHouten added 4 commits October 23, 2025 11:55

Replace old datasets with new one

bdaa084

Signed-off-by: Albert van Houten <[email protected]>

Update hierarchical label tests to use new datumaro categories

eb6fbda

Signed-off-by: Albert van Houten <[email protected]>

Cleanup remaining tests

a7d0c51

Signed-off-by: Albert van Houten <[email protected]>

AlbertvanHouten changed the title ~~Feature/datumaro~~ Experimental datumaro implementation Oct 29, 2025

AlbertvanHouten marked this pull request as ready for review October 29, 2025 09:24

AlbertvanHouten requested a review from a team as a code owner October 29, 2025 09:24

Copilot AI review requested due to automatic review settings October 29, 2025 09:24

Copilot AI reviewed Oct 29, 2025

View reviewed changes

leoll2 added 2 commits November 1, 2025 10:13

Merge branch 'develop' into feature/datumaro

3efe5aa

# Conflicts: # library/tests/unit/data/dataset/test_base.py # library/tests/unit/data/test_tiling.py

fix random generator

715bbb5

AlbertvanHouten commented Dec 17, 2025

View reviewed changes

leoll2 reviewed Dec 17, 2025

View reviewed changes

AlbertvanHouten requested a review from kprokofi December 18, 2025 09:00

kprokofi reviewed Dec 19, 2025

View reviewed changes

addres a few PR comments.

1c14e75

Signed-off-by: Albert van Houten <[email protected]>

AlbertvanHouten force-pushed the feature/datumaro branch from 9095a35 to 1c14e75 Compare January 5, 2026 12:13

AlbertvanHouten added 2 commits January 6, 2026 10:22

Fix segmentation sample

26fcd55

Signed-off-by: Albert van Houten <[email protected]>

Merge instance segmentation samples and remove include_polygons flag …

d83f9dd

…from dataset Signed-off-by: Albert van Houten <[email protected]>

github-actions bot added the DOC Improvements or additions to documentation label Jan 6, 2026

AlbertvanHouten added 3 commits January 6, 2026 15:03

Remove ipex reference

1182f0e

Signed-off-by: Albert van Houten <[email protected]>

Revert channel order checks in torch vision

9d1433a

Signed-off-by: Albert van Houten <[email protected]>

Fix typing for images in torchvision conversion function

fe1a1c2

Signed-off-by: Albert van Houten <[email protected]>

AlbertvanHouten requested a review from kprokofi January 6, 2026 15:48

leoll2 reviewed Jan 7, 2026

View reviewed changes

leoll2 previously approved these changes Jan 7, 2026

View reviewed changes

Remove unused labelinfo methods

d718b3e

Signed-off-by: Albert van Houten <[email protected]>

AlbertvanHouten dismissed leoll2’s stale review via d718b3e January 8, 2026 08:37

AlbertvanHouten added 2 commits January 8, 2026 10:02

Merge branch 'develop' of https://github.com/open-edge-platform/train…

1606079

…ing_extensions into feature/datumaro

Fix failing test

5fb4814

Signed-off-by: Albert van Houten <[email protected]>

kprokofi approved these changes Jan 9, 2026

View reviewed changes

AlbertvanHouten enabled auto-merge January 9, 2026 11:02

mramotowski approved these changes Jan 9, 2026

View reviewed changes

AlbertvanHouten added this pull request to the merge queue Jan 9, 2026

Merged via the queue into develop with commit 4376a84 Jan 9, 2026
45 checks passed

kprokofi mentioned this pull request Jan 14, 2026

Cleanup unnecessary OTX entities #5201

Open

This was referenced Jan 15, 2026

Improve consistency and reliability of OTX data transformations #5208

Open

Remove support for legacy Datumaro format #5209

Open

	rescaled_polygons[i] = np.array([[0, 0], [0, 0], [0, 0]], dtype=np.float32)
	rescaled_polygons[i] = np.empty((0, 2), dtype=np.float32)

Experimental datumaro implementation #4920

Experimental datumaro implementation #4920

Uh oh!

Conversation

AlbertvanHouten commented Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

How to test

Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI Oct 29, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AlbertvanHouten Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kprokofi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kprokofi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AlbertvanHouten commented Jan 6, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kprokofi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

AlbertvanHouten commented Oct 20, 2025 •

edited

Loading

AlbertvanHouten Dec 19, 2025 •

edited

Loading