Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reworking datasets #541

Merged
merged 10 commits into from
Apr 20, 2024
Merged

Reworking datasets #541

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
7b8e86c
reworking datasets
AlekseySh Apr 19, 2024
cf3f1db
removed ListDataset
AlekseySh Apr 19, 2024
d2e8215
upd
AlekseySh Apr 19, 2024
ad3fe93
minor
AlekseySh Apr 19, 2024
6919668
fix
AlekseySh Apr 19, 2024
e75f991
upd
AlekseySh Apr 19, 2024
4d381ed
update_naming
AlekseySh Apr 19, 2024
0b20c07
updated classes
AlekseySh Apr 20, 2024
018d026
addressed comments
AlekseySh Apr 20, 2024
bc799ee
upd
AlekseySh Apr 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ tmp.ipynb
*outputs*
*.hydra*
*predictions.json*
*ml-runs*

# __________________________________

Expand Down
27 changes: 18 additions & 9 deletions docs/source/contents/datasets.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,42 +7,51 @@ Datasets
.. contents::
:local:

BaseDataset
ImageBaseDataset
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. autoclass:: oml.datasets.base.BaseDataset
.. autoclass:: oml.datasets.images.ImageBaseDataset
:undoc-members:
:show-inheritance:

.. automethod:: __init__
.. automethod:: __getitem__
.. automethod:: visualize

DatasetWithLabels
ImageLabeledDataset
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. autoclass:: oml.datasets.base.DatasetWithLabels
.. autoclass:: oml.datasets.images.ImageLabeledDataset
:undoc-members:
:show-inheritance:

.. automethod:: __init__
.. automethod:: __getitem__
.. automethod:: get_labels
.. automethod:: get_label2category
.. automethod:: visualize

DatasetQueryGallery
ImageQueryGalleryLabeledDataset
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. autoclass:: oml.datasets.base.DatasetQueryGallery
.. autoclass:: oml.datasets.images.ImageQueryGalleryLabeledDataset
:undoc-members:
:show-inheritance:

.. automethod:: __init__
.. automethod:: __getitem__
.. automethod:: get_query_ids
.. automethod:: get_gallery_ids
.. automethod:: get_labels
.. automethod:: visualize

ListDataset
ImageQueryGalleryDataset
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. autoclass:: oml.datasets.list_dataset.ListDataset
.. autoclass:: oml.datasets.images.ImageQueryGalleryDataset
:undoc-members:
:show-inheritance:

.. automethod:: __init__
.. automethod:: __getitem__
.. automethod:: get_query_ids
.. automethod:: get_gallery_ids
.. automethod:: visualize

EmbeddingPairsDataset
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Expand Down
35 changes: 30 additions & 5 deletions docs/source/contents/interfaces.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,22 +52,39 @@ ITripletLossWithMiner

.. automethod:: forward

IDatasetWithLabels
IBaseDataset
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. autoclass:: oml.interfaces.datasets.IDatasetWithLabels
.. autoclass:: oml.interfaces.datasets.IBaseDataset
:undoc-members:
:show-inheritance:

ILabeledDataset
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. autoclass:: oml.interfaces.datasets.ILabeledDataset
:undoc-members:
:show-inheritance:

.. automethod:: __getitem__
.. automethod:: get_labels

IDatasetQueryGallery
IQueryGalleryDataset
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. autoclass:: oml.interfaces.datasets.IDatasetQueryGallery
.. autoclass:: oml.interfaces.datasets.IQueryGalleryDataset
:undoc-members:
:show-inheritance:

.. automethod:: __getitem__
.. automethod:: get_query_ids
.. automethod:: get_gallery_ids

IQueryGalleryLabeledDataset
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. autoclass:: oml.interfaces.datasets.IQueryGalleryLabeledDataset
:undoc-members:
:show-inheritance:

.. automethod:: get_query_ids
.. automethod:: get_gallery_ids
.. automethod:: get_labels

IPairsDataset
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Expand All @@ -78,6 +95,14 @@ IPairsDataset
.. automethod:: __init__
.. automethod:: __getitem__

IVisualizableDataset
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. autoclass:: oml.interfaces.datasets.IVisualizableDataset
:undoc-members:
:show-inheritance:

.. automethod:: visualize

IBasicMetric
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. autoclass:: oml.interfaces.metrics.IBasicMetric
Expand Down
6 changes: 5 additions & 1 deletion oml/const.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
import tempfile
from pathlib import Path
from sys import platform
from typing import Any, Dict, Tuple, Union
from typing import Any, Dict, Optional, Sequence, Tuple, Union

from omegaconf import DictConfig

Expand Down Expand Up @@ -50,6 +50,7 @@ def get_cache_folder() -> Path:
GREEN = (0, 255, 0)
BLUE = (0, 0, 255)
GRAY = (120, 120, 120)
BLACK = (0, 0, 0)
PAD_COLOR = (255, 255, 255)

TCfg = Union[Dict[str, Any], DictConfig]
Expand All @@ -62,6 +63,9 @@ def get_cache_folder() -> Path:
MEAN_CLIP = (0.48145466, 0.4578275, 0.40821073)
STD_CLIP = (0.26862954, 0.26130258, 0.27577711)

TBBox = Tuple[int, int, int, int]
TBBoxes = Sequence[Optional[TBBox]]

CROP_KEY = "crop" # the format is [x1, y1, x2, y2]

# Required dataset format:
Expand Down
Loading
Loading