Some changes for OML 3.0 #549

AlekseySh · 2024-04-27T10:09:39Z

CHANGELOG

Moved category wise metrics calculation logic from EmbeddingMetrics to functional metrics.
Removed fnmr@fmr metric from EmbeddingMetrics because we cannot guarantee correctness of its behaviour when postprocessor is presented and the metric is computationally heavy. [decided not to remove this metric]
Reworked handling empty bboxes (use one None instead of 4 Nones)
calc_retrieval_metrics_on_full, calc_gt_mask, calc_mask_to_ignore, apply_mask_to_ignore finally moved to tests to serve as adapters between the old and the new ways of computing metrics
pipelines: a bit of refactoring and improved type hints
added show argument to RetrievalResults.visualise()

…onfusing test

…rning into rework_reranking_before_merge

…pen-metric-learning into refactoring_integration

…rning into refactoring_integration

…tric-learning into continue_refactoring

AlekseySh · 2024-05-11T23:47:18Z

README.md

 dataset = ImageQueryGalleryLabeledDataset(df_val, transform=transform)

+# you can optionally provide categories to have category wise metrics
+query_categories = np.array(dataset.extra_data["category"])[dataset.get_query_ids()]


does it seem okay as an example?

deepslug · 2024-05-13T06:20:50Z

@AlekseySh

Removed fnmr@fmr metric from EmbeddingMetrics because we cannot guarantee correctness of its behaviour when postprocessor is presented and the metric is computationally heavy.

Is it possible to keep this metric available as an option if user wants to use it, because this metric is considered the most important metric in the field of biometrics (e.g., face/fingerprint recoginition) where score thresholding is often employed?

Having said that, it is also super helpful if this metric (or any other "the-lower-the-better" metrics) can be specified to metric_for_checkpointing like "OVERALL/fnmr@fmr/0.001" with the "mode" as "min" via YAML. The mode is hard-coded as "max" in the current implementation:

def parse_ckpt_callback_from_config(cfg: TCfg) -> ModelCheckpoint:
    return ModelCheckpoint(
        dirpath=Path.cwd() / "checkpoints",
        monitor=cfg["metric_for_checkpointing"],
        mode="max",
        save_top_k=1,
        verbose=True,
        filename="best",
    )

DaloroAT · 2024-05-13T17:04:44Z

oml/datasets/images.py

@@ -162,8 +161,9 @@ def __len__(self) -> int:
        return len(self._paths)

    def visualize(self, item: int, color: TColor = BLACK) -> np.ndarray:
-        bbox = torch.tensor(self._bboxes[item]) if (self._bboxes is not None) else torch.tensor([torch.nan] * 4)
-        image = get_img_with_bbox(im_path=self._paths[item], bbox=bbox, color=color)
+        img = np.array(imread_pillow(self.read_bytes(self._paths[item])))


you can read with cv2 directly to np. In addition it's faster

I did it to support paths as urls later on

DaloroAT · 2024-05-13T17:06:21Z

oml/functional/metrics.py

@@ -57,51 +61,30 @@ def calc_retrieval_metrics(
        metrics["precision"] = dict(zip(precision_top_k, precision))

    if map_top_k:
-        map = calc_map(gt_tops, n_gts, map_top_k)
-        metrics["map"] = dict(zip(map_top_k, map))
+        map_ = calc_map(gt_tops, n_gts, map_top_k)


respect built-in

DaloroAT · 2024-05-13T17:12:38Z

oml/functional/metrics.py

+        metrics["map"] = dict(zip(map_top_k, map_))
+
+    if query_categories is not None:
+        metrics_cat = {c: take_unreduced_metrics_by_mask(metrics, query_categories == c) for c in query_categories}


A bit strange to have a simple dict metrics but metrics keys are mixed with some categories keys on the same top level

metrics["map"][5] = 0.5 metrics["cmc"][3] = 0.4 metrics["OVERALL"]["cmc"][3] = 0.4 metrics["cats"]["cmc"][3] = 0.1 metrics["pigs"]["cmc"][3] = 0.2 ...

no, it's:

{ "cat": {"cmc": {1: 1.0}, "precision": {3: 2 / 3, 5: 2 / 3}}, "dog": {"cmc": {1: 1.0}, "precision": {3: 1 / 2, 5: 1 / 2}}, OVERALL_CATEGORIES_KEY: ..., }

DaloroAT · 2024-05-13T17:16:41Z

oml/metrics/embeddings.py

-
-        # todo 522: put back fnmr metric
+    def compute_metrics(self) -> TMetricsDict:  # type: ignore
+        self.acc = self.acc.sync()  # gathering data from devices happens here if DDP


wrong comment

DaloroAT · 2024-05-13T17:32:01Z

oml/metrics/embeddings.py


-                mask_dataset_sz = categories == category
-                metrics[category].update(calc_topological_metrics(embeddings[mask_dataset_sz], self.pcf_variance))
+            self.metrics_unreduced = {cat: {**metrics_r[cat], **metrics_t[cat]} for cat in metrics_r.keys()}


On this line metrics_r have all retrieval metrics and categories as top-level keys because of this, so

You can't do metrics_t[cat] for retrieval metric keys

self.metrics_unreduced won't include metrics_t

discussed offline

AlekseySh · 2024-05-19T12:16:04Z

@deepslug okay, I got your point

AlekseySh · 2024-05-19T23:27:49Z

tests/test_oml/test_functional/test_metrics/test_retrieval_metrics.py

+        map_top_k=tuple(),
+    )
+
+    assert math.isclose(metrics["cat"]["cmc"][1], 1)


rework this assers: compare dicts instead (to check if there are unwilling extra keys)

AlekseySh · 2024-05-22T14:50:16Z

Changes from this PR have been moved to other PRs:

AlekseySh added 30 commits April 19, 2024 08:06

reworking datasets

7b8e86c

removed ListDataset

cf3f1db

upd

d2e8215

minor

ad3fe93

fix

6919668

upd

e75f991

update_naming

4d381ed

simplified postprocessing and inference

08d7048

upd

05b8144

upd

5ebd696

upd

33f11ae

upd

db2d7e4

merge

997c88b

minor

bb21628

upd

548de97

addressed comments and introduced IIndexedDataset

6212be9

updated examples

2736922

update

aedec8b

upd

5e38ea0

upd

daf0c47

upd

19a6ed2

put_back_test

06864e2

fixes: type of dataset root, model link in validation

b54b6a1

optimizer postprocessing; solved issue with half precision; removed c…

b9d02a6

…onfusing test

minor: hotfix none postproc + raise error if no images in predict

f2be9b2

merge and update test

3df8de4

merge

b4c8b67

upd

b60727b

upd

ceac6b6

upd

ca74eb0

AlekseySh changed the base branch from main to refactoring_integration April 28, 2024 16:00

AlekseySh changed the base branch from refactoring_integration to oml_3.0_release April 28, 2024 16:06

AlekseySh added 2 commits May 10, 2024 22:08

Merge branch 'oml_3.0_release' of github.com:OML-Team/open-metric-lea…

52ec021

…rning into rework_reranking_before_merge

Merge branch 'rework_reranking_before_merge' of github.com:OML-Team/o…

11cdb68

…pen-metric-learning into refactoring_integration

AlekseySh changed the base branch from oml_3.0_release to refactoring_integration May 10, 2024 16:30

AlekseySh added 2 commits May 10, 2024 22:31

Merge branch 'oml_3.0_release' of github.com:OML-Team/open-metric-lea…

8881489

…rning into refactoring_integration

Merge branch 'refactoring_integration' of github.com:OML-Team/open-me…

aedc3e1

…tric-learning into continue_refactoring

AlekseySh changed the base branch from refactoring_integration to oml_3.0_release May 10, 2024 16:32

AlekseySh added the rework label May 10, 2024

AlekseySh changed the title ~~Refactoring polishing~~ Refactoring, moved matrix functions to tests, rm visualization.ipynb, reworked bboxes handling May 10, 2024

AlekseySh added 5 commits May 11, 2024 00:05

turn on ci

4131fcb

rm fnmr@fmr metric from EmbeddingMetrics

d04de1c

categories in metrics, rm fnmr from EmbMetrics

31f99b0

upd

1a82e27

minor

59fb478

AlekseySh changed the title ~~Refactoring, moved matrix functions to tests, rm visualization.ipynb, reworked bboxes handling~~ Changes for OML 3.0 May 11, 2024

AlekseySh commented May 11, 2024

View reviewed changes

AlekseySh changed the title ~~Changes for OML 3.0~~ Some changes for OML 3.0 May 11, 2024

AlekseySh added 3 commits May 12, 2024 20:45

merge

30ee330

upd

a63a565

tests

521df23

deepslug mentioned this pull request May 13, 2024

Differentiable version of FNMR@FMR metric to use it as loss #246

Open

DaloroAT reviewed May 13, 2024

View reviewed changes

AlekseySh commented May 19, 2024

View reviewed changes

AlekseySh marked this pull request as draft May 22, 2024 12:41

AlekseySh closed this May 22, 2024

AlekseySh deleted the continue_refactoring branch May 22, 2024 14:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Some changes for OML 3.0 #549

Some changes for OML 3.0 #549

AlekseySh commented Apr 27, 2024 •

edited

Loading

AlekseySh May 11, 2024 •

edited

Loading

deepslug commented May 13, 2024

DaloroAT May 13, 2024

AlekseySh May 22, 2024

DaloroAT May 13, 2024

DaloroAT May 13, 2024

AlekseySh May 20, 2024

DaloroAT May 13, 2024

DaloroAT May 13, 2024

AlekseySh May 22, 2024

AlekseySh commented May 19, 2024

AlekseySh May 19, 2024

AlekseySh commented May 22, 2024

Some changes for OML 3.0 #549

Some changes for OML 3.0 #549

Conversation

AlekseySh commented Apr 27, 2024 • edited Loading

AlekseySh May 11, 2024 • edited Loading

Choose a reason for hiding this comment

deepslug commented May 13, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AlekseySh commented May 19, 2024

Choose a reason for hiding this comment

AlekseySh commented May 22, 2024

AlekseySh commented Apr 27, 2024 •

edited

Loading

AlekseySh May 11, 2024 •

edited

Loading