Adding custom metrics

[AmitMY]

There are four metrics available here https://github.com/sign-language-processing/signwriting-evaluation that I would like to report.
How can I add custom metrics? (I would like to optimize for one of the metrics)

[mjdenkowski]

Hi Amit,

Sockeye's loss functions are implemented in loss.py. You could use an existing loss function (e.g., MSELoss as a template for creating your own. You could also trace instances of MSELoss through the code to see where and how loss functions are connected into the training logic.

If your metrics require decoding, you could add them to the checkpoint decoder's decode_and_evaluate method (after BLEU, chrF, etc.). These metrics will be reported for every checkpoint but Sockeye does not currently support optimizing them directly.

Best,
Michael

[AmitMY]

By optimizing in my case I meant early-stopping. I'll try to add to the decode and evaluate and see if I can then early stop on those

[mjdenkowski]

You should be able to add your metric as a choice for --optimized-metric to enable using it for early stopping. One approach would be to do this first (add the name of your metric), start a training run with a tiny model, and see where it hits an error since your metric isn't yet implemented. This can identify where the next code change would be needed to support your metric.