MLFlow integration

[alxn4]

Goals

• Interactive Interface that showcases the generated plots and important metrics. The link to it is provided to the user in the mlflow output.
• Alternatively: single pager that statically contains all plots and important metrics
• Map getML entities to mlflow entities (pipe.{fit,predict} : run → (feature learning : subrub, training : subrun, predict : subrun) → target : subrun))

Considerations & Ideas

• For Map getML entities to mlflow entities (pipe.{fit,predict} : run → (feature learning : subrub, training : subrun, predict : subrun) → target : subrun)) we need to able to identify events emitted by the engine
  • currently, there is no way to intercept the event stream
  • events are handled in communication.log
  • To allow for handling of events (in this and other cases) we should introduce an event handling mechanism to the python-api
  • We can introduce another dedicated event handler (EngineLogHandlerRegistry, cf. [EngineExceptionHandlerRegistry](https://github.com/getml/getml-community/blob/main/src/python-api/getml/exceptions.py#L94)) that allows callers to hook into the engine’s raw event stream

Breakdown

• Dataset management:
  • Gain better understanding of where and how dataset information is logged.
  • Include relational structure of data in logging
  • Figure out how and where to differentially log performance metrics across datasets
• Tag logging customization: Come up with an intuitive way to log tags. Keep in mind the gap between getml’s native list based tags versus mlflow’s dictionary based tags.
• Introduce EngineLogHandlerRegistry @sören Nikolaus

Nest Step

• Introduce EngineLogHandlerRegistry @sören Nikolaus
• ??? @jan Meyer