Implement total core-hours and distribution of time of an experiment in /v3/performance/{expid}

[MissterP]

Hi!

The predecessor of this issue is this one.

This issue is related to this one.

I plan to enhance the API endpoint /v3/performance/{expid} by adding the total core-hours and distribution of time of an experiment. I would like to do the implementation to the file performance_metrics.py, this way:

• Total core-hours: Add to the function def to_json(self) -> Dict: a new atribute called Total_core_hours; the value would be self.valid_sim_core_hours_sum which is a new class attribute of PerformanceMetrics,that would be calculated at def _calculate_global_metrics and would use a new function called def _calculate_total_energy(self): which returns the sum of the core-hours of the considered SIM jobs.

• Distribution of time: Add to the function def to_json(self) -> Dict: two new atribute called phases_total_time and phases_critical_path_time which represent two embedded vectors with the time of each phase of an experiment for the two approaches (Total time and critical path); the values would be self.phases_total_time and self.phases_critical_path_time which are two new class attributes of PerformanceMetrics, they would be calculated at def _calculate_global_metrics and would use two new functions called def _calculate_phases_total_time(self) and def _calculate_phases_critical_path_time(self); which returns a vector with the total time used in the different phases depending of the approach. Also, between the def _calculate_global_metrics and the two functions to calculate the times there will be a new function called def _extract_phases_jobs which returns a list of the jobs of each phase.

[MissterP]

To obtain all the jobs necessary for the calculation of the different phases. I would suggest to obtain them from the PklOrganizer class. First, convert the pkl representing the jobs (from self.current_content) to standard jobs (using the factory; because not all sections are implemented) then, send them to PerformanceMetrics and to do the respective calculations (extract phases, ...). Other alternative the could be is to let PklOrganizer obtain the jobs (same as before), extract the phases and then, send them to PerformanceMetrics.

[LuiggiTenorioK]

@MissterP what do you mean by phases exactly?

About changing the PklOrganizer, it is ok to do it in any of the ways you described since it will still organize the jobs by certain criteria.

Also, it is true that this class doesn't have all the job sections because this class is quite old and didn't follow the new trends that the workflow developers were doing. However, there was never a need to update it.

[MissterP]

@MissterP what do you mean by phases exactly?

The phases would be the three phases of an experiment (in a general way) (jobs before sim jobs, sim jobs and jobs after sim jobs).

[mcastril]

I think that we will need a better way to categorize what is a SIM, a POST, a TRANSFER... So far we are only using the name of the job and this is not the best way

[MissterP]

I think that we will need a better way to categorize what is a SIM, a POST, a TRANSFER... So far we are only using the name of the job and this is not the best way

Could be possible to add a new attribute at the configuration of a job in the .yml where the jobs are stated and configured? Then, the value of that attribute would be an specific section?