cherry-picked `c18035e1` #1324

Job	Run time
e2e (v1.29.2, long-running-resume,from-volume-resume,median-stop)	15m 28s
e2e (v1.29.2, grid,bayesian-optimization,tpe,multivariate-tpe,cma-es,hyperband)	18m 43s
e2e (v1.29.2, hyperopt-distribution,optuna-distribution)	11m 58s
e2e (v1.29.2, file-metrics-collector,pytorchjob-mnist)	14m 11s
e2e (v1.29.2, median-stop-with-json-format,file-metrics-collector-with-json-format)	14m 24s
e2e (v1.30.7, long-running-resume,from-volume-resume,median-stop)	16m 0s
e2e (v1.30.7, grid,bayesian-optimization,tpe,multivariate-tpe,cma-es,hyperband)	18m 33s
e2e (v1.30.7, hyperopt-distribution,optuna-distribution)	12m 26s
e2e (v1.30.7, file-metrics-collector,pytorchjob-mnist)	14m 0s
e2e (v1.30.7, median-stop-with-json-format,file-metrics-collector-with-json-format)	12m 46s
e2e (v1.31.3, long-running-resume,from-volume-resume,median-stop)	13m 30s
e2e (v1.31.3, grid,bayesian-optimization,tpe,multivariate-tpe,cma-es,hyperband)	21m 5s
e2e (v1.31.3, hyperopt-distribution,optuna-distribution)	12m 32s
e2e (v1.31.3, file-metrics-collector,pytorchjob-mnist)	12m 8s
e2e (v1.31.3, median-stop-with-json-format,file-metrics-collector-with-json-format)	12m 42s
	3h 40m 26s

Provide feedback