Weight Loading

[avinash-218]

In this implementation, when the coco weights are loaded, there seems a bug in the loading process. There is no error but the weights are loaded incorrectly because of the load_weights function or tensorflow and h5py version compatibility issues.

I could successfully load the coco weights and train my model with my custom dataset but since the weights are loaded incorrectly, the training is done from scratch. I came to know this issue, when I tried to evaluate the model by visualization - when visualized the masks are very poor and the mAp is 0. This is because, when I loaded my trained weights for evaluation, because of the improper implementation or version compatibility issue, the weights are not loaded properly and hence the predictions were poor.

Correct me if I am wrong

[avinash-218]

The ‘model.load_weights’ seem to load the weights incorrectly due to version compatibility issues, resulting in training from scratch. So during training and evaluation, the coco and the previously trained weights were not loaded properly and hence in case of training, the training happens from scratch and in case of evaluation the loaded model predicts worse on the sample data.

Because of this reason, the losses at earlier steps at earlier epochs were too high and also the visual results looked random, not even close to the ground truth, and also the evaluation metrics such as mAP, mAR, F1 were 0. This can be solved by two ways :
By using ‘tf.keras.Model.load_weights’ instead of ‘model.load_weights’ - But still this can’t be used since it doesn’t support the ‘exclude’ argument.

By downgrading tensorflow from 2.7 to 2.5 worked in both training (from coco using exclude argument and from previously trained weights) and also in evaluations.

This worked for me. Correct me if i am wrong or my understanding is wrong

[Kamlesh364]

have you verified Keras requirements as well?

[avinash-218]

have you verified Keras requirements as well?

Yes, i have installed keras 2.7.0 version only

[caribeedu]

The ‘model.load_weights’ seem to load the weights incorrectly due to version compatibility issues, resulting in training from scratch. So during training and evaluation, the coco and the previously trained weights were not loaded properly and hence in case of training, the training happens from scratch and in case of evaluation the loaded model predicts worse on the sample data.

Because of this reason, the losses at earlier steps at earlier epochs were too high and also the visual results looked random, not even close to the ground truth, and also the evaluation metrics such as mAP, mAR, F1 were 0. This can be solved by two ways : By using ‘tf.keras.Model.load_weights’ instead of ‘model.load_weights’ - But still this can’t be used since it doesn’t support the ‘exclude’ argument.

By downgrading tensorflow from 2.7 to 2.5 worked in both training (from coco using exclude argument and from previously trained weights) and also in evaluations.

This worked for me. Correct me if i am wrong or my understanding is wrong

Thank you @avinash-218, your tip about this version problem saved me!