Pre-training log

[ctxya1207]

Could you please give the change of loss during unsupervised pre-training?

[pavancm]

It's been a while since this model was trained. Unfortunately I don't have the logs saved. But if you use the training script as is without changing the hyperparameters and the training data, it should be very similar to what I would have got.

[ctxya1207]

When I use this loss function, the loss tends to remain unchanged. Can you tell me why this is?