model in ddp

[valiantljk]

Hi,
I noticed that in distributed training, there will be two models, one is the original model, and another is the wrapped model. For example,

allennlp/allennlp/training/gradient_descent_trainer.py

Line 351 in c5bff8b

output_dict = self._pytorch_model(**batch)

I have a general question (not specific to allennlp), in this case, can we make sure the two versions of model share the same parameters?
In runtime, is this some python magic, that they both refer to the same object but each has some customization? I'm a bit confused here.

[epwalsh]

Yes, they share the same parameters since the _pytorch_model is just a thin wrapper around the model. In other words, _pytorch_model just holds a (mutable) reference to model, it doesn't clone the parameters.

[valiantljk]

Also, why those customized functions, e.g., get_metrics will be lost in the wrapped ddp model?

[epwalsh]

I'm not familiar with Ray so I'm not sure.

Also, why those customized functions, e.g., get_metrics will be lost in the wrapped ddp model?

The _pytorch_model is just an instance of torch.nn.parallel.DistributedDataParallel. That class doesn't have a .get_metrics() method. We could call _pytorch_model.module.get_metrics() though, since _pytorch_model.module is just a reference to model.

[valiantljk]

Cool, Thanks @epwalsh. So can I use self._pytorch_model.module.get_regularization_penalty() to replace self.model.get_regularization_penalty()
as in

allennlp/allennlp/training/gradient_descent_trainer.py

Line 356 in c5bff8b

regularization_penalty = self.model.get_regularization_penalty()

,
If so, why do we need to keep two reference?

[epwalsh]

Yeup, you can do that. It's not necessary to keep two references, but it doesn't hurt either and makes things slightly simpler I think since _pytorch_model.module is only valid in the distributed context. In the non-distributed context, _pytorch_model is just model.

[valiantljk]

Got it! Very appreciate!