intro

realization

1. Distilling Knowledge via Knowledge Review

   • paper: https://jiaya.me/papers/kdreview_cvpr21.pdf
   • code: https://github.com/Jia-Research-Lab/ReviewKD
   • paddle realization: link

2. Distilling Knowledge via Knowledge Review

   • paper: https://arxiv.org/abs/2009.04759
   • code: https://github.com/nmaac/acon
   • paddle realization: link

3. Rethinking Soft Labels for Knowledge Distillation: A Bias-variance Tradeoff Perspective

   • paper: https://arxiv.org/abs/2102.00650
   • code: https://github.com/bellymonster/Weighted-Soft-Label-Distillation
   • paddle realization: link

• Exponential Moving Average
  • paper: https://en.wikipedia.org/wiki/Moving_average
  • code: so many, not listed here
  • gen ema model mannually: link