2025-01-14-wang25a.md

title	When and How to Grow? On Efficient Pre-training via Model Growth
booktitle	Proceedings of the 16th Asian Conference on Machine Learning
year	2025
volume	260
series	Proceedings of Machine Learning Research
month	0
publisher	PMLR
pdf	https://raw.githubusercontent.com/mlresearch/v260/main/assets/wang25a/wang25a.pdf
url	https://proceedings.mlr.press/v260/wang25a.html
openreview	DDI359KC7v
abstract	The remarkable performance of GPT models has attracted widespread attention for large-scale language models. Despite their stunning performance, the huge pre-training cost is prohibitive. Progressive pre-training takes advantage of the faster convergence speed of small models to save computing overhead and shows great potential in accelerating pre-training. This work studies the two key issues in progressive pre-training: growth schedule and growth operation. First, we estimate the optimal growth point in theory. Then, we find in experiments that the growth operation can be performed after the model enters the convergence stage to achieve a high speed-up ratio. On the other hand, we propose progressive dimensionality growth for width expansion and redundant layers for depth expansion. Progressive dimensionality growth is a smoothed operation and improves training stability. Redundant layers implement function-preserving at a small cost and inherit the core parameters of adjacent layers, improving the utilization of knowledge learned by the original model. Our method follows strict function preservation and produces good training dynamics. Experimental results show that our method outperforms the baselines and achieves an acceleration rate of about 1.5 times while achieving the same training effect.
layout	inproceedings
issn	2640-3498
id	wang25a
tex_title	When and How to Grow? On Efficient Pre-training via Model Growth
firstpage	95
lastpage	110
page	95-110
order	95
cycles	false
bibtex_editor	Nguyen, Vu and Lin, Hsuan-Tien
editor	given family Vu Nguyen given family Hsuan-Tien Lin
bibtex_author	Wang, Jikai and Li, Juntao and Zhang, Min and Li, Zechang and Xia, Qingrong and Duan, Xinyu and Wang, Zhefeng and Huai, Baoxing
author	given family Jikai Wang given family Juntao Li given family Min Zhang given family Zechang Li given family Qingrong Xia given family Xinyu Duan given family Zhefeng Wang given family Baoxing Huai
date	2025-01-14
address
container-title	Proceedings of the 16th Asian Conference on Machine Learning
genre	inproceedings
issued	date-parts 2025 1 14
extras