Knowledge Distillation

ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillation
We propose ShortOPD, a short-to-long on-policy distillation schedule that recovers the free-form generation ability of pruned LLMs at a fraction of the training cost.
Qingyu Zhang, Qianhao Yuan, Hongyu Lin, Yaojie Lu, Xianpei Han, Le Sun, Xiang Li, Ming Xu, Jiarui Li, Xiuyin Zhao
PDF Cite
ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillation