Abstract
Structured pruning compresses Large Language Models in a hardware-friendly way, but is usually evaluated on multiple-choice tasks, while pruned models can fail badly at free-form generation. We identify two findings that explain this gap: pass@1 with greedy decoding collapses after compression while pass@k rebounds with repeated sampling, meaning useful generations are demoted rather than erased, and failures mostly manifest as repetitive suffixes. This motivates On-Policy Distillation (OPD), where the original model acts as a frozen teacher over the compressed model’s own rollouts. Since long rollouts waste early training on low-value repetitive text, we introduce ShortOPD, a short-to-long schedule that identifies teacher-confirmed repetitive suffixes, keeps the useful prefix as the effective rollout length, and budgets future rollouts accordingly. Across math, code, and open-ended tasks, ShortOPD lifts the compressed model’s score to roughly 9x its unrecovered level and 1.6-4.4x standard baselines, while matching a fixed 8192-token horizon within two points at about a quarter of the training time with 71% fewer rollout tokens.
Qingyu Zhang
Master Student of Computer Science and Technology
I work on AI sales and customer-service agents, with experience across LLM pretraining, post-training, evaluation, and model efficiency.