Publications

Xin Men, Xin Men, Mingyu Xu, Mingyu Xu, Qingyu Zhang, Qingyu Zhang, Qianhao Yuan, Qianhao Yuan, Bingning Wang, Bingning Wang, Hongyu Lin, Hongyu Lin, Yaojie Lu, Yaojie Lu, Xianpei Han, Xianpei Han, Weipeng Chen, Weipeng Chen
May, 2025 In ACL Findings 2025
ShortGPT: Layers in Large Language Models are More Redundant Than You Expect
ShortGPT: Layers in Large Language Models are More Redundant Than You Expect

We investigate the redundancy within Transformer layers and propose an effective layer-based pruning method.

PDF Cite
Xin Men*, Xin Men*, Mingyu Xu*, Mingyu Xu*, Bingning Wang†, Bingning Wang†, Qingyu Zhang, Qingyu Zhang, Hongyu Lin, Hongyu Lin, Xianpei Han, Xianpei Han, Weipeng Chen, Weipeng Chen
September, 2024 In NeurIPS 2024
Base of RoPE Bounds Context Length
Base of RoPE Bounds Context Length

This work contributes to the investigation of the lower bounds of the Base in RoPE, providing a theoretical foundation for the long-context extrapolation of models.

PDF Cite